Department of Health and Human Services 
OFFICE OF 
INSPECTOR GENERAL 
RYAN WHITE EVALUATION SYSTEMS 
Title I: Grants to Metropolitan Areas 
JUNE GIBBS BROWN 
Inspector General 
AUGUST 1999 
OEI-05-98-00392  OFFICE OF INSPECTOR GENERAL 
The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, is to 
protect the integrity of the Department of Health and Human Services programs as well as the 
health and welfare of beneficiaries served by them.  This statutory mission is carried out through a 
nationwide program of audits, investigations, inspections, sanctions, and fraud alerts.  The 
Inspector General informs the Secretary of program and management problems and recommends 
legislative, regulatory, and operational approaches to correct them. 
Office of Evaluation and Inspections 
The Office of Evaluation and Inspections (OEI) is one of several components of the Office of 
Inspector General.  It conducts short-term management and program evaluations (called 
inspections) that focus on issues of concern to the Department, the Congress, and the public.  The 
inspection reports provide findings and recommendations on the efficiency, vulnerability, and 
effectiveness of departmental programs. 
OEI's Chicago Regional Office prepared this report under the direction of William Moran, 
Regional Inspector General.  Principal OEI staff included: 
REGION  HEADQUARTERS 
William Moran, Project Leader
 Elise Stein, Program Specialist 
Ann Maxwell, Lead Analyst
 Rada Spencer, Program Specialist 
Victoria Jacobs, Analyst
 Alan Levine, Program Specialist 
Nora Leibowitz, Analyst

Vince Greiber, New York Regional Office

To obtain copies of this report, please call the Chicago Regional Office at (312) 353-9867. 
Reports are also available on the World Wide Web at our home page address: 
http://www.dhhs.gov/progorg/oei  E X E C U T I V E   S U M M A R Y  
PURPOSE 
To examine the evaluation systems used by Title I Ryan White grantees to monitor the 
compliance, measure the outcomes and measure the overall effectiveness of the programs 
they fund. 
BACKGROUND 
In 1990, Congress passed the Ryan White Comprehensive AIDS Resources Emergency

(CARE) Act as a comprehensive response to the HIV epidemic and its impact on

individuals, families, communities, cities and States.  The Ryan White CARE Act was re-

authorized in fiscal year (FY) 1996 through FY 2000.  Title I of the Act provides

emergency relief grants to eligible metropolitan areas (EMAs) disproportionately affected

by the epidemic to provide community-based HIV-related health and support services.  In

FY 1999, 51 EMAs in 21 States, Puerto Rico and the District of Columbia received $486

million in Title I formula and supplemental funds.

The Ryan White CARE Act will be considered for its second re-authorization in 2000. 

As Congress debates this re-authorization, questions are certain to be asked about the

effectiveness and impact of the programs funded.  This inspection examines the capacity of

Title I Ryan White grantees to answer those questions.

We conducted an examination of the evaluation mechanisms used by 12 EMAs through an

extensive documentation review and on-site interviews.  The 12 EMAs visited represent

the majority of Ryan White Title I funds and nearly half of HIV/AIDS cases. 

FINDINGS 
Title I Grantees Have Mechanisms in Place to Adequately Monitor Their 
Sub-grantees Activities 
The 12 Title I grantees we visited receive frequent program and fiscal reports from their 
sub-grantees, regularly review their financial audits and conduct routine site visits.  A 
majority of the grantees are tightening fiscal control through unit cost and fee-for-service 
contracting and tightening programmatic control through EMA-wide databases and the 
collection of client-level data.  Ensuring sub-grantees use Ryan White funds as the payor 
of last resort is one area where further monitoring mechanisms may be necessary. 
1 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  Few Grantees Currently Require Outcome Measurement of Their Sub-grantees; 
Most Are in the Planning Stage of Outcome Measurement 
Currently, only 2 of the 12 grantees visited require sub-grantees to measure client 
outcomes.  The other 10 Title I grantees visited are in various stages of preparing to 
require sub-grantees to collect and report outcome measures in the next few years.  The 
majority of sample Title I grantees have concerns regarding outcome measures and desire 
more specific guidance from the Health Resources and Services Administration (HRSA) 
on how to proceed. 
The Majority of Title I Grantees Visited Assess the Needs of Sub-populations in 
HIV/AIDS Care but Do Not Solicit Needs Information from Those Outside of 
HIV/AIDS Care 
In addition to analysis of public health data, information on client needs is often collected 
through provider and consumer surveys, focus groups and public forums.  These 
mechanisms focus on populations currently in HIV/AIDS care.  To detect the needs of 
those not in HIV/AIDS care, the majority of the Title I grantees visited match the 
demographics of HIV/AIDS cases to the demographic data of the clients receiving Title I 
funded services in their eligible metropolitan area (EMA).  Matches of epidemiological 
and utilization data do not provide a complete picture of the needs of persons not in 
HIV/AIDS care.  The matches do not provide information on the barriers individuals face 
to care or their unique service needs. 
RECOMMENDATIONS 
The HRSA Should Promote the Development of Metropolitan-wide Data Systems 
to Further Improve the Program Management Activities of the Sub-grantees 
The HRSA should provide grantees with technical assistance on how to develop 
EMA-wide data systems through which the grantees and sub-grantees can access 
client-level data.  Sub-grantee access to client-level data eliminates inefficient intake 
duplication and facilitates service coordination.  Data systems capacity is a vital ingredient 
for HIV/AIDS service agencies to improve the efficiency and effectiveness with which 
they provide their services, track their outcomes and inform their fund sources of their 
activities and results.  There are several potential sources of funds which could be used to 
assist grantees in building client-level data systems. 
The HRSA Should Implement a Multi-year Outcomes Initiative Including Grantee 
Requirements and Technical Assistance 
The HRSA should build upon their efforts to promote grantee outcome measurement by 
establishing a multi-year implementation plan.  Outcome measurement should start on the 
2 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  sub-grantee level and lead through subsequent steps to grantee analysis of client outcomes 
across the EMA.  Outcome measurement would foster program improvements and greater 
accountability for expenditures and highlight the value of Ryan White funds. 
The HRSA Should Provide Technical Assistance to Improve the Ability of 
Grantees to Estimate the Needs of Infected Persons Not in HIV/AIDS Care 
The HRSA currently requires grantees to estimate unmet need of persons not in 
HIV/AIDS care by comparing service utilization data to the HIV epidemic in their area. 
The HRSA should require the grantees to improve their knowledge of unmet need by 
soliciting information directly from persons not in HIV/AIDS care.  The input of infected 
persons who are not receiving care would add a richness of understanding regarding their 
needs and the barriers to service they encounter. 
Grantee solicitation of the input of persons un-connected to care would assist the grantees 
in complying with the re-authorized CARE Acts directive to prioritize a portion of funds 
for emerging populations.  Grantee solicitation of this input would also contribute to the 
Departments efforts to address the greater burden of HIV/AIDS on racial and ethnic 
minorities. 
Several grantees indicated a need for technical assistance on using outreach strategies to 
solicit information from persons not in HIV/AIDS care.  The HRSA could provide the 
grantees with a technical assistance document detailing how to use outreach strategies 
with examples of effective strategies for replication.  Particular attention should also be 
paid to reaching persons who receive HIV/AIDS services but may not be receiving 
primary health care services. 
AGENCY COMMENTS 
The HRSA provided comments to the draft report.  While HRSA concurred with the 
reports findings and recommendations, they offered suggestions for clarifying the report 
and making other technical changes.  Where appropriate, we changed the report to reflect 
their comments.  The complete text of HRSAs comments can be found in Appendix A. 
The HRSA pointed out several ongoing and planned activities in response to the report 
recommendation to promote the development of metropolitan-wide data systems.  We 
commend their efforts in this area.  The HRSA cautioned, however, that they do not have 
the financial resources to enable grantees to fully develop and implement these systems. 
We recommend that HRSA, in partnership with the Department, secure funds to assist in 
the development of such data systems.  The FY 2000 re-authorization of the CARE Act 
3 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  provides an optimal opportunity to consider funding for infrastructure development vital 
to ensuring efficient and effective service delivery in the future.  The grantees could build 
upon an initial infusion of funds to streamline their systems, potentially reducing their 
administrative costs and freeing up funds to maintain their systems over time. 
We agree with HRSA that the participation of non-Ryan White public health agencies is 
necessary to provide additional resources, skills and data to develop client-level data 
systems which capture the full array of services used by HIV-infected persons.  To this 
end, we recommend that HRSA build upon the HRSA-Health Care Financing 
Administration (HCFA)- Centers for Disease Control and Prevention (CDC) inter-agency 
data sharing agreement to promote the development of integrated public health data 
systems. 
The HRSA briefly described a few activities planned or underway in support of our 
recommendation to implement a multi-year outcomes initiative.  We commend HRSA on 
these activities and plans.  Despite their efforts in this area, HRSA expressed a few 
concerns regarding their ability to carry out this recommendation: 1) data limitations, 
2) funding limitations,  and 3) confidentiality concerns. 
We do not expect the outcomes initiative to lead to rigorous scientific evaluations 
conducted by every grantee.  By examining the client-level impact of the services they 
fund, the grantees can assess in general terms the effectiveness of these services. 
Although we expect the costs involved to be minimized by emphasizing the use of data 
resources already available to the providers, many grantees may need additional financial 
resources to develop an outcomes system.  The metropolitan-wide client-level data 
systems discussed in our first recommendation would develop a sound infrastructure upon 
which to build the outcome measures initiative.  We recommend that HRSA work with the 
Department to secure funds for these initiatives critical to the future of the Ryan White 
CARE Act programs. 
The HRSAs third area of concern - assuring client confidentiality, can be addressed by 
using unique identifiers and staff confidentiality agreements. 
4 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  T A B L E   O F   C O N T E N T S  
EXECUTIVE SUMMARY  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  1

INTRODUCTION  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  6

FINDINGS

Adequate Monitoring Mechanisms in Place  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  10

Few Require Outcomes  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  12

Solicit Information on Needs from Clients, Not Others . . . . . . . . . . . . . . . . . . . . . . . . .  14

RECOMMENDATIONS  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  15

AGENCY COMMENTS  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  21

APPENDICES

A: Agency Comments  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  23

B: Related Office of Inspector General Reports  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  31

5 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  I N T R O D U C T I O N  
PURPOSE 
To examine the evaluation systems used by Title I Ryan White grantees to monitor the 
compliance, measure the outcomes and measure the overall effectiveness of the programs 
they fund. 
BACKGROUND 
This study complements our reports, Ryan White Evaluation Systems, Title II: Grants to 
States (OEI-05-98-00393) and Ryan White Evaluation Systems, AIDS Drug Assistance 
Program (OEI-05-98-00390). 
The Ryan White CARE Act 
In 1990, Congress passed the Ryan White Comprehensive AIDS Resources Emergency 
(CARE) Act as a response to the HIV epidemic and its impact on individuals, families, 
communities, cities and States.  The Ryan White programs provide health and support 
services to persons with HIV/AIDS who would otherwise not have access to care.  The 
Ryan White programs constitute slightly over $1 billion of the $6.8 billion in HIV/AIDS 
expenditures in the U.S. Department of Health and Human Services fiscal year (FY) 1998 
budget. 
The Ryan White CARE Act was re-authorized in FY 1996 through FY 2000.  Changes to 
the Act included a directive to Title I planning councils to consider cost and outcomes in 
funding allocation decisions to the extent that such data are reasonably available. 
The Ryan White CARE Act's four titles and Part F direct resources to various entities and 
allow grantees maximum flexibility in the use of funds.  In FY 1998, nearly 90 percent of 
the Acts funds were awarded to Title I and Title II grantees. 
Title I of the Ryan White CARE Act 
Title I provides emergency relief grants to eligible metropolitan areas (EMAs) 
disproportionately affected by the HIV epidemic. Title I grants are for community-based 
HIV-related outpatient and ambulatory health and support services, including case 
management, comprehensive treatment services, prescription drugs, counseling, 
transportation, meals-on-wheels programs, home care and hospice care. 
6 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  Grantees are required to establish a planning council that reflects the demographics of the 
epidemic in the EMA and is representative of the range of individuals involved in HIV 
care including health and social service providers, government entities, planning agencies, 
community leaders and other Federal HIV program grantees. The planning council 
develops a comprehensive plan for delivering HIV services in the EMA, prioritizes the 
allocation of funds in the EMA, assesses the grantees fund allocation mechanisms and 
evaluates the effectiveness of services in meeting identified need. 
Metropolitan areas are eligible for Title I formula funding if they have a population of 
500,000 or more and have a total of more than 2,000 cases of AIDS reported to the U.S. 
Center for Disease Control (CDC) for the most recent period of 5 calendar years.  In FY 
1999, there were 51 EMAs in 21 States, Puerto Rico and the District of Columbia.  In FY 
1999, EMAs received $486 million in Title I formula and supplemental funds. 
OIG Evaluation Recommendations in 1995 
In a June 1995 Office of Inspector General (OIG) report entitled The Ryan White CARE 
Act: Local Implementation Issues(OEI-05-93-00336),  the OIG found an absence of 
program outcome evaluation to measure the CARE Acts impact.  Beyond surveys of 
client satisfaction undertaken by providers at the local level, grantees and planning 
councils were not assessing the impact of CARE Act funds on their service area. 
The OIG recommended that the Health Resources and Services Administration (HRSA) 
and the Assistant Secretary for Planning and Evaluation (ASPE) fund appropriate studies 
to assess the overall effectiveness of the Ryan White programs.  The OIG also 
recommended that HRSA ensure grantees evaluate their own programs.  The OIG 
suggested local evaluations include provider compliance reviews, outcome measure 
studies and overall program effectiveness reviews.  The OIG asked HRSA to provide 
grantees with technical assistance and evaluation instruments that guarantee consistency in 
data collection and evaluation. 
The ASPE did not conduct any evaluations of the Ryan White program with HRSA as 
recommended in the 1995 OIG report.  However, ASPE is currently funding the HIV 
Costs and Services Utilization Study (HCSUS) with HRSA and the Agency for Health 
Care Policy and Research (AHCPR).  The HCSUS examines who is treated for HIV, the 
health services HIV patients receive, the costs of the services and means of payment.  The 
ASPE and AHCPR are also sponsoring a related study on the impact of health care 
financing arrangements on access to and use of new and emerging HIV therapies. 
The HIV/AIDS Bureau of HRSA has a national Ryan White evaluation agenda which 
includes client-level data studies, links with secondary data sets, topic specific evaluations 
and the aggregate reporting system.  The Bureau is focusing on evaluating the following 
areas: 1) Enrolling under-served, vulnerable populations; 2) Providing clients with care 
equal to that received by the insured; 3) Providing services that remove barriers to 
7 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  receiving primary care; 4) Reducing morbidity, including opportunistic infections and 
perinatal transmission, and maintaining or increasing CD4 levels; 5) Reducing mortality; 
and 6) Adapting to a changing service and cost environment. 
HRSAs Current Evaluation Requirements 
The FY 1997 and 1998 Grant Application Guidances asked grantees to specify outcomes 
for four service categories, including primary medical care.  The FY 1999 Guidance asks 
grantees to describe their processes and results in developing outcomes.  Grantees are also 
instructed to describe the impact of Title I funds. Suggested outcomes to use include: 
providing access to care whose quality meets or exceeds Public Health Service and other 
standards, removal of barriers to care, and morbidity reductions. 
Title I grantees are required to conduct a needs assessment of HIV-infected populations in 
their service area and describe its use in funding decisions. The FY 1998 Application 
Guidance instructed applicants for supplemental funds to estimate unmet need for: persons 
with HIV by level of illness or transmission category, race/ethnicity, and sex; or enhanced 
primary care access.  Applicants were also instructed to estimate the number of persons 
with HIV without access to new treatments and/or requisite primary health care. 
The FY 1999 Application Guidance asks the grantees for more information on unmet need 
to be obtained through comparisons of HIV prevalence and service utilization data.  The 
Guidance emphasizes the importance of documenting unmet need and indicates future 
HRSA plans to work with the EMAs in this endeavor. 
HRSAs Technical Assistance on Evaluation 
The HRSA has provided technical assistance to Title I grantees on conducting evaluations 
through a variety of means including: self assessment protocols, meetings to share 
evaluation strategies, and a series of monograph reports on evaluation and assessment. 
The most recent report emphasizes the importance of outcome measurement and provides 
some guidance to grantees on how to approach it.  The report does not suggest outcomes 
to measure or steps to develop outcome measures.  More recently HRSA has suggested 
grantees focus on primary health care outcomes and will soon disseminate a publication 
focusing on primary health care outcome measures. 
Re-authorization in 2000 
The Ryan White CARE Act will be considered for its second re-authorization in 2000. 

As Congress debates this re-authorization, questions are certain to be asked about the

effectiveness and impact of the programs funded.  This inspection examines the capacity of

Title I Ryan White grantees to answer those questions.

8 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  SCOPE AND METHODOLOGY 
In this inspection we examined: the mechanisms Title I grantees use to monitor sub-
grantee program and financial compliance, whether and how the grantees measure the 
outcomes of the programs they fund, and whether and how the grantees measure their 
effectiveness in meeting the evolving needs of the affected population.  This evaluation 
does not judge the compliance, measure the outcomes or assess the effectiveness of the 
Title I grantees, but rather judges whether Title I grantees have and appropriately utilize 
the tools to perform these evaluation functions. We also did not review HRSAs 
evaluation of the grantees. 
For the purposes of this analysis, we define compliance as serving the population, 
providing the services and spending the resources as agreed to in return for the award of 
program funds.  Outcomes are defined as measures of the benefits clients derived as a 
result of the program.  Finally, we define overall effectiveness as how well the mix of 
services meets the evolving needs of the population affected by the epidemic. 
We conducted an on-site examination of these evaluation mechanisms in 12 EMAs: New 
York City, Newark, Bergen-Passaic, Baltimore, Washington D.C., Miami, Orlando, 
Austin, Houston, Chicago, Los Angeles and Sacramento.  We selected these EMAs as a 
purposive sample representative of: comparatively high rates of incidence through 
intravenous drug use (IDU) transmission, comparatively high rates of infections among 
minority populations, early and recent EMA designation and regional diversity.  In 
addition, our selected EMAs represent the majority of Ryan White Title I funds and nearly 
half of the HIV/AIDS cases.  These 12 EMAs contain 38 percent of the living AIDS cases 
reported in 1997 and 51 percent of the Title I funding for FY 1998. 
We conducted an extensive documentation review in the 12 EMAs.  We examined their 
FY 1998 Title I grant applications, the contracts of the sample sub-grantees, samples of all 
program, fiscal, outcome and needs assessment reports, site visit protocols and additional 
evaluation materials provided by grantees, planning councils and sub-grantees. 
We supplemented the documentation review with on-site interviews.  In each EMA we 
interviewed; (1) the Title I grantee, (2) a representative of the Title I planning council, and 
(3) at least 10 percent of the Title I sub-grantees.  The number of sub-grantees visited in 
each EMA ranged from 3 to 14.  In each EMA we interviewed sub-grantees of differing 
program size, service delivery purpose and location. 
All of our data collection was conducted from October through December 1998. 
We conducted our review in accordance with the Quality Standards for Inspections issued 
by the Presidents Council on Integrity and Efficiency. 
9 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  F I N D I N G S  
Title I grantees have mechanisms in place to adequately 
monitor their sub-grantees activities 
Grantees receive regular program and fiscal reports and conduct routine site 
visits 
Ten of the 12 Title I grantees visited require their sub-grantees to submit monthly 
program and fiscal reports.  The other two Title I grantees require quarterly reports. 
Program reports typically contain units of service provided, number and demographics of 
clients served and a narrative section on program accomplishments and barriers.  Fiscal 
reports typically contain line item expenditures or detailed billing invoices for unit cost and 
fee-for-service reimbursement.  Most grantees also require fiscal reports to disclose year-
to-date expenditures and the unexpended balance.  All of the grantees visited report that 
they review annual fiscal audits from their sub-grantees. 
Nine of the 12 sample grantees conduct formal site visits to all of their sub-grantees at 
least once a year.  The Baltimore grantee visits each sub-grantee every 2 years.  The Los 
Angeles and Austin grantees use a risk assessment model to determine which sub-grantees 
to visit.  In all of the sample EMAs, site visits to sub-grantees include review of a sample 
of client files, financial records and the supporting documentation used for reporting. 
Many of the grantees also informally monitor sub-grantees through ongoing 
communication and individualized technical assistance. 
Grantees move toward unit cost reimbursement to tighten fiscal accountability 
Seven of the 12 grantees we visited reimburse the sub-grantees using a unit cost or fee-
for-service method for all or some of their service categories.  One additional grantee is 
planning to use the unit cost method in FY 1999.  The other three grantees reimburse their 
sub-grantees on a line item basis. 
The grantees we visited use unit cost reimbursement to tighten their control over costs, to 
inform their resource allocation, to encourage sub-grantee efficiency and to measure sub-
grantee performance.  For example, in Newark the sub-grantees propose a unit cost for 
each service in their grant application.  The proposed cost is considered in procurement 
decisions and is subject to contract negotiations.  According to a respondent, this helps the 
grantee to be an informed purchaser of services and has led to lower service costs. 
10 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  Many grantees use metropolitan-wide databases and client-level data for program 
management 
Nine of the 12 grantees we visited use or are implementing an EMA-wide database, 
through which all of the sub-grantees enter client level data on a uniform system.  In seven 
of the eligible metropolitan areas, the grantee can access the agency data at any time 
directly from the system, rather than via down-loads or a contract administrator.  In six of 
the EMAs, the sub-grantees have access to one anothers data.  An agency can look up a 
clients identification number and learn what services the client has received elsewhere. 
This enables the agencies to track clients through the Title I system of care, improving 
client management and reducing service duplication. 
In the EMAs we visited, we noted three implementation concerns regarding unified data 
systems.  A few respondents reported that it has been difficult to include hospitals in the 
establishment of a common system since hospitals have distinct computer system needs.  A 
second concern expressed by respondents is the data delay caused by a shortage of 
computer terminals.  Many agencies with only one computer terminal linked to the EMA 
system enter data in weekly or monthly batches.  Delays in data entry curtail the benefits 
of agency to agency client management.  A third concern is that where EMA data systems 
are incompatible with Title II systems or reporting requirements, agencies funded by both 
Title I and Title II programs may have to collect and report data through dual systems. 
Client Management in Newark: The Comprehensive HIV/AIDS Management Program System 
(CHAMPS): 
Newark Title I sub-grantees use CHAMPS to track client care, to manage administrative 
functions, to report program and fiscal data to the grantee and to aggregate client outcomes. 
Through the CHAMPS database all sub-grantees and the grantee can access client-level data 
at any time.  Information is entered through a unique client identification number.  Sub-
grantees can view a clients demographic and health profile and review what services the client 
has received through other sub-grantees. Multiple fields of information are available for each 
client, including CD4 & T-cell count, usual source of primary care, and referred services.  The 
planning council uses CHAMPS to facilitate its needs assessment and planning work.  For 
example, the council has used CHAMPS data to review client residency, service location, and 
the peak months and hours of service utilization to forecast service use. 
Grantees try to ensure that Ryan White funds are used as the payor of last resort; 
Most sub-grantees ensure payor of last resort status 
All of the grantees convey HRSAs rules regarding payor of last resort status to the sub-
grantees in their contracts and/or policy and procedure manuals.  Some grantees require 
11 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  sub-grantees to report all sources of revenue to verify the use of Ryan White funds as the 
payor of last resort.  In addition, all of the grantees we visited check that sub-grantees are 
documenting Ryan White eligibility in client files during site visits. 
Most of the sub-grantees we visited have mechanisms in place to ensure that they use 
Ryan White funds as the payor of last resort, such as documenting Medicaid and private 
insurance coverage in client files and billing accordingly.  However, a few of the ancillary 
service providers we visited appeared unaware of the rules or their responsibility in this 
area.  A few also appeared to be unclear on the services paid for by Medicaid in their 
State.  In addition, sub-grantees have varying capacity to check Medicaid eligibility. 
Few grantees currently require outcome measurement of 
sub-grantees; most are in the planning stage of outcome 
measurement 
Only 2 of the 12 grantees visited require sub-grantees to measure outcomes 
The Newark and Sacramento Title I grantees require sub-grantees to measure the 
outcomes of the services they are contracted to provide.  However, sub-grantee interviews 
in these EMAs indicate that some sub-grantees do not routinely measure outcomes.  The 
Newark and Sacramento grantees are taking steps to increase sub-grantee outcome 
measurement through greater standardization and increased reporting requirements. 
Starting in FY 1998, the Sacramento grantee required sub-grantees to report outcome 
measures on a quarterly basis.  Due to a low level of sub-grantee compliance in the initial 
6 months, the grantee developed a standard format for outcome reporting.  As of FY 
1999, outcome measures are a required component of the sub-grantee quarterly report. 
The Newark grantee requires all sub-grantees to outline expected outcomes and their 
means to measure the outcomes in a logic model in their annual grant proposal.  The sub-
grantees are required to include the following elements in their logic model: 1) service 
gaps and consumer needs, 2) activities to address the needs, 3) outcomes defined as short-
term consumer benefits, 4) measures to quantify the outcomes, and 5) expected long-term 
impacts on the system or community.  The logic model is subject to contract negotiations. 
Sub-grantees are not currently required to report outcome measures to the Newark 
grantee.  However, the grantee requires the sub-grantees to measure the outcomes in their 
logic model and to conduct an internal evaluation documenting their progress.  On site 
visits the grantee reviews the sub-grantees internal evaluation reports.  Sub-grantee 
outcome measurement, documentation and achievement is scored and used as criteria in 
determining annual contract awards to continuing applicants. 
12 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  The other 10 Title I grantees visited are preparing to require sub-grantees to 
collect outcome measures in the next few years 
The remaining 10 grantees visited reported plans to require sub-grantees to measure client 
outcomes, however the grantees are at varying levels of planning and implementation.  Six 
of the 10 grantees are currently developing outcome measures, half of them are using a 
consultant to draft suggested outcomes.  The other four grantees are laying the 
groundwork for outcomes by developing standards of care and data collection 
mechanisms.  Seven of the 10 grantees expect to require some or all providers to report 
outcomes by FY 2000. 
Most of the planning councils in the 12 sample eligible metropolitan areas (EMAs) are 
involved in the outcome measure efforts underway in their EMA. In six of the EMAs, the 
councils are involved in the actual determination of the outcome measures.  In other 
EMAs, council involvement includes activities such as establishing broad outcome areas, 
participating in the development of service standards and reviewing suggested outcomes. 
Most of the grantees solicited sub-grantee input in developing outcomes.  A few of the 
grantees cited the United Ways manual on outcome measurement as the primary source 
of guidance they have relied upon to direct their efforts in this area.  The United Way 
approach emphasizes developing outcomes reflective of an agencys goals and 
recommends using the data sources readily available to agencies, such as client records, 
client and staff interviews and mechanical tests and measurements. 
Many of the sub-grantees we visited collect client information which could be used to 
measure program outcomes if aggregated.  Sub-grantees often document client progress in 
a clients chart or case-notes.  Most of the sub-grantees we visited do not aggregate the 
client outcomes and are not able to compare outcomes across client groups, service 
strategies or over time.  Nor do they report outcomes to the grantee or other funders. 
Majority of sampled Title I grantees have concerns regarding outcome measures 
Most of the Title I grantees we visited expressed concerns regarding outcome measure 
requirements.  The primary area of concern cited is the lack of financial resources and the 
administrative spending cap.  Other concerns include: the burden of additional reporting 
requirements on sub-grantees; a lack of sub-grantee and planning council expertise in 
collection, analysis and use of outcome measures; and the difficulty of isolating the 
influence of Ryan White services on client outcomes.  A couple of grantees also expressed 
a hesitancy to advance outcome measurement efforts in the absence of more specific 
guidance from HRSA out of concern that local efforts may conflict with future 
requirements.  All of the Title I grantees visited said that they would like more guidance 
from HRSA on how to proceed with establishing and collecting outcome measures. 
13 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  The majority of Title I grantees visited assess the needs of 
sub-populations in HIV/AIDS care but do not solicit needs 
information from those outside of HIV/AIDS care 
The grantees collect information on service and sub-population needs through annual or 
bi-annual comprehensive needs assessments or targeted assessments of client populations 
and service categories.  All of the grantees use both qualitative and quantitative data 
sources in their needs assessment work, including public health data, consumer and 
provider surveys, focus groups, public forums and key informant interviews.  Outside of 
the public health data, information on the needs of infected individuals not receiving 
HIV/AIDS care is usually not obtained. 
The majority of Title I grantees visited match the demographics of HIV cases to 
demographic data on the clients receiving Title I services to determine unmet need.  These 
data matches do not provide information on the barriers individuals face to care or the 
unique service needs they may have beyond those implied through co-morbidity data. 
Furthermore, in the three EMAs visited which do not have unduplicated client counts, the 
service utilization data most likely count individual clients more than once, inflating the 
numbers of persons appearing to receive care.  Hence, a higher proportion of the 
HIV/AIDS population is considered receiving care than do in reality. 
Only a few of the grantees have gone beyond data matches to obtain information on the 
needs of persons who are not in HIV/AIDS care.  One planning council tried to solicit this 
information by sending trained peer outreach workers to homeless shelters, without much 
success. Two other councils were more successful in obtaining information from persons 
not in HIV/AIDS care, one through street outreach and the State substance abuse 
program, and the other through focus groups of current and former prison inmates. 
Assessing Unmet Need in New York City: The CHAIN Study 
Since 1995, the New York City planning council has contracted with the Columbia University 
School of Public Health to conduct a longitudinal study of persons living with HIV/AIDS 
(PLWH/A).  Through the Community Health Advisory Information Network (CHAIN) study, 
the planning council collects data on a sample of 700 PLWHA, systematically drawn from 
over 40 agencies in New York City, including agencies which receive Ryan White funding and 
those which do not. The sample includes infected persons who are un-connected to care.  The 
CHAIN study collects information on client demographics, risk factors, family and social 
supports, barriers to care, environmental impacts and client outcomes. Through the CHAIN 
study, the council has identified unmet needs of persons in and out of care. 
14 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  R E C O M M E N D A T I O N S  
The HRSA should promote the development of 
Metropolitan-wide data systems to further improve the 
program management activities of the sub-grantees 
Building on the client-level data demonstration projects underway,1  HRSA should provide 
grantees with technical assistance to facilitate the development of eligible metropolitan 
area-wide data systems.  The sophistication and use of current EMA data systems vary 
widely by grantee.  The HRSA should aid grantee efforts to build or improve unified data 
systems by disseminating a variety of technical information including information on model 
systems, effective system development strategies and specific elements to include to 
maximize benefits.  This information should incorporate the lessons learned through the 
client-level data demonstration projects. 
Sub-grantee access to client-level data eliminates inefficient intake duplication and 
facilitates service coordination by informing providers of a clients existing referrals and 
services received.  Confidentiality concerns can be overcome using staff confidentiality 
agreements and unique identification numbers, rather than client name, to track clients. 
Sub-grantees can also use EMA-wide data systems to generate routine program and fiscal 
reports, reducing the time and resources needed by sub-grantees for reporting. 
In addition, sub-grantees can use the systems to track and report outcomes.  One concern 
we heard from several grantees regarding outcome measurement is that they do not want 
to increase the reporting burden on the sub-grantees.  If outcome fields are integrated into 
the EMAs data systems as they are built, sub-grantees can track, aggregate and report 
client-level outcome measures through the existing system.  Toward this end, HRSA is 
developing new software for the Annual Administrative Report (AAR) that will include a 
voluntary module on health-related client-level information. 
The development of client-level data systems requires a substantial financial investment. 
Based on the client-level data demonstration projects, HRSA estimates that it costs 
between $100,000 and $500,000 in start-up costs for each grantee to build a client level 
data system and between $50,000 and $335,000/year for each grantee to maintain the 
1	 Under the client level data demonstration project, four EMAs and three States have been 
collecting and analyzing client-level data since 1994. Collected data includes client 
characteristics, health indicators and services received. The HRSA plans to expand the client 
level data set in 1999. 
15 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  system.  Currently, grantees are restricted in the amount of funds they can invest in 
infrastructure by an administrative cap and by competing demands to fund services. 
Recognizing the current limitations on funds available for data collection and evaluation in 
the Ryan White CARE Act, HRSA should work in partnership with the Department to 
secure funds to facilitate the development of needed infrastructure in the CARE Act 
program. 
There are several potential sources of funds which could be used, either alone or in 
combination, to assist grantees in building client-level data systems including: 1) the one 
percent set-aside of program funds reserved for evaluation of programs authorized under 
the Public Health Service Act; 2) an appropriation specifically designated for data system 
development, similar to the funding allocated for such development within the child 
support enforcement program; and 3) a designation of funds for evaluation of the Ryan 
White programs included in the CARE Acts FY 2000 re-authorization, carved outside of 
the administrative spending cap. 
Grantees could build on an initial infusion of funds, received through one or more of the 
above means, to streamline their tracking and reporting systems, potentially reducing their 
administrative costs over the long term.  A reduction in administrative costs achieved 
through more efficient tracking and reporting processes could free up funds to support the 
improved data collection systems in the future. 
As HIV/AIDS has moved from an emergency epidemic to a chronic epidemic, it is critical 
for service providers to move from a crisis mode of operations to more long-term capacity 
building.  Data systems are a vital ingredient for HIV/AIDS service agencies to improve 
the efficiency and effectiveness with which they provide their services, track their 
outcomes and inform their funding sources of their activities and results.  Promoting the 
development of these systems is one practical step the administration can take in its 
initiative to strengthen the capacity of small, community-based organizations to serve the 
emerging HIV/AIDS population.  Agencies with the most skill in serving the needs of 
disenfranchised populations often lack the resources for necessary infrastructure. 
In their data systems promotion and technical assistance efforts, the HRSA should urge 
Title I grantees to work with the other grantees in their State to develop a shared system 
so as to reduce multiple system requirements on sub-grantees receiving funding through 
more than one Ryan White Title.  Recent HRSA efforts to develop a cross-Title AAR 
should facilitate the development of shared data systems. 
In addition, HRSA should encourage the CARE Act grantees to reach out to other public 
health programs serving persons with HIV/AIDS in their State to share services data.  The 
1998 inter-agency agreement between HRSA, HCFA and CDC to promote the 
development of integrated public health information systems and the inter-departmental 
activities supporting this agreement are intended to foster such collaborations. 
16 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  The HRSA should implement a multi-year outcomes initiative 
including grantee requirements and technical assistance 
The HRSA should build upon their efforts to promote grantee outcome measurement by 
establishing a multi-year implementation plan.  Outcome measurement should start on the 
sub-grantee level and lead through subsequent steps to grantee analysis of client outcomes 
across the EMA. Outcome measurement would foster program improvements and greater 
accountability for expenditures and highlight the value of Ryan White funds. 
Here is one option of how HRSA could proceed: 
<	 In stage one, HRSA could develop one or more core outcomes for each service 
category and require Title I grantees to develop three additional outcomes for each 
service category in collaboration with Title II Consortia where they are co-located. 
The core outcomes HRSA requires the grantees to measure should be aligned with 
HRSAs performance goals outlined in their annual performance plan. 
<	 In stage two, HRSA could require grantees to collect the established outcome 
measures from the providers on a quarterly basis.  Outcome measures should 
eventually be reported as client-level data in preparation for stage three. 
<	 In stage three, HRSA could require the grantees to analyze the reported outcome 
measures across services and report EMA-wide outcomes to HRSA annually. 
In establishing the core outcomes, HRSA should place a greater emphasis on primary 
health care outcomes since the ultimate goal of the Ryan White CARE Act is improved 
health status.  However, at least one core outcome should be developed for each support 
service category.  The outcomes for the support services should reflect the contribution 
the support services make to improved health status such as facilitating access to primary 
care or creating conditions conducive to following and benefitting from a structured health 
care regimen.  Measuring these incremental outcomes would provide valuable information 
on the contribution of these services to health outcomes.  This information could be used 
for accountability and program improvement purposes.  Confidentiality concerns can be 
overcome using staff confidentiality agreements and unique identification numbers, rather 
than client name, to track clients. 
The outcome measurement initiative can serve as a means for HRSA to further promote 
continuous quality improvement (CQI) efforts.  Through CQI, information on the 
effectiveness and quality of program delivery is fed back to program managers on a 
continuous basis, serving as a tool for managers to improve program delivery.  While 
quality assurance and process indicators are used to gauge the quality of services 
delivered, outcome measures can be used to gauge service effectiveness.  Service 
providers could use outcome measures along with quality of care indicators to improve 
their performance through program alterations and further self-evaluation under CQI. 
As the Ryan White CARE Act faces its second decade of providing care to persons 
17 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  infected with HIV/AIDS, it is time to ensure that the grantees have the capacity to 
evaluate the outcomes of the services they deliver.  Outcome evaluation as a CQI tool is 
vital to strengthening the effectiveness of the services delivered and the efficiency of the 
funds expended to provide the services.  This basic need for appropriate funding should be 
addressed in the re-authorization of the CARE Act in FY 2000. 
The HRSA should provide the grantees with technical assistance to implement the core 
outcome measures and to develop and implement their own additional outcome measures 
within a CQI framework.  The technical assistance could emphasize soliciting provider 
input in the development of the additional outcomes, using the United Way approach. 
The United Way approach to outcome measurement emphasizes using outcome measures 
for program improvement purposes.  To this end, the United Way emphasizes soliciting 
provider input in the development of outcome measures to make sure that outcome 
measures reflect program goals.  The United Way also stresses the importance of 
considering the data resources already available to the providers when developing 
outcome measures so as not to impose too much additional administrative or cost burden. 
The United Ways approach recognizes the limited capacity of health and human service 
providers to engage in rigorous outcome evaluation.  Hence, the United Way model 
emphasizes contribution rather than attribution.  Using outcome measures, providers can 
determine if clients achieve an expected result after receiving their service intervention. 
While the provider is not able to scientifically prove that the result is due solely to their 
intervention, they can see whether their service is contributing to positive outcomes on a 
regular basis. 
The HRSA should urge Title I grantees to work with co-located Title II consortia in 
establishing the outcome measures sub-grantees will be required to measure, so as to 
avoid imposing a greater measurement burden on sub-grantees funded under both Titles. 
The HRSA should also urge the grantees to require sub-grantees to report outcome 
measures as client-level and aggregate data.  Grantees will need client-level outcome data 
in order to analyze client outcomes across the EMA.  The HRSA should urge the grantees 
to consider this need in the creation or redesign of unified data systems. 
In recent years, HRSAs application guidance to Title I grantees has stressed the 
importance of documenting outcomes and has asked grantees to include outcome data in 
their applications.  All of our respondents have received the message from HRSA that 
outcome measures are important and that HRSA expects them to move ahead in this area. 
However, respondents in the 12 EMAs we visited stated that they would like more 
specificity from HRSA on what outcomes they are expected to measure and on HRSAs 
implementation plans. 
Likewise, in response to a July 1998 survey sent out by HRSAs technical assistance 
contractor, 81 percent of the 37 responding EMAs indicated that clear written guidance 
18 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  from the HIV/AIDS Bureau on outcome evaluation expectations or requirements would 
be among the most useful  types of technical assistance they could receive.  Seventy-
eight percent of the responding EMAs indicated that lists of suggested outcome 
indicators for other (non-primary care) service categories would be among the most 
useful assistance. 
The HRSA should provide technical assistance to improve 
the ability of grantees to estimate the needs of infected 
persons not in HIV/AIDS care 
In the FY 1999 Application Guidance, HRSA emphasizes the importance of obtaining 
information on unmet need, acknowledges the difficulty of obtaining this information and 
pledges to work with EMAs to address this area in the future.  The HRSA should build on 
this effort by requiring the grantees to solicit needs information from persons not in 
HIV/AIDS care and by providing the grantees with technical assistance on how to do this. 
The public health data grantees are currently required to use to estimate unmet need 
provides useful information on the needs of infected persons.  However, the input of 
persons outside of the care system would add a richness of understanding regarding their 
needs and the barriers to service they encounter. 
Grantee solicitation of the input of persons un-connected to care would assist the grantees 
in complying with the re-authorized CARE Acts directive to prioritize a portion of funds 
for emerging populations.  Grantee solicitation of this input would also contribute to the 
Departments efforts to address the greater burden of HIV/AIDS on racial and ethnic 
minorities. 
In their attempt to fulfill the CARE Acts requirement to prioritize funds for emerging 
populations, HRSA should urge the planning councils to strive to reach persons 
unconnected to HIV/AIDS care, in addition to efforts to reach persons who 
demographically reflect the emerging populations.  The HRSA should urge the councils to 
recognize the complexity and disproportionate costs which may be involved in serving 
persons previously unconnected to HIV/AIDS care.  Persons infected by HIV/AIDS are 
increasingly individuals who are traditionally not connected to the health care system for 
reasons such as poverty and historic mistrust of institutional health care.  A focused 
outreach and planning strategy is required to address their needs. 
The HRSAs technical assistance materials on conducting needs assessments urge grantees 
to solicit information from persons not in HIV/AIDS care.  Although the materials suggest 
avenues for soliciting this information, they do not include any information on how to use 
these means.  Several grantees indicated that they would like more guidance on how to 
proceed in this area. 
19 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  The HRSA could provide the grantees with a technical assistance document devoted to 
soliciting needs information from PLWH/A who are not in HIV/AIDS care,  providing 
steps for conducting outreach and examples of effective strategies.  The HRSA could 
highlight outreach strategies which have been used by other Ryan White grantees, CDC 
HIV prevention community planning groups and service providers in other arenas. 
Particular attention should also be paid to reaching persons who receive HIV/AIDS 
services but may not be receiving primary health care services. 
20 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  A G E N C Y   C O M M E N T S  
The HRSA provided comments to the draft report.  While HRSA concurred with the 
reports findings and recommendations, they offered suggestions for clarifying the report 
and making other technical changes.  Where appropriate, we changed the report to reflect 
their comments.  The complete text of HRSAs comments can be found in Appendix A. 
The HRSA pointed out several ongoing and planned activities in response to the report 
recommendation to promote the development of metropolitan-wide data systems.  We 
commend their efforts to assist the grantees in the development of these data systems. 
The revisions to the Annual Administrative Report (AAR) to allow for cross-Title 
reporting and the addition of the voluntary module on health-related client-level 
information in the AAR software will provide the grantees with added flexibility to track 
client-level information across CARE Act Titles.  We also commend HRSA on their 
plans to host a FY 1999 meeting of grantees with expertise in client-level reporting to 
launch a peer-based technical assistance effort. 
In response to our recommendation that HRSA provide technical assistance and financial 
resources to support the development of EMA-wide data systems, HRSA cautioned that it 
would be unreasonable to expect technical assistance and financial resources from 
HRSAs HIV/AIDS Bureau to be sufficient to fully develop and implement those systems 
in an appropriate manner.  The HRSA has made a great start in the provision of technical 
assistance through building expertise via the seven client-level data demonstration sites 
and the planned FY 1999 meeting to launch the peer-based technical assistance effort. 
These endeavors will facilitate an exchange of information on model systems and lessons 
learned, providing a knowledge base to support local system development efforts.  We 
urge HRSA to continue down this path. 
We do not expect HRSA to provide all of the financial resources necessary to fully 
develop and implement metropolitan-wide data systems.  We recommend that HRSA, in 
partnership with the Department, secure funds to assist in the development of such data 
systems.  The FY 2000 re-authorization of the CARE Act provides an optimal opportunity 
to consider funding for infrastructure development vital to ensuring efficient and effective 
service delivery in the future.  The grantees could build upon an initial infusion of funds to 
streamline their systems, potentially reducing their administrative costs and freeing up 
funds to maintain their systems over time. 
Furthermore, we agree with HRSA that the participation of non-Ryan White public health 
agencies is necessary to provide additional administrative resources, skills and data to 
develop client-level data systems which capture the full array of services used by HIV-
infected persons.  To this end, we recommend that HRSA build upon the HRSA-HCFA-
CDC inter-agency data sharing agreement to promote the development of integrated 
21 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  public health data systems by encouraging Ryan White CARE Act grantees to work with 
the other public health programs in their State serving persons with HIV/AIDS. 
The HRSA briefly described a few activities planned or underway in support of our 
recommendation to implement a multi-year outcomes initiative.  We commend HRSA on 
these activities and plans.  Despite their efforts in this area, HRSA expressed a few 
concerns regarding their ability to carry out this recommendation: 1) limitations on 
accessibility and compatibility of data sets, 2) restrictions on administrative costs 
impacting a Grantees ability to use complex data sets for outcome studies, and 
3) confidentiality concerns. 
These limitations are to be expected in a large endeavor but are not insurmountable.  For 
this reason we recommend an outcomes approach designed with feasibility as a 
cornerstone.  We do not expect the outcomes initiative to lead to rigorous scientific 
evaluations conducted by every grantee.  The HRSA is funding such projects on a limited 
basis where the capacity exists.  It is not feasible, nor is it necessary, for every grantee to 
participate in this work.  What is feasible and necessary for every grantee to undertake is 
an assessment of the effectiveness of the services they fund.  By examining the 
contributions made by the services to client outcomes, the grantees can assess, in general 
terms, the effectiveness of these services.  We highlight the United Way approach which 
emphasizes contribution, rather than attribution, for this reason.  The United Way 
approach is designed for service providers, like those in the Ryan White program, who 
face data and cost limitations. 
Although we expect the costs involved to be minimized by emphasizing the use of data 
resources already available to the providers, many grantees may need additional financial 
resources to develop an outcomes system, depending upon the size of their grant and 
competing demands on limited administrative funds.  The metropolitan-wide client-level 
data systems discussed in our first recommendation, would develop a sound infrastructure 
upon which to build the outcome measures initiative.  We recommend that HRSA work 
with the Department to secure funds for these initiatives critical to the future of the Ryan 
White CARE Act programs. 
The HRSAs third area of concern - assuring client confidentiality, can be addressed by 
using unique identifiers and staff confidentiality agreements.  Grantees collecting client-
level information have used these measures successfully to address confidentiality 
concerns. 
We commend HRSAs efforts to address our recommendation to improve the ability of 
grantees to estimate the needs of infected persons not in HIV/AIDS care.  We urge HRSA 
to disseminate technical assistance information to grantees in this area as soon as possible, 
as the population of infected persons outside of care continues to grow. 
22 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  A P P E N D I X  A  
Agency Comments 

23 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  A P P E N D I X  A  
Agency Comments 

24 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  A P P E N D I X  A  
Agency Comments 

25 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  A P P E N D I X  A  
Agency Comments 

26 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  A P P E N D I X  A  
Agency Comments 

27 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  A P P E N D I X  A  
Agency Comments 

28 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  A P P E N D I X  A  
Agency Comments 

29 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  A P P E N D I X  A  
Agency Comments 

30 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392  A P P E N D I X  B  
Related Office of Inspector General Reports

The Ryan White CARE Act: Local Implementation Issues (OEI-05-93-00336) 
The Ryan White CARE Act: Examples of Local Coordination (OEI-05-93-00335) 
The Ryan White CARE Act: Special Projects of National Significance (OEI-05-93-00332) 
The Ryan White CARE Act: Technical Report of 1992 Expenditures (OEI-05-93-00334) 
The Ryan White CARE Act: Funding Formulas (OEI-05-93-00330) 
Medicaid Managed Care and HIV/AIDS (OEI-05-97-00210) 
Audit of State AIDS Drug Assistance Programs Use of Drug Price Discounts (A-01-97-01501) 
Review of Maximizing Drug Discounts Under the Missouri Ryan White Program 
(A-01-97-00926) 
Audit of the Ryan White Comprehensive AIDS Resource Emergency Act of 1990, Title II, 
Administered by the Health Resources and Services Administration (A-01-97-01500) 
Audit of Eligibility Under Title I of the Ryan White Comprehensive AIDS Resource Emergency 
Act of 1990, New York Metropolitan Area (A-02-95-02517) 
Audit of the Ryan White Comprehensive AIDS Resource Emergency Act of 1990, Title II, 
Administered by the State of Connecticut (A-01-96-01501) 
Audit of the Ryan White Comprehensive AIDS Resource Emergency Act of 1990 in the Boston 
Metropolitan Area for Fiscal Year 1994 (A-01-95-01504) 
31 
Ryan White Evaluation Systems: Title I  OEI-05-98-00392