Department of Health and Human Services 
OFFICE OF 
INSPECTOR GENERAL 
RYAN WHITE EVALUATION SYSTEMS 
Title II: Grants to the States 
JUNE GIBBS BROWN 
Inspector General 
AUGUST 1999 
OEI-05-98-00393  OFFICE OF INSPECTOR GENERAL 
The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, is to 
protect the integrity of the Department of Health and Human Services programs as well as the 
health and welfare of beneficiaries served by them.  This statutory mission is carried out through a 
nationwide program of audits, investigations, inspections, sanctions, and fraud alerts.  The 
Inspector General informs the Secretary of program and management problems and recommends 
legislative, regulatory, and operational approaches to correct them. 
Office of Evaluation and Inspections 
The Office of Evaluation and Inspections (OEI) is one of several components of the Office of 
Inspector General.  It conducts short-term management and program evaluations (called 
inspections) that focus on issues of concern to the Department, the Congress, and the public.  The 
inspection reports provide findings and recommendations on the efficiency, vulnerability, and 
effectiveness of departmental programs. 
OEI's Chicago Regional Office prepared this report under the direction of William Moran, 
Regional Inspector General.  Principal OEI staff included: 
REGION  HEADQUARTERS 
William Moran, Project Leader
 Elise Stein, Program Specialist 
Ann Maxwell, Lead Analyst
 Rada Spencer, Program Specialist 
Victoria Jacobs, Analyst
 Alan Levine, Program Specialist 
Nora Leibowitz, Analyst

Vince Greiber, New York Regional Office

To obtain copies of this report, please call the Chicago Regional Office at (312) 353-9867. 
Reports are also available on the World Wide Web at our home page address: 
http://www.dhhs.gov/progorg/oei  E X E C U T I V E   S U M M A R Y  
PURPOSE 
To examine the evaluation systems used by Title II Ryan White grantees to monitor the 
compliance, measure the outcomes and measure the overall effectiveness of the programs 
they fund. 
BACKGROUND 
In 1990, Congress passed the Ryan White Comprehensive AIDS Resources Emergency 
(CARE) Act as a comprehensive response to the HIV epidemic and its impact on 
individuals, families, communities, cities, and States.  The Ryan White CARE Act was re-
authorized in fiscal year (FY) 1996 through FY 2000.  Title II of the Act provides grants 
to States to improve the quality, availability and organization of health care and support 
services for individuals and families with HIV disease.  In FY 1999, the States received 
$710 million in Title II funds.  Most grantees disburse a portion of these funds to service 
providers through regional fiscal agents called lead agencies and planning bodies called 
consortia. 
The Ryan White CARE Act will be considered for its second re-authorization in 2000.  As 
Congress debates this re-authorization, questions are certain to be asked about the 
effectiveness and impact of the programs funded.  This inspection examines the capacity of 
Title II Ryan White grantees to answer those questions. 
Through an extensive documentation review and on-site interviews, we examined the 
evaluation mechanisms used in 8 States, 12 consortia/lead agencies and 5 consortia 
responsible for planning only.  The eight States visited represent 61 percent of FY 1998 
Ryan White Title II funds and 68 percent of reported AIDS cases in FY 1997. 
FINDINGS 
In Most Sampled States, Program and Fiscal Compliance of Service Providers Is 
Monitored Through Fiscal Intermediaries; Intermediaries Have Monitoring 
Mechanisms in Place 
In the eight States we visited, seven of the grantees use regionally based lead agencies to 
serve as fiscal agents for the service providers.  In six of the States, the lead agencies have 
primary responsibility for monitoring the program and fiscal compliance of the service 
providers.  The lead agencies receive regular program and fiscal reports from the 
providers and conduct frequent site visits to providers.  Likewise, the grantees receive 
1 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  regular program and fiscal reports from the lead agencies and conduct periodic site visits 
to them.  Most of the grantees and lead agencies have mechanisms in place to ensure that 
Ryan White funds are used only for eligible clients and as the payor of last resort. 
Few Grantees or Lead Agencies Currently Require Outcome Measurement of 
Their Service Providers; Most Are in the Early Stages of Discussion Regarding 
Outcome Measures 
Currently, only 1 of the 8 State grantees and 1 of the 12 lead agencies we visited, require 
client outcomes to be measured.  The majority of grantees and lead agencies are in the 
early stages of planning for outcome measurement.  Concerns regarding outcomes 
measures and a desire for technical assistance from the Health Resources and Services 
Administration (HRSA) were expressed by over half of the grantees and lead agencies. 
Grantees Rely on Consortia to Assess Needs; Most Consortia Routinely Assess 
the Needs of Clients but Not of Persons Outside of HIV/AIDS Care 
All of the State grantees we visited rely on regional consortia to assess the needs of 
persons with HIV/AIDS and require them to do so on a routine basis.  Slightly over half 
of the grantees require the consortia to submit the needs assessment results as part of the 
consortiums annual application for funds. 
Most of the consortia or local planning bodies use both qualitative and quantitative data 
sources in their needs assessment work, including public health data, consumer and 
provider surveys, focus groups and public forums.  Outside of the public health data, 
information on the needs of infected individuals not receiving HIV/AIDS care is typically 
not obtained.  Of the 17 planning bodies we interviewed, only 3 reported that they have 
successfully solicited needs information from persons with HIV/AIDS not in care. 
RECOMMENDATIONS 
The HRSA Should Implement a Multi-year Outcomes Initiative 
The HRSA should build upon their efforts to promote outcome measurement by 
establishing a multi-year implementation plan.  Outcome measurement should start on the 
provider level and lead through subsequent steps to consortia-wide analysis of client 
outcomes by the responsible fiscal agent.  Outcome measurement would foster program 
improvements and greater accountability for expenditures and promote the achievements 
of the Ryan White programs. 
2 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  The HRSA Should Provide Technical Assistance to Improve the Ability of 
Grantees and Consortia to Estimate the Needs of Infected Persons Not in 
HIV/AIDS Care 
The HRSA currently requires grantees to estimate unmet need of persons not in 
HIV/AIDS care by comparing service utilization data to the HIV epidemic in their area. 
The HRSA should require the grantees and consortia to improve their knowledge of 
unmet need by soliciting information directly from persons not in HIV/AIDS care. 
Grantee solicitation of the input of persons unconnected to HIV/AIDS care would assist 
the grantees in complying with the re-authorized CARE Acts directive to prioritize a 
portion of funds for emerging populations.  Grantee solicitation of this input would also 
contribute to the Departments efforts to address the greater burden of HIV/AIDS on 
racial and ethnic minorities. 
Several respondents indicated a need for technical assistance on using outreach strategies 
to solicit information from persons not in HIV/AIDS care.  The HRSA could provide the 
grantees and consortia with a technical assistance document detailing effective outreach 
strategies.  Particular attention should also be paid to reaching persons who receive 
HIV/AIDS services but may not be receiving primary health care services. 
AGENCY COMMENTS 
While HRSA concurred with the findings and recommendations in our draft report, they 
offered suggestions for clarifying the report. Where appropriate, we changed the report to 
reflect their comments.  The HRSAs complete comments can be found in Appendix B. 
The HRSA briefly described a few activities in support of our recommendation to 
implement a multi-year outcomes initiative.  We commend HRSA on these activities and 
plans.  Despite their efforts in this area, HRSA expressed a few concerns regarding data 
limitations, funding limitations, and confidentiality concerns. 
We do not expect the outcomes initiative to lead to rigorous scientific evaluations 
conducted by every grantee.  By examining the client-level impact of the services they 
fund, the grantees can assess in general terms the effectiveness of these services. 
Although we expect the costs involved to be minimized by emphasizing the use of data 
resources already available to the providers, many grantees may need additional financial 
resources to develop an outcomes system.  We recommend that HRSA work with the 
Department to secure funds for these initiatives.  The FY 2000 re-authorization of the 
CARE Act provides an optimal opportunity to consider funding for infrastructure 
development vital to ensuring efficient and effective service delivery in the future. 
The HRSAs third area of concern - assuring client confidentiality, can be addressed by 
using unique identifiers and staff confidentiality agreements. 
3 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  T A B L E   O F   C O N T E N T S  
EXECUTIVE SUMMARY  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  1

INTRODUCTION  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  5

FINDINGS

Fiscal Intermediaries Perform Monitoring  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  9

Few Require Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  11

Adequately Assessing Client Needs, Not Those Outside of Care  . . . . . . . . . . . . . . .  14

RECOMMENDATIONS  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  16

AGENCY COMMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  20

APPENDICES 
A: Title II Administrative Structure by State  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

B: Agency Comments  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

C: Related Office of Inspector General Reports  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  I N T R O D U C T I O N  
PURPOSE 
To examine the evaluation systems used by Title II Ryan White grantees to monitor the 
compliance, measure the outcomes and measure the overall effectiveness of the programs 
they fund. 
BACKGROUND 
This study complements our reports entitled, Ryan White Evaluation Systems, Title I: 
Grants to Metropolitan Areas (OEI-05-98-00392) and Ryan White Evaluation Systems, 
AIDS Drug Assistance Programs (OEI-05-98-00390). 
The Ryan White CARE Act 
In 1990, Congress passed the Ryan White Comprehensive AIDS Resources Emergency 
(CARE) Act as a response to the HIV epidemic and its impact on individuals, families, 
communities, cities and States.  The Ryan White programs provide health and support 
services to persons with HIV/AIDS who would otherwise not have access to care.  The 
Ryan White programs constitute slightly over $1 billion of the $6.8 billion in HIV/AIDS 
expenditures in the U.S. Department of Health and Human Services Fiscal Year (FY) 
1998 budget. 
The Ryan White CARE Act was re-authorized in FY 1996 through FY 2000.  Changes to 
the Act included a directive to Title II grantees to prioritize a portion of their funds for 
emerging populations. 
The Health Resources and Services Administration (HRSA) directs Ryan White resources 
to various local entities through the Act's four titles and Part F and allows grantees 
maximum flexibility in the use of funds.  In FY 1998, nearly 90 percent of the Acts funds 
were awarded to Title I and Title II grantees. 
Title II of the Ryan White CARE Act 
Title II provides grants to States to improve the quality, availability and organization of 
health care and support services for individuals and families with HIV disease.  Grants are 
awarded to States in accordance with the number of living AIDS cases in the State or 
territory and the estimated number of living AIDS cases outside of Title I funded eligible 
metropolitan areas.  In addition to the base award, States receive earmarked funds for the 
5 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  AIDS Drug Assistance Programs (ADAPs) through which States provide medications to 
low-income individuals with HIV disease with no or limited insurance coverage. 
Outside of the earmarked funds, States can use Title II funds to support home and 
community based health and support services, continuation of health insurance coverage, 
pharmaceutical treatments through the ADAP, HIV care consortia, and direct health and 
support services.  This inspection focuses on evaluation mechanisms within the consortia 
program, as this program receives the largest share of Title II non-ADAP funds. 
Consortia are community-based regional planning entities established by Title II grantees. 
The consortia plan and prioritize Title II funds in their area, promote coordination of 
services and serve as a community forum.  Representatives of local public and non-profit 
health and support service providers serve as consortium members.  Incorporated 
consortium may serve as the local fiscal agent for Title II funds.  In other cases, 
consortium designate a member agency to be the lead agency,  responsible for grants 
administration and provider monitoring of Title II funds in the region.  In some areas, the 
lead agency is the same agency that is responsible for the administration of Title I funds. 
(See Appendix A for a description of the consortia/lead agency structure in visited States) 
OIG Evaluation Recommendations in 1995 
In a June 1995 Office of Inspector General (OIG) report, The Ryan White CARE Act: 
Local Implementation Issues(OEI-05-93-00336),  the OIG found an absence of program 
outcome evaluation to measure the CARE Acts impact.  Beyond surveys of client 
satisfaction undertaken by providers at the local level, grantees and consortia were not 
assessing the impact of CARE Act funds on their service area. 
The OIG recommended that the Health Resources and Services Administration and the 
Assistant Secretary for Planning and Evaluation (ASPE) fund appropriate studies to assess 
the overall effectiveness of the Ryan White programs.  The OIG also recommended that 
HRSA ensure grantees evaluate their own programs.  The OIG suggested local 
evaluations include provider compliance reviews, outcome measure studies and overall 
program effectiveness reviews.  The OIG asked HRSA to provide grantees with technical 
assistance and evaluation instruments that guarantee consistency in data collection and 
evaluation. 
The ASPE did not conduct any evaluations of the Ryan White program as recommended 
in the 1995 OIG report.  However, ASPE is currently funding the HIV Costs and Services 
Utilization Study (HCSUS) with HRSA and the Agency for Health Care Policy and 
Research (AHCPR).  The HCSUS examines who is treated for HIV, the health services 
HIV patients receive, the costs of the services and means of payment.  The ASPE and 
AHCPR are also sponsoring a related study on the impact of health care financing 
arrangements on access to and use of new and emerging HIV therapies. 
6 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  The Office of Science and Epidemiology, within the HIV/AIDS Bureau of HRSA, has a 
national Ryan White evaluation agenda which includes client-level data studies, links with 
secondary data sets, topic specific evaluations and the aggregate reporting system.  The 
office is focusing on evaluating the following areas: 1) Enrolling under-served populations; 
2) Providing clients with care equal to that received by the insured; 
3) Providing services that remove barriers to receiving primary care; 4) Reducing 
morbidity, including opportunistic infections and perinatal transmission, and maintaining or 
increasing CD4 levels; 5) Reducing mortality; and 6) Adapting to a changing service and 
cost environment. 
HRSAs Current Evaluation Requirements 
In recent years, HRSAs Grant Application Guidances have placed a growing emphasis on 
the ability of grantees to document and quantify the impact of the services they fund.  The 
FY 1999 application guidance asks grantees to begin developing outcome measures in 
relation to four program principles: responding to the needs of the underserved and hard-
to-serve, assuring access to emerging therapies, adapting to changes in the health care 
market, and documenting client-level outcomes.  Suggested client-level outcomes include: 
decreased viral load, declines in perinatal transmission of HIV, and morbidity and 
mortality reductions.  In the absence of defined outcome measures, the grantee is asked to 
describe activities they will undertake to measure impact in these areas. 
The FY 1999 Application Guidance instructs applicants to estimate unmet need for those 
in care and those not currently in the system of care in terms of their demographics, 
geographic variation across the State and their unmet need.  Applicants are also instructed 
to provide quantitative data on co-morbidities, poverty status and insurance status of the 
States living HIV/AIDS cases. 
The FY 1999 Application Guidance asks the grantees for more information on unmet need 
to be obtained through comparisons of HIV prevalence and service utilization data.  The 
Guidance emphasizes the importance of documenting unmet need and indicates future 
HRSA plans to work with the States in this endeavor. 
HRSAs Technical Assistance on Evaluation 
The HRSA has provided technical assistance to Title II grantees on conducting 
evaluations  through a variety of means including: self assessment protocols, meetings to 
share evaluation strategies, and a series of monograph reports on evaluation and 
assessment.  The most recent report emphasizes the importance of outcome measurement 
and provides some guidance to grantees on how to approach it.  The report does not 
suggest outcomes to measure or steps to develop outcome measures.  More recently, 
HRSA has suggested grantees focus on primary health care outcomes and will soon 
disseminate a publication focusing on primary health care outcome measures. 
7 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  Re-authorization in 2000 
The Ryan White CARE Act will be considered for its second re-authorization in 2000.  As 
Congress debates this re-authorization, questions are certain to be asked about the 
effectiveness and impact of the programs funded.  This inspection examines the capacity of 
Title II Ryan White grantees to answer those questions. 
SCOPE AND METHODOLOGY 
In this inspection, we examined the mechanisms Title II grantees use to monitor 
sub-grantee program and fiscal compliance, measure the outcomes of the programs they 
fund and assess their effectiveness in meeting the evolving needs of the affected 
population.  This evaluation does not judge the compliance, measure the outcomes or 
assess the effectiveness of the Title II grantees, but rather judges whether the grantees 
have and appropriately utilize the tools to perform these evaluation functions.  We did not 
review HRSAs evaluation of the grantees. 
For the purposes of this analysis, we define compliance as serving the population, 
providing the services and spending the resources as agreed to in return for the award of 
program funds.  Outcomes are defined as measures of the benefits clients derive as a result 
of the program.  Finally, we define overall effectiveness as how well the mix of services 
meets the evolving needs of the population affected by the epidemic. 
We conducted an on-site examination of these evaluation mechanisms in eight States: New 
York, New Jersey, Maryland, Florida, Illinois, Texas, California and the District of 
Columbia.  We selected these States as a purposive sample representative of 
comparatively high rates of incidence through intravenous drug use transmission, 
comparatively high rates of infections among minority populations, and regional diversity. 
In addition, our selected States contain 68 percent of the living AIDS cases reported in 
1997 and 61 percent of the Title II funding for FY 1998. 
We conducted an extensive review of documentation provided by the grantees, consortia 
and providers.  We examined States FY 1998 Title II grant applications, service provider 
contracts, samples of all program, fiscal, outcome and needs assessment reports, site visit 
protocols and additional evaluation materials. 
We supplemented the documentation review with on-site interviews from October through 
December 1998.  In each State we interviewed: the Title II grantee, the Title II 
consortia/lead agency in up to three different regions, and at least 10 percent of the Title II 
providers.  The number of  providers interviewed ranged from 4 to 15.  They were chosen 
to represent various program sizes, service delivery purposes and locations. 
We conducted our review in accordance with the Quality Standards for Inspections issued 
by the Presidents Council on Integrity and Efficiency. 
8 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  F I N D I N G S  
In most sampled States, program and fiscal compliance of 
service providers is monitored through fiscal intermediaries; 
intermediaries have monitoring mechanisms in place 
In the eight States we visited, seven of the grantees use regionally based lead agencies to 
serve as fiscal agents for the service providers.  The grantees monitor the lead agencies 
and the lead agencies monitor the providers.  The lead agencies determine funding awards 
in accordance with a service plan developed by the regional consortium or planning body. 
In six of the States, the lead agencies have primary responsibility for monitoring the 
program and fiscal compliance of the service providers.  In two of these six States, the 
grantees keep abreast of individual provider performance by reviewing the reports 
providers submit to the lead agency or by accompanying the lead agency on site visits.  In 
the other four States, the grantees leave all provider monitoring responsibilities to the lead 
agencies. 
Grantees receive regular program and fiscal reports from lead agencies and most 
conduct periodic site visits to lead agencies 
All of the grantees we visited receive monthly or quarterly program and fiscal reports from 
the lead agencies.  Program reports typically contain compiled program data, such as the 
units of service provided and the number and demographics of clients served, as well as a 
narrative section on program accomplishments and barriers.  Fiscal reports are typically 
expense vouchers or invoices of line item costs. 
Most of the grantees we visited reimburse the lead agencies based on a line item budget. 
A couple of the grantees reimburse lead agencies on a fee-for-service or unit cost basis. 
Some grantees advance the lead agencies a portion of their funds to facilitate the flow of 
the grant making process.  All but one of the grantees requires the lead agencies to have 
an independent audit performed.  The majority of grantees require the audits to be 
submitted for review on an annual basis. 
Most of the grantees conduct formal site visits to the lead agencies at least once every 2 
years.  Two of the grantees have not routinely visited lead agencies but plan to conduct 
routine visits in the future.  One of these grantees plans to visit lead agencies once every 
3 years. 
9 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  Lead agencies receive regular program and fiscal reports from providers and 
conduct frequent site visits to providers 
All of the lead agencies we visited receive monthly or quarterly program and fiscal reports 
from their service providers.  The program and fiscal reports contain similar information to 
that which the lead agencies report to the grantees.  While most of the grantees reimburse 
lead agencies based on a line item budget, the majority of lead agencies reimburse 
providers on a fee-for-service or unit cost basis or have plans to do so.  Nearly all of the 
lead agencies require and review annual audits of the providers. 
The majority of sampled lead agencies conduct formal site visits to service providers at 
least once a year and as often as quarterly.  Site visits typically include a review of a 
sample of client files, financial records and other support documentation used for 
reporting.  Most of the lead agencies also informally monitor providers through ongoing 
communication and individualized technical assistance. 
Most grantees and lead agencies have mechanisms to ensure that Ryan White 
funds are used only for eligible clients and as the payor of last resort 
All of the grantees and lead agencies have mechanisms in place to ensure that providers 
verify client eligibility for Ryan White services.  Six of the eight grantees and most of the 
lead agencies also have mechanisms to ensure that providers use Ryan White funding as 
the payor of last resort.  The grantees and lead agencies include eligibility and payor 
requirements in written policy and procedural manuals and check relevant documentation 
on site visits.  A few lead agencies do not routinely verify provider use of Ryan White 
funds as the payor of last resort.  These lead agencies indicated awareness of the need to 
devote more attention to this area in the future. 
Most of the service providers we visited have mechanisms in place to ensure that they use 
Ryan White funds as the payor of last resort, such as documenting Medicaid and private 
insurance coverage in client files and billing accordingly.  However, a few of the ancillary 
service providers we visited appeared unaware of the rules or their responsibility in this 
area.  A few also appeared to be unclear of the services paid for by Medicaid in their 
State.  In addition, service providers have varying capacity to check Medicaid eligibility. 
10 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  Few Grantees or Lead Agencies Currently Require Outcome 
Measurement of Their Service Providers; Most Are in the 
Early Stages of Discussion Regarding Outcome Measures 
Currently, Only One of the Eight State Grantees and One of the Twelve Lead 
Agencies We Visited, Require Client Outcomes to Be Measured 
The New York grantee currently requires outcome measures for three service categories, 
primary care, dentistry and case management.  Primary care and dentistry are measured 
according to professional clinical guidelines and expected clinical health outcomes.  An 
independent peer review organization monitors providers performance on a set of core 
indicators such as CD4 count and gynecological health.  These are tracked by a common 
software package that aggregates individual level clinical data entered to manage patient 
care.  The grantee is also piloting outcomes in case management in the following areas: 
optimal health, financial/entitlements, independent living, adult education/employment, 
family stability, mental health, substance abuse and client empowerment. 
The Sacramento County Department of Health and Human Services, the lead agency for 
Title II funds in the Sacramento area, requires providers to report outcome measures on a 
quarterly basis.  Since reporting compliance was low in the initial 6 months, the lead 
agency is developing a standard format for outcome reporting.  Outcome measures are a 
required component of the provider quarterly reports as of FY 1999. 
The rest of the grantees and lead agencies rely on process measures to measure the 
success of service providers.  These measures, along with annual goals, are usually 
outlined in the scope of work section of a providers contract.  The process measures 
typically consist of the number of clients served and the number of service units delivered 
in the contract year.  Some contracts require that the measures be broken down by sub-
population, with a special emphasis on women and children. 
A few providers currently report outcomes; many others have client outcomes in case files 
A few providers in several localities have developed aggregate outcomes to measure the 
impact of the services they provide.  These providers have taken the initiative due to the 
requirements of other funding sources or their own desire for greater program 
accountability.  Though they are not required to, these agencies typically submit their 
outcome information to their lead agencies as part of their quarterly or annual progress 
reports. 
The outcome measures developed by the Mount Carmel Guild in Mercer County New 
Jersey provide an example of the type of outcome measures some providers are using. The 
11 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  Mount Carmel Guild provides nutritional counseling, nutritional food packages and 
personal care packages for HIV infected clients.  They have a set of outcome measures 
that are tracked for each client and aggregated to evaluate overall program performance. 
The measures include weight loss, weight gain, weight stabilization, cholesterol level, 
albumin level and routine vitamin intake.  The nutritionist routinely tracks progress on 
these outcomes by client and aggregates them annually through a random audit of 50 
client charts.  The organization has plans underway to refine this process by analyzing 
outcomes across sub-populations. 
Many other providers we visited collect client information which could be used to measure 
program outcomes if aggregated.  For example, case managers often establish client goals 
at intake and document progress towards these goals in a clients case file.  Physicians also 
typically record clinical progress in patient charts.  In these situations, a clients progress 
or the progress of the caseload in general may be discussed at organizational meetings, but 
agencies do not aggregate the client outcomes and are not able to compare outcomes 
across client groups, service strategies or over time.  Nor do they report outcomes to the 
lead agency or grantee. 
The Majority of Grantees and Lead Agencies We Visited Are in the Early Stages of 
Planning for Outcome Measurement 
Most State grantees are in various stages of planning for outcome measures.  Two State 
grantees are discussing the development of outcome measures in the near future.  Both are 
planning on developing a core set of outcomes at the State level with provider input.  One 
has begun by introducing the concept of outcome measurement to its providers to solicit 
their feedback.  The other grantee is setting up a task force to focus on outcome 
measures.  This task force is soliciting information from other Ryan White programs and is 
searching for a consultant to help them devise a strategy to develop outcome measures in 
a participatory manner.  They are also considering contracting a local university to define 
the measures once they are selected. 
Other State grantees, while not yet engaged in outcomes discussions, are laying the 
groundwork for future outcome measurement.  A few are developing standards of care.  It 
is their belief that this standardization of service provision is necessary in order to hold 
providers accountable to a common set of outcome measures.  Several grantees are 
working to improve their databases which will facilitate the collection of outcome 
measures.  The improvements include projects such as developing the capacity to collect 
client-level data and devising standard data definitions to assure accurate and reliable 
reporting. 
Half of the lead agencies are also engaged in the development of outcome measures.  Half 
of these agencies indicated that they are actively engaged in putting outcome measures in 
place by the FY 1999 grant cycle.  The other half are in the very early stages of discussion 
surrounding the implementation of outcome measurement. 
12 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  Agencies actively pursuing outcome measures are doing so by formulating a core set of 
outcome measures.  One agency has, based on the United Way model1, met with all of the 
providers to assess the data they are collecting and their capacity to aggregate outcome 
measures with minimal additional burden.  This information will be used to create a set of 
outcomes for each service category.  Another lead agency, already receiving voluntary 
outcome measures from its providers, has compiled a list of all of the outcome measures 
reported and is in the process of selecting a core set of measures.  Another agency has 
hired a consultant to develop a set of quality indicators in consultation with staff and 
providers.  This agency plans to recommend a set of outcome measures and require that 
providers incorporate a minimum of three outcome measures per service category. 
Other lead agencies are aware of the management and accountability benefits of utilizing 
outcome measures, but are still figuring out their approach.  While they indicate plans to 
have outcomes in place by FY 2000, they have no specific plans underway. 
More than Half of the Grantees and Lead Agencies Expressed Concerns 
Regarding Outcomes Measures 
Grantees and lead agencies share a variety of concerns regarding the development of 
outcome measures.  Several respondents noted that the administrative cap limits the 
amount of money available to track outcomes.  Additional grant funds that can be set 
aside for outcome measurement would be in direct competition with direct service funds. 
There are also capacity concerns.  A number of grantees and lead agencies are concerned 
that they do not possess the technical expertise to develop outcome measures that would 
appropriately reflect the diversity of service delivery and which could be attributed solely 
to the infusion of Ryan White funds.  Respondents also questioned the ability of planning 
bodies to properly understand and utilize outcome measures.  Providers voiced similar 
concerns regarding outcome measures.  In particular, they felt that they did not have the 
resources or skills to properly develop or monitor client outcomes. 
Respondents also questioned whether the State and local levels are the appropriate place 
to develop and conduct outcome measurement.  Many respondents feel that outcome 
measures should be uniform across Ryan White Titles, the State and the country in order 
to reduce the reporting burden and to allow for meaningful comparisons between 
programs.  Some also feel that Ryan White outcome measures should relate to the 
outcomes measures of other Federal agencies that support HIV services, such as the 
Department of Housing and Urban Development, and the Public Health Service goals for 
the year 2000.  In addition, several respondents suggested that tracking outcomes at the 
national level would be more cost-effective.  Respondents indicated that with uniform 
measures HRSA could take advantage of sampling methodologies to reduce the amount of 
1	 The United Way approach emphasizes developing outcomes reflective of an agencys goals and 
recommends using data sources readily available to agencies such as client records and responses 
to routinely administered standardized tests and measurements. 
13 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  labor involved.  The HRSA could also tap into the capacity and resources that already 
exist in the medical research field. 
Majority of grantees desire technical assistance from HRSA on outcome 
measurement 
Grantees requested that HRSA clearly delineate and communicate the direction they are 
headed with regards to outcome measures.  Grantees asked for a definition of what is 
meant by outcome measures and what outcome measures are appropriate for specific 
service categories. 
Several grantees also want HRSA to offer guidance on how to design an inclusive process 
of developing outcomes, how to measure service outcomes and how to build 
organizational capacity to interpret and analyze outcomes.  They want answers to 
questions such as: How do you create indicators that provide the information you are 
after? How do you track outcome measures over time? How do you get the necessary 
information from the client or their provider? What about confidentiality issues?  The 
respondents suggested that the answers to these questions could be provided through 
workshops, dissemination of best practices and national conferences, designed to promote 
information exchange among the grantees. 
Grantees rely on consortia to assess needs; most consortia 
routinely assess the needs of clients but not of persons 
outside of HIV/AIDS care 
All of the State grantees we visited rely on regional consortia to assess the needs of 
persons with HIV/AIDS and require them to conduct periodic assessments.  Slightly over 
half of the grantees we visited require the consortia to submit needs assessment results as 
part of the consortiums annual application for funds.  A couple of grantees supplement 
the local needs assessments with limited Statewide assessments of specific issues or sub-
populations.  None of the grantees currently conduct comprehensive assessments of the 
needs of the HIV/AIDS population in their State.  However, in compliance with Federal 
law, all of the State grantees review local needs assessments to identify Statewide needs as 
part of the Statewide Coordinated Statement of Need (SCSN) process.2  Building on the 
momentum of the SCSN process,  two of the grantees are planning to conduct a 
Statewide needs assessment in FY 1999. 
2	 The 1996 CARE Act legislation requires all Ryan White grantees to participate in a Statewide 
Coordinated Statement of Need (SCSN) process to collaborate in identifying and addressing HIV 
care issues. The SCSN is to be based on existing information from local needs assessments. 
Starting in FY 1998 Title II grantees are required to submit a copy of the SCSN with their grant 
applications and update the SCSN every three years, at a minimum. 
14 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  Most grantees do not specify what to include in the needs assessment 
Most of the grantees we visited do not specify what the consortia must include in their 
needs assessment or provide a standard tool for conducting the needs assessment.  A few 
of the grantees provide some guidance to the consortia on how to conduct the needs 
assessment and on the types of information they are expected to collect.  For example, the 
Florida grantee specifies what types of entities and population groups the consortium 
should consult with in the assessment.  Lead agencies in Florida are also directed to 
include an epidemiological profile, a demographic survey, focus groups, a resource 
inventory and a profile of provider capacity.  Two of the State grantees are in the process 
of developing a standard needs assessment tool for use by their consortium. 
Most consortia do not assess the needs of infected persons not in HIV/AIDS care 
We interviewed 17 planning bodies responsible for conducting local Title II needs 
assessments, including 5 HIV/AIDS care networks in New York and several Title I 
planning councils who assess needs for both Title I and Title II services.  Most of the 
consortia or planning bodies use both qualitative and quantitative data sources in their 
needs assessment work, including public health data, consumer and provider surveys, 
focus groups and public forums. 
Outside of the public health data, information on the needs of infected individuals not 
receiving HIV care is usually not obtained.  Of the 17 planning bodies interviewed, only 3 
reported that they have successfully solicited needs information from persons with 
HIV/AIDS who are not in care.  A few others indicated that they may have obtained needs 
information from infected persons not in care through public forums, but these forums 
were not targeted to this population.  Two of the other needs assessment bodies tried to 
solicit information from persons not in HIV/AIDS care through advertising in local 
newspapers and through substance abuse facilities, but were reportedly unsuccessful. 
The three consortia successful in obtaining the input of persons not in HIV/AIDS care 
used peers and street outreach to solicit input.  Two of the consortia trained persons living 
with HIV to survey infected persons in non-HIV/AIDS provider locations such as shelters 
and community groups.  One care network in New York conducted focus groups in 
shooting galleries and crack houses to solicit information on needs from persons outside of 
the HIV/AIDS care system. 
15 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  R E C O M M E N D A T I O N S  
The HRSA should implement a multi-year outcomes initiative 
The HRSA should build upon their efforts to promote outcome measurement by 
establishing a multi-year implementation plan.  Outcome measurement should start on the 
provider level and lead through subsequent steps to consortia-wide analysis of client 
outcomes by the responsible lead agency.3  Outcome measurement would foster program 
improvements and greater accountability for expenditures and promote the achievements 
of the Ryan White program. 
Here is one option of how HRSA could proceed: 
<	 In stage one, HRSA could develop one or more core outcomes for each service 
category and require the grantees to direct the lead agencies of the Title II consortia 
to develop three additional outcomes for each service category.  The HRSAs core 
outcomes could be used for both Title II and Title I grantees and should be aligned 
with the performance goals outlined in HRSAs annual performance plan.  The 
HRSA should urge Title II lead agencies to work with co-located Title I grantees 
and planning bodies in establishing the consortia-level outcome measures, so as to 
avoid imposing a greater measurement burden on providers funded under both 
Titles. 
<	 In stage two, HRSA could require the grantees to direct the lead agencies of the 
consortia to collect the established outcome measures from the providers on a 
quarterly basis.  The HRSA should also urge the grantees and the lead agencies to 
require providers to report outcome measures as client-level data.  These agencies 
will need client-level outcome data in order to analyze client outcomes across the 
region in stage three. 
<	 In stage three, HRSA could require the grantees to direct the lead agencies of the 
consortia to analyze the reported outcome measures across services and report 
consortia-wide outcomes to HRSA via the grantees on an annual basis. 
In establishing the core outcomes, HRSA should place a greater emphasis on primary 
health care outcomes since the ultimate goal of the Ryan White CARE Act is improved 
3  In States where grantees, rather then regional consortia, have direct oversight over providers, the 
responsibility of developing, collecting and analyzing outcomes would fall to the State. 
16 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  health status.  However, at least one core outcome should be developed for each support 
service category.  The outcomes for the support services should reflect the contribution 
the support services make to improved health status such as facilitating access to primary 
care or creating conditions conducive to following and benefitting from a structured health 
care regimen.  Measuring these incremental outcomes would provide valuable information 
on the contribution of these services to health outcomes.  This information could be used 
for accountability and program improvement purposes.  Confidentiality concerns can be 
overcome using staff confidentiality agreements and unique identification numbers, rather 
than client name, to track clients. 
The outcome measurement initiative can serve as a means for HRSA to further promote 
continuous quality improvement (CQI) efforts.  Through CQI, information on the 
effectiveness and quality of program delivery is fed back to program managers on a 
continuous basis, serving as a tool for managers to improve program delivery.  While 
quality assurance and process indicators are used to gauge the quality of services 
delivered, outcome measures can be used to gauge service effectiveness.  The HIV/AIDS 
service providers could use outcome measures along with quality of care indicators to 
improve their performance through program alterations and further self-evaluation under 
CQI. 
As the Ryan White CARE Act faces its second decade of providing care to persons 
infected with HIV/AIDS, it is time to ensure that the grantees have the capacity to 
evaluate the outcomes of the services they deliver.  Outcome evaluation as a CQI tool is 
vital to strengthening the effectiveness of the services delivered and the efficiency of the 
funds expended to provide the services.  This basic need for appropriate funding should be 
addressed in the re-authorization of the CARE Act in FY 2000. 
The HRSA should provide the grantees with technical assistance to implement the core 
outcome measures and to develop and implement their own additional outcome measures 
within a CQI framework.  The technical assistance could emphasize soliciting provider 
input in the development of the additional outcomes, using the United Way approach. 
The United Way approach to outcome measurement emphasizes using outcome measures 
for program improvement purposes.  To this end, the United Way emphasizes soliciting 
provider input in the development of outcome measures to make sure that outcome 
measures reflect program goals.  The United Way also stresses the importance of 
considering the data resources already available to the providers when developing 
outcome measures so as not to impose too much additional administrative or cost burden. 
The United Ways approach recognizes the limited capacity of health and human service 
providers to engage in rigorous outcome evaluation.  Hence, the United Way model 
emphasizes contribution rather than attribution.  Using outcome measures, providers can 
determine if clients achieve an expected result after receiving their service intervention. 
While the provider is not able to scientifically prove that the result is due solely to their 
17 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  intervention, they can see whether their service is contributing to positive outcomes on a 
regular basis. 
In recent years, HRSAs application guidance to Title II grantees has stressed the 
importance of documenting outcomes and has asked grantees to include outcome data in 
their applications.  All of our respondents have received the message from HRSA that 
outcome measures are important and that HRSA expects them to move ahead in this area. 
However, respondents stated that they would like more specificity from HRSA on what 
outcomes they are expected to measure and on HRSAs implementation plans. 
Stronger leadership from HRSA on outcome measurement would help coordinate efforts 
that remain fragmented at various levels of the Title II program.  Further, outlining a 
multi-year initiative would provide grantees and the lead agencies with the necessary 
perspective to strategically plan for the future.  As HIV/AIDS has evolved from an 
emergency epidemic to a chronic disease in a tight health care market, it is critical that the 
agencies serving infected individuals build the capacity to provide care efficiently and 
effectively through the productive use of outcome and impact studies. 
The HRSA should provide technical assistance to improve 
the ability of grantees to estimate the needs of infected 
persons not in HIV/AIDS care 
In the FY 1999 Application Guidance, HRSA emphasizes the importance of obtaining 
information on unmet need, acknowledges the difficulty of obtaining this information and 
pledges to work with States to address this area in the future.  The HRSA should build on 
this effort by requiring the grantees and consortia to solicit needs information from 
persons not in HIV/AIDS care and by providing technical assistance on how to do this.  In 
turn, the grantees should provide the consortia and other Title II planning bodies with 
guidance on needs assessment expectations and on methods to obtain the required 
information. 
Currently, grantees are required to use public health data to estimate unmet need in their 
applications to HRSA.  This data provides useful information on the needs of infected 
persons.  However, the input of persons outside of the care system would add a richness 
of understanding regarding their needs and the barriers to service they encounter. 
Grantee solicitation of the input of persons un-connected to HIV/AIDS care would assist 
the grantees in complying with the re-authorized CARE acts directive to prioritize a 
portion of funds for emerging populations.  Grantee solicitation of this input would also 
contribute to the Departments efforts to address the greater burden of HIV/AIDS on 
racial and ethnic minorities. 
18 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  In their attempt to fulfill the CARE Acts requirement to prioritize funds for emerging 
populations, HRSA should urge the grantees to strive to reach persons unconnected to 
HIV/AIDS care, in addition to efforts to reach persons who demographically reflect the 
emerging populations.  The HRSA should urge the grantees to recognize the complexity 
and disproportionate costs which may be involved in serving persons unconnected to 
HIV/AIDS care.  Persons infected by HIV/AIDS are increasingly individuals who are 
traditionally not connected to the health care system for reasons such as poverty and 
historic mistrust of institutional health care.  A focused outreach and planning strategy is 
required to address their needs. 
The HRSAs technical assistance materials on conducting needs assessments urge grantees 
to solicit information from persons not in care.  The materials suggest avenues for 
soliciting this information, such as substance abuse clinics and homeless shelters. Outside 
of suggesting that consortia provide incentives like food vouchers or use the media to 
solicit participation, the materials do not describe specific outreach strategies.  As these 
ideas are discussed only generally, we suggest HRSA provide more detailed information 
on steps to solicit needs information.  Several consortia representatives indicated that they 
would like more guidance on how to reach infected persons not in care. 
The HRSA could provide the grantees and the consortia with a technical assistance 
document devoted to soliciting needs information from people living with HIV/AIDS who 
are not in HIV/AIDS care,  providing steps for conducting outreach and examples of 
effective strategies.  The HRSA could highlight outreach strategies which have been used 
by other Ryan White grantees, Centers for Disease Control and Prevention HIV 
prevention community planning groups and service providers in other arenas.  Particular 
attention should also be paid to reaching persons who receive HIV/AIDS services but may 
not be receiving primary health care services. 
19 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  A G E N C Y  C O M M E N T S  
The HRSA provided comments to the draft report.  While HRSA concurred with the 
reports findings and recommendations, they offered suggestions for clarifying the report 
and making other technical changes.  Where appropriate, we changed the report to reflect 
their comments.  The complete text of HRSAs comments can be found in Appendix B. 
The HRSA briefly described a few activities planned or underway in support of our 
recommendation to implement a multi-year outcomes initiative.  We commend HRSA on 
these activities and plans.  Despite their efforts in this area, HRSA expressed a few 
concerns regarding their ability to carry out this recommendation: 1) limitations on 
accessibility and compatibility of data sets, 2) restrictions on administrative costs 
impacting a Grantees ability to use complex data sets for outcome studies, and 
3) confidentiality concerns. 
These limitations are to be expected in a large endeavor but are not insurmountable.  For 
this reason we recommend an outcomes approach designed with feasibility as a 
cornerstone.  We do not expect the outcomes initiative to lead to rigorous scientific 
evaluations conducted by every grantee.  The HRSA is funding such projects on a limited 
basis where the capacity exists.  It is not feasible, nor is it necessary, for every grantee to 
participate in this work.  What is feasible and necessary for every grantee to undertake is 
an assessment of the effectiveness of the services they fund.  By examining the 
contributions made by the services to client outcomes, the grantees can assess, in general 
terms, the effectiveness of these services.  We highlight the United Way approach which 
emphasizes contribution, rather than attribution, for this reason. 
Although we expect the costs involved to be minimized by emphasizing the use of data 
resources already available to the providers, many grantees may need additional financial 
resources to develop an outcomes system, depending upon the size of their grant and 
competing demands on limited administrative funds.  We recommend that HRSA work 
with the Department to secure funds for these initiatives.  The FY 2000 re-authorization 
of the CARE Act provides an optimal opportunity to consider funding for infrastructure 
development vital to ensuring efficient and effective service delivery in the future. 
The HRSAs third area of concern - assuring client confidentiality, can be addressed by 
using unique identifiers and staff confidentiality agreements.  Grantees collecting client-
level information have used these measures successfully to address confidentiality 
concerns. 
We commend HRSAs efforts to address our recommendation to improve the ability of 
grantees to estimate the needs of infected persons not in HIV/AIDS care.  We urge HRSA 
to disseminate technical assistance information to grantees in this area as soon as possible, 
as the population of infected persons outside of care continues to grow. 
20 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  APPENDIX A 
Structure of Title II Programs

CALIFORNIA 
The California Department of Public Health is the Title II grantee agency.  Funds are 
dispersed through 37 regionally-based consortium.  Each consortium has a lead agency 
which is responsible for the fiscal and administrative functions of the Title II consortia 
program in their region. The State monitors the lead agencies and requires them to 
monitor the programmatic and fiscal compliance of the service providers they fund. 
In Eligible Metropolitan Areas, the lead agency is typically the same as the Title I grantee. 
For example, the Los Angeles County Department of Health Services receives Title I 
funds and also acts as the fiscal intermediary for the Title II consortium funds.  In these 
situations, a planning council that has merged with the local consortia is responsible for 
advising the lead agency regarding needs and service priorities.  In non-EMA areas, the 
lead agencies are counseled by a regional consortia as to the service priorities of that area. 
THE DISTRICT OF COLUMBIA 
The District of Columbia Department of Health is the Title II grantee. The Department of 
Health has designated the DC CARE Consortium to be responsible for conducting needs 
assessments, planning service delivery, promoting service coordination and monitoring the 
program and fiscal activities of the service providers.  The Department of Health maintains 
some oversight over provider-level activities by reviewing individual provider reports 
submitted to the consortium and by accompanying the consortium on some provider site 
visits. 
FLORIDA 
The Florida Department of Health is the Title II grantee agency.  The Department of 
Health has established 14 regionally based consortia to provide a continuum of care for 
HIV infected persons throughout the State.  The consortia plan and prioritize Title II 
funds in their area, promote coordination of services and serve as a community forum. 
Each consortium has a lead agency which is responsible for the fiscal and administrative 
functions for the Title II consortia program in their region.  In some consortia areas, the 
Department of Health serves as the lead agency for the consortium. In other areas, the 
21 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  APPENDIX A 
Structure of Title II Programs

local consortia designate a lead agency.  All of the lead agencies in the State act as the fiscal 
conduit and data coordinator for the service providers in their area. 
ILLINOIS 
The Illinois Department of Public Health HIV/AIDS Section administers the Title II 
program in Illinois.  The Department relies on 11 lead agencies to disperse the funds 
regionally and to monitor service providers.  The State monitors the lead agencies. 
Regional consortium plan and prioritize Title II funds in their area and promote 
coordination of services. 
MARYLAND 
The Maryland Department of Health and Mental Hygiene AIDS Administration is the Title 
II grantee agency.  There are now five lead agencies that disperse the Title II funds 
regionally in accordance with a consortia developed service priority and allocation plan. 
The State has oversight over the lead agencys funding decisions.  The State also retains 
control over monitoring the service providers.  Both the lead agencies and service 
providers are required to submit their routine program and fiscal reports to the State 
Title II staff. 
NEW YORK 
The New York State Department of Health is the designated Title II grantee.  The AIDS 
Institute of the Department of Health operates the Title II program.  The AIDS Institute 
contracts with 17 Consortia, 16 of which are regionally based and one which is a 
Statewide consortium for special populations.  The consortia, otherwise known as HIV 
care networks are responsible for conducting local needs assessments, identifying local 
service priorities and developing service delivery plans.  The AIDS Institute determines 
funding allocations and awards contracts directly to service providers, in accordance with 
the networks service delivery plans. The Institute maintains fiscal and administrative 
oversight of the service providers funded through the Title II program, rather than use 
regionally based lead agencies. 
22 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  APPENDIX A 
Structure of Title II Programs

NEW JERSEY 
The New Jersey Department of Health and Senior Services is the Title II grantee.  The 
Department allocates its Title II consortia funds to 10 consortia, 9 of which are regionally 
based and one of which is a Statewide network of hospital sites serving HIV-infected 
women and children.  New Jersey is in the process of phasing out the six regional 
consortia which are located in Title I EMAs.  The New Jersey consortia are responsible 
for planning and coordination activities.  Each consortium designates a lead agency to 
handle fiscal and administrative responsibilities for the region.  Although monitoring 
responsibilities primarily fall to the lead agencies, the State grantee maintains some 
oversight over the activities of local service providers.  The lead agencies determine 
contract awards but provider contracts are subject to the review of the grantee.  The 
grantee also conducts annual site visits to all service providers. 
TEXAS 
The Texas Department of Health is the Title II grantee agency.  They allocate funds to 26 
regionally-based lead agencies which are either a local health department or a community 
organization.  The State monitors the lead agencies and relies on the lead agencies to 
monitor the programmatic and fiscal compliance of the service providers they fund. 
Funding decisions made at the local level are directed by regional consortia. 
The Title II structure in Austin differs from the rest of Texas.  In Austin, the Austin/Travis 
County Health and Human Services Department is the lead agency for the Title I funds 
and Title II consortium funds.  The local planning council performs the  priority setting 
and allocation for both Title I and Title II funds to all 10 counties. 
23 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  APPENDIX B 
Agency Comments

24 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  APPENDIX B 
Agency Comments

25 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  APPENDIX B 
Agency Comments

26 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  APPENDIX B 
Agency Comments

27 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  APPENDIX B 
Agency Comments

28 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  APPENDIX B 
Agency Comments

29 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  APPENDIX B 
Agency Comments

30 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  APPENDIX B 
Agency Comments

31 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393  APPENDIX C 
Related Office of Inspector General Reports

The Ryan White CARE Act: Local Implementation Issues (OEI-05-93-00336)

The Ryan White CARE Act: Examples of Local Coordination (OEI-05-93-00335)

The Ryan White CARE Act: Special Projects of National Significance (OEI-05-93-00332)

The Ryan White CARE Act: Technical Report of 1992 Expenditures (OEI-05-93-00334)

The Ryan White CARE Act: Funding Formulas (OEI-05-93-00330)

Medicaid Managed Care and HIV/AIDS (OEI-05-97-00210)

Audit of State AIDS Drug Assistance Programs Use of Drug Price Discounts (A-01-97-01501)

Review of Maximizing Drug Discounts Under the Missouri Ryan White Program 
(A-01-97-00926) 
Audit of the Ryan White Comprehensive AIDS Resource Emergency Act of 1990, Title II, 
Administered by the Health Resources and Services Administration (A-01-97-01500) 
Audit of Eligibility Under Title I of the Ryan White Comprehensive AIDS Resource Emergency 
Act of 1990, New York Metropolitan Area (A-02-95-02517) 
Audit of the Ryan White Comprehensive AIDS Resource Emergency Act of 1990, Title II, 
Administered by the State of Connecticut (A-01-96-01501) 
Audit of the Ryan White Comprehensive AIDS Resource Emergency Act of 1990 in the Boston 
Metropolitan Area for Fiscal Year 1994 (A-01-95-01504) 
32 
Ryan White Evaluation Systems: Title II  OEI-05-98-00393