Department of Health and Human Services 

OFFICE OF 

INSPECTOR GENERAL 

JUNE GIBBS BROWN 

Inspector General 

AUGUST 1999 

OEI-05-98-00392 

RYAN WHITE EVALUATION SYSTEMS 

Title I: Grants to Metropolitan Areas 
OFFICE OF INSPECTOR GENERAL 

The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, is to 

protect the integrity of the Department of Health and Human Services programs as well as the 

health and welfare of beneficiaries served by them. This statutory mission is carried out through a 

nationwide program of audits, investigations, inspections, sanctions, and fraud alerts. The 

Inspector General informs the Secretary of program and management problems and recommends 

legislative, regulatory, and operational approaches to correct them. 

Office of Evaluation and Inspections 

The Office of Evaluation and Inspections (OEI) is one of several components of the Office of 

Inspector General. It conducts short-term management and program evaluations (called 

inspections) that focus on issues of concern to the Department, the Congress, and the public. The 

inspection reports provide findings and recommendations on the efficiency, vulnerability, and 

effectiveness of departmental programs. 

OEI's Chicago Regional Office prepared this report under the direction of William Moran, 

Regional Inspector General. Principal OEI staff included: 

REGION 

William Moran, Project Leader

Ann Maxwell, Lead Analyst

Victoria Jacobs, Analyst

Nora Leibowitz, Analyst

Vince Greiber, New York Regional Office

HEADQUARTERS 

Elise Stein, Program Specialist 

Rada Spencer, Program Specialist 

Alan Levine, Program Specialist 

To obtain copies of this report, please call the Chicago Regional Office at (312) 353-9867. 

Reports are also available on the World Wide Web at our home page address: 

http://www.dhhs.gov/progorg/oei 
EXECUTIVE SUMMARY 

PURPOSE 

To examine the evaluation systems used by Title I Ryan White grantees to monitor the 

compliance, measure the outcomes and measure the overall effectiveness of the programs 

they fund. 

BACKGROUND 

In 1990, Congress passed the Ryan White Comprehensive AIDS Resources Emergency

(CARE) Act as a comprehensive response to the HIV epidemic and its impact on

individuals, families, communities, cities and States. The Ryan White CARE Act was reauthorized in fiscal year (FY) 1996 through FY 2000. Title I of the Act provides

emergency relief grants to eligible metropolitan areas (EMAs) disproportionately affected

by the epidemic to provide community-based HIV-related health and support services. In

FY 1999, 51 EMAs in 21 States, Puerto Rico and the District of Columbia received $486

million in Title I formula and supplemental funds.

The Ryan White CARE Act will be considered for its second re-authorization in 2000. 

As Congress debates this re-authorization, questions are certain to be asked about the

effectiveness and impact of the programs funded. This inspection examines the capacity of

Title I Ryan White grantees to answer those questions.

We conducted an examination of the evaluation mechanisms used by 12 EMAs through an

extensive documentation review and on-site interviews. The 12 EMAs visited represent

the majority of Ryan White Title I funds and nearly half of HIV/AIDS cases. 

FINDINGS 

Title I Grantees Have Mechanisms in Place to Adequately Monitor Their 

Sub-grantees’ Activities 

The 12 Title I grantees we visited receive frequent program and fiscal reports from their 

sub-grantees, regularly review their financial audits and conduct routine site visits. A 

majority of the grantees are tightening fiscal control through unit cost and fee-for-service 

contracting and tightening programmatic control through EMA-wide databases and the 

collection of client-level data. Ensuring sub-grantees use Ryan White funds as the payor 

of last resort is one area where further monitoring mechanisms may be necessary. 

Ryan White Evaluation Systems: Title I 1 OEI-05-98-00392 
Few Grantees Currently Require Outcome Measurement of Their Sub-grantees; 

Most Are in the Planning Stage of Outcome Measurement 

Currently, only 2 of the 12 grantees visited require sub-grantees to measure client 

outcomes. The other 10 Title I grantees visited are in various stages of preparing to 

require sub-grantees to collect and report outcome measures in the next few years. The 

majority of sample Title I grantees have concerns regarding outcome measures and desire 

more specific guidance from the Health Resources and Services Administration (HRSA) 

on how to proceed. 

The Majority of Title I Grantees Visited Assess the Needs of Sub-populations in 

HIV/AIDS Care but Do Not Solicit Needs Information from Those Outside of 

HIV/AIDS Care 

In addition to analysis of public health data, information on client needs is often collected 

through provider and consumer surveys, focus groups and public forums. These 

mechanisms focus on populations currently in HIV/AIDS care. To detect the needs of 

those not in HIV/AIDS care, the majority of the Title I grantees visited match the 

demographics of HIV/AIDS cases to the demographic data of the clients receiving Title I 

funded services in their eligible metropolitan area (EMA). Matches of epidemiological 

and utilization data do not provide a complete picture of the needs of persons not in 

HIV/AIDS care. The matches do not provide information on the barriers individuals face 

to care or their unique service needs. 

RECOMMENDATIONS 

The HRSA Should Promote the Development of Metropolitan-wide Data Systems 

to Further Improve the Program Management Activities of the Sub-grantees 

The HRSA should provide grantees with technical assistance on how to develop 

EMA-wide data systems through which the grantees and sub-grantees can access 

client-level data. Sub-grantee access to client-level data eliminates inefficient intake 

duplication and facilitates service coordination. Data systems capacity is a vital ingredient 

for HIV/AIDS service agencies to improve the efficiency and effectiveness with which 

they provide their services, track their outcomes and inform their fund sources of their 

activities and results. There are several potential sources of funds which could be used to 

assist grantees in building client-level data systems. 

The HRSA Should Implement a Multi-year Outcomes Initiative Including Grantee 

Requirements and Technical Assistance 

The HRSA should build upon their efforts to promote grantee outcome measurement by 

establishing a multi-year implementation plan. Outcome measurement should start on the 

Ryan White Evaluation Systems: Title I 2 OEI-05-98-00392 
sub-grantee level and lead through subsequent steps to grantee analysis of client outcomes 

across the EMA. Outcome measurement would foster program improvements and greater 

accountability for expenditures and highlight the value of Ryan White funds. 

The HRSA Should Provide Technical Assistance to Improve the Ability of 

Grantees to Estimate the Needs of Infected Persons Not in HIV/AIDS Care 

The HRSA currently requires grantees to estimate unmet need of persons not in 

HIV/AIDS care by comparing service utilization data to the HIV epidemic in their area. 

The HRSA should require the grantees to improve their knowledge of unmet need by 

soliciting information directly from persons not in HIV/AIDS care. The input of infected 

persons who are not receiving care would add a richness of understanding regarding their 

needs and the barriers to service they encounter. 

Grantee solicitation of the input of persons un-connected to care would assist the grantees 

in complying with the re-authorized CARE Act’s directive to prioritize a portion of funds 

for emerging populations. Grantee solicitation of this input would also contribute to the 

Department’s efforts to address the greater burden of HIV/AIDS on racial and ethnic 

minorities. 

Several grantees indicated a need for technical assistance on using outreach strategies to 

solicit information from persons not in HIV/AIDS care. The HRSA could provide the 

grantees with a technical assistance document detailing how to use outreach strategies 

with examples of effective strategies for replication. Particular attention should also be 

paid to reaching persons who receive HIV/AIDS services but may not be receiving 

primary health care services. 

AGENCY COMMENTS 

The HRSA provided comments to the draft report. While HRSA concurred with the 

report’s findings and recommendations, they offered suggestions for clarifying the report 

and making other technical changes. Where appropriate, we changed the report to reflect 

their comments. The complete text of HRSA’s comments can be found in Appendix A. 

The HRSA pointed out several ongoing and planned activities in response to the report 

recommendation to promote the development of metropolitan-wide data systems. We 

commend their efforts in this area. The HRSA cautioned, however, that they do not have 

the financial resources to enable grantees to fully develop and implement these systems. 

We recommend that HRSA, in partnership with the Department, secure funds to assist in 

the development of such data systems. The FY 2000 re-authorization of the CARE Act 

Ryan White Evaluation Systems: Title I 3 OEI-05-98-00392 
provides an optimal opportunity to consider funding for infrastructure development vital 

to ensuring efficient and effective service delivery in the future. The grantees could build 

upon an initial infusion of funds to streamline their systems, potentially reducing their 

administrative costs and freeing up funds to maintain their systems over time. 

We agree with HRSA that the participation of non-Ryan White public health agencies is 

necessary to provide additional resources, skills and data to develop client-level data 

systems which capture the full array of services used by HIV-infected persons. To this 

end, we recommend that HRSA build upon the HRSA-Health Care Financing 

Administration (HCFA)- Centers for Disease Control and Prevention (CDC) inter-agency 

data sharing agreement to promote the development of integrated public health data 

systems. 

The HRSA briefly described a few activities planned or underway in support of our 

recommendation to implement a multi-year outcomes initiative. We commend HRSA on 

these activities and plans. Despite their efforts in this area, HRSA expressed a few 

concerns regarding their ability to carry out this recommendation: 1) data limitations, 

2) funding limitations, and 3) confidentiality concerns. 

We do not expect the outcomes initiative to lead to rigorous scientific evaluations 

conducted by every grantee. By examining the client-level impact of the services they 

fund, the grantees can assess in general terms the effectiveness of these services. 

Although we expect the costs involved to be minimized by emphasizing the use of data 

resources already available to the providers, many grantees may need additional financial 

resources to develop an outcomes system. The metropolitan-wide client-level data 

systems discussed in our first recommendation would develop a sound infrastructure upon 

which to build the outcome measures initiative. We recommend that HRSA work with the 

Department to secure funds for these initiatives critical to the future of the Ryan White 

CARE Act programs. 

The HRSA’s third area of concern - assuring client confidentiality, can be addressed by 

using unique identifiers and staff confidentiality agreements. 

Ryan White Evaluation Systems: Title I 4 OEI-05-98-00392 
TABLE OF CONTENTS 

EXECUTIVE SUMMARY ............................................... 1

INTRODUCTION .......................................................... 6

FINDINGS

Adequate Monitoring Mechanisms in Place ................................. 10

Few Require Outcomes ................................................ 12

Solicit Information on Needs from Clients, Not Others ......................... 14

RECOMMENDATIONS .................................................... 15

AGENCY COMMENTS .................................................... 21

APPENDICES

A: Agency Comments ................................................. 23

B: Related Office of Inspector General Reports .............................. 31

Ryan White Evaluation Systems: Title I 5 OEI-05-98-00392 
INTRODUCTION 

PURPOSE 

To examine the evaluation systems used by Title I Ryan White grantees to monitor the 

compliance, measure the outcomes and measure the overall effectiveness of the programs 

they fund. 

BACKGROUND 

This study complements our reports, Ryan White Evaluation Systems, Title II: Grants to 

States (OEI-05-98-00393) and Ryan White Evaluation Systems, AIDS Drug Assistance 

Program (OEI-05-98-00390). 

The Ryan White CARE Act 

In 1990, Congress passed the Ryan White Comprehensive AIDS Resources Emergency 

(CARE) Act as a response to the HIV epidemic and its impact on individuals, families, 

communities, cities and States. The Ryan White programs provide health and support 

services to persons with HIV/AIDS who would otherwise not have access to care. The 

Ryan White programs constitute slightly over $1 billion of the $6.8 billion in HIV/AIDS 

expenditures in the U.S. Department of Health and Human Services fiscal year (FY) 1998 

budget. 

The Ryan White CARE Act was re-authorized in FY 1996 through FY 2000. Changes to 

the Act included a directive to Title I planning councils to consider cost and outcomes in 

funding allocation decisions to the extent that such data are reasonably available. 

The Ryan White CARE Act's four titles and Part F direct resources to various entities and 

allow grantees maximum flexibility in the use of funds. In FY 1998, nearly 90 percent of 

the Act’s funds were awarded to Title I and Title II grantees. 

Title I of the Ryan White CARE Act 

Title I provides emergency relief grants to eligible metropolitan areas (EMAs) 

disproportionately affected by the HIV epidemic. Title I grants are for community-based 

HIV-related outpatient and ambulatory health and support services, including case 

management, comprehensive treatment services, prescription drugs, counseling, 

transportation, meals-on-wheels programs, home care and hospice care. 

Ryan White Evaluation Systems: Title I 6 OEI-05-98-00392 
Grantees are required to establish a planning council that reflects the demographics of the 

epidemic in the EMA and is representative of the range of individuals involved in HIV 

care including health and social service providers, government entities, planning agencies, 

community leaders and other Federal HIV program grantees. The planning council 

develops a comprehensive plan for delivering HIV services in the EMA, prioritizes the 

allocation of funds in the EMA, assesses the grantee’s fund allocation mechanisms and 

evaluates the effectiveness of services in meeting identified need. 

Metropolitan areas are eligible for Title I formula funding if they have a population of 

500,000 or more and have a total of more than 2,000 cases of AIDS reported to the U.S. 

Center for Disease Control (CDC) for the most recent period of 5 calendar years. In FY 

1999, there were 51 EMAs in 21 States, Puerto Rico and the District of Columbia. In FY 

1999, EMAs received $486 million in Title I formula and supplemental funds. 

OIG Evaluation Recommendations in 1995 

In a June 1995 Office of Inspector General (OIG) report entitled “The Ryan White CARE 

Act: Local Implementation Issues”(OEI-05-93-00336), the OIG found an absence of 

program outcome evaluation to measure the CARE Act’s impact. Beyond surveys of 

client satisfaction undertaken by providers at the local level, grantees and planning 

councils were not assessing the impact of CARE Act funds on their service area. 

The OIG recommended that the Health Resources and Services Administration (HRSA) 

and the Assistant Secretary for Planning and Evaluation (ASPE) fund appropriate studies 

to assess the overall effectiveness of the Ryan White programs. The OIG also 

recommended that HRSA ensure grantees evaluate their own programs. The OIG 

suggested local evaluations include provider compliance reviews, outcome measure 

studies and overall program effectiveness reviews. The OIG asked HRSA to provide 

grantees with technical assistance and evaluation instruments that guarantee consistency in 

data collection and evaluation. 

The ASPE did not conduct any evaluations of the Ryan White program with HRSA as 

recommended in the 1995 OIG report. However, ASPE is currently funding the HIV 

Costs and Services Utilization Study (HCSUS) with HRSA and the Agency for Health 

Care Policy and Research (AHCPR). The HCSUS examines who is treated for HIV, the 

health services HIV patients receive, the costs of the services and means of payment. The 

ASPE and AHCPR are also sponsoring a related study on the impact of health care 

financing arrangements on access to and use of new and emerging HIV therapies. 

The HIV/AIDS Bureau of HRSA has a national Ryan White evaluation agenda which 

includes client-level data studies, links with secondary data sets, topic specific evaluations 

and the aggregate reporting system. The Bureau is focusing on evaluating the following 

areas: 1) Enrolling under-served, vulnerable populations; 2) Providing clients with care 

equal to that received by the insured; 3) Providing services that remove barriers to 

Ryan White Evaluation Systems: Title I 7 OEI-05-98-00392 
receiving primary care; 4) Reducing morbidity, including opportunistic infections and 

perinatal transmission, and maintaining or increasing CD4 levels; 5) Reducing mortality; 

and 6) Adapting to a changing service and cost environment. 

HRSA’s Current Evaluation Requirements 

The FY 1997 and 1998 Grant Application Guidances asked grantees to specify outcomes 

for four service categories, including primary medical care. The FY 1999 Guidance asks 

grantees to describe their processes and results in developing outcomes. Grantees are also 

instructed to describe the impact of Title I funds. Suggested outcomes to use include: 

providing access to care whose quality meets or exceeds Public Health Service and other 

standards, removal of barriers to care, and morbidity reductions. 

Title I grantees are required to conduct a needs assessment of HIV-infected populations in 

their service area and describe its use in funding decisions. The FY 1998 Application 

Guidance instructed applicants for supplemental funds to estimate unmet need for: persons 

with HIV by level of illness or transmission category, race/ethnicity, and sex; or enhanced 

primary care access. Applicants were also instructed to estimate the number of persons 

with HIV without access to new treatments and/or requisite primary health care. 

The FY 1999 Application Guidance asks the grantees for more information on unmet need 

to be obtained through comparisons of HIV prevalence and service utilization data. The 

Guidance emphasizes the importance of documenting unmet need and indicates future 

HRSA plans to work with the EMAs in this endeavor. 

HRSA’s Technical Assistance on Evaluation 

The HRSA has provided technical assistance to Title I grantees on conducting evaluations 

through a variety of means including: self assessment protocols, meetings to share 

evaluation strategies, and a series of monograph reports on evaluation and assessment. 

The most recent report emphasizes the importance of outcome measurement and provides 

some guidance to grantees on how to approach it. The report does not suggest outcomes 

to measure or steps to develop outcome measures. More recently HRSA has suggested 

grantees focus on primary health care outcomes and will soon disseminate a publication 

focusing on primary health care outcome measures. 

Re-authorization in 2000 

The Ryan White CARE Act will be considered for its second re-authorization in 2000. 

As Congress debates this re-authorization, questions are certain to be asked about the

effectiveness and impact of the programs funded. This inspection examines the capacity of

Title I Ryan White grantees to answer those questions.

Ryan White Evaluation Systems: Title I 8 OEI-05-98-00392 
SCOPE AND METHODOLOGY 

In this inspection we examined: the mechanisms Title I grantees use to monitor subgrantee program and financial compliance, whether and how the grantees measure the 

outcomes of the programs they fund, and whether and how the grantees measure their 

effectiveness in meeting the evolving needs of the affected population. This evaluation 

does not judge the compliance, measure the outcomes or assess the effectiveness of the 

Title I grantees, but rather judges whether Title I grantees have and appropriately utilize 

the tools to perform these evaluation functions. We also did not review HRSA’s 

evaluation of the grantees. 

For the purposes of this analysis, we define compliance as serving the population, 

providing the services and spending the resources as agreed to in return for the award of 

program funds. Outcomes are defined as measures of the benefits clients derived as a 

result of the program. Finally, we define overall effectiveness as how well the mix of 

services meets the evolving needs of the population affected by the epidemic. 

We conducted an on-site examination of these evaluation mechanisms in 12 EMAs: New 

York City, Newark, Bergen-Passaic, Baltimore, Washington D.C., Miami, Orlando, 

Austin, Houston, Chicago, Los Angeles and Sacramento. We selected these EMAs as a 

purposive sample representative of: comparatively high rates of incidence through 

intravenous drug use (IDU) transmission, comparatively high rates of infections among 

minority populations, early and recent EMA designation and regional diversity. In 

addition, our selected EMAs represent the majority of Ryan White Title I funds and nearly 

half of the HIV/AIDS cases. These 12 EMAs contain 38 percent of the living AIDS cases 

reported in 1997 and 51 percent of the Title I funding for FY 1998. 

We conducted an extensive documentation review in the 12 EMAs. We examined their 

FY 1998 Title I grant applications, the contracts of the sample sub-grantees, samples of all 

program, fiscal, outcome and needs assessment reports, site visit protocols and additional 

evaluation materials provided by grantees, planning councils and sub-grantees. 

We supplemented the documentation review with on-site interviews. In each EMA we 

interviewed; (1) the Title I grantee, (2) a representative of the Title I planning council, and 

(3) at least 10 percent of the Title I sub-grantees. The number of sub-grantees visited in 

each EMA ranged from 3 to 14. In each EMA we interviewed sub-grantees of differing 

program size, service delivery purpose and location. 

All of our data collection was conducted from October through December 1998. 

We conducted our review in accordance with the Quality Standards for Inspections issued 

by the President’s Council on Integrity and Efficiency. 

Ryan White Evaluation Systems: Title I 9 OEI-05-98-00392 
FINDINGS 

Title I grantees have mechanisms in place to adequately 

monitor their sub-grantees’ activities 

Grantees receive regular program and fiscal reports and conduct routine site 

visits 

Ten of the 12 Title I grantees visited require their sub-grantees to submit monthly 

program and fiscal reports. The other two Title I grantees require quarterly reports. 

Program reports typically contain units of service provided, number and demographics of 

clients served and a narrative section on program accomplishments and barriers. Fiscal 

reports typically contain line item expenditures or detailed billing invoices for unit cost and 

fee-for-service reimbursement. Most grantees also require fiscal reports to disclose yearto-date expenditures and the unexpended balance. All of the grantees visited report that 

they review annual fiscal audits from their sub-grantees. 

Nine of the 12 sample grantees conduct formal site visits to all of their sub-grantees at 

least once a year. The Baltimore grantee visits each sub-grantee every 2 years. The Los 

Angeles and Austin grantees use a risk assessment model to determine which sub-grantees 

to visit. In all of the sample EMAs, site visits to sub-grantees include review of a sample 

of client files, financial records and the supporting documentation used for reporting. 

Many of the grantees also informally monitor sub-grantees through ongoing 

communication and individualized technical assistance. 

Grantees move toward unit cost reimbursement to tighten fiscal accountability 

Seven of the 12 grantees we visited reimburse the sub-grantees using a unit cost or feefor-service method for all or some of their service categories. One additional grantee is 

planning to use the unit cost method in FY 1999. The other three grantees reimburse their 

sub-grantees on a line item basis. 

The grantees we visited use unit cost reimbursement to tighten their control over costs, to 

inform their resource allocation, to encourage sub-grantee efficiency and to measure subgrantee performance. For example, in Newark the sub-grantees propose a unit cost for 

each service in their grant application. The proposed cost is considered in procurement 

decisions and is subject to contract negotiations. According to a respondent, this helps the 

grantee to be an “informed purchaser” of services and has led to lower service costs. 

Ryan White Evaluation Systems: Title I 10 OEI-05-98-00392 
Many grantees use metropolitan-wide databases and client-level data for program 

management 

Nine of the 12 grantees we visited use or are implementing an EMA-wide database, 

through which all of the sub-grantees enter client level data on a uniform system. In seven 

of the eligible metropolitan areas, the grantee can access the agency data at any time 

directly from the system, rather than via down-loads or a contract administrator. In six of 

the EMAs, the sub-grantees have access to one another’s data. An agency can look up a 

client’s identification number and learn what services the client has received elsewhere. 

This enables the agencies to track clients through the Title I system of care, improving 

client management and reducing service duplication. 

In the EMAs we visited, we noted three implementation concerns regarding unified data 

systems. A few respondents reported that it has been difficult to include hospitals in the 

establishment of a common system since hospitals have distinct computer system needs. A 

second concern expressed by respondents is the data delay caused by a shortage of 

computer terminals. Many agencies with only one computer terminal linked to the EMA 

system enter data in weekly or monthly batches. Delays in data entry curtail the benefits 

of agency to agency client management. A third concern is that where EMA data systems 

are incompatible with Title II systems or reporting requirements, agencies funded by both 

Title I and Title II programs may have to collect and report data through dual systems. 

Client Management in Newark: The Comprehensive HIV/AIDS Management Program System 

(CHAMPS): 

Newark Title I sub-grantees use CHAMPS to track client care, to manage administrative 

functions, to report program and fiscal data to the grantee and to aggregate client outcomes. 

Through the CHAMPS database all sub-grantees and the grantee can access client-level data 

at any time. Information is entered through a unique client identification number. Subgrantees can view a client’s demographic and health profile and review what services the client 

has received through other sub-grantees. Multiple fields of information are available for each 

client, including CD4 & T-cell count, usual source of primary care, and referred services. The 

planning council uses CHAMPS to facilitate its needs assessment and planning work. For 

example, the council has used CHAMPS data to review client residency, service location, and 

the peak months and hours of service utilization to forecast service use. 

Grantees try to ensure that Ryan White funds are used as the payor of last resort; 

Most sub-grantees ensure payor of last resort status 

All of the grantees convey HRSA’s rules regarding payor of last resort status to the subgrantees in their contracts and/or policy and procedure manuals. Some grantees require 

Ryan White Evaluation Systems: Title I 11 OEI-05-98-00392 
sub-grantees to report all sources of revenue to verify the use of Ryan White funds as the 

payor of last resort. In addition, all of the grantees we visited check that sub-grantees are 

documenting Ryan White eligibility in client files during site visits. 

Most of the sub-grantees we visited have mechanisms in place to ensure that they use 

Ryan White funds as the payor of last resort, such as documenting Medicaid and private 

insurance coverage in client files and billing accordingly. However, a few of the ancillary 

service providers we visited appeared unaware of the rules or their responsibility in this 

area. A few also appeared to be unclear on the services paid for by Medicaid in their 

State. In addition, sub-grantees have varying capacity to check Medicaid eligibility. 

Few grantees currently require outcome measurement of 

sub-grantees; most are in the planning stage of outcome 

measurement 

Only 2 of the 12 grantees visited require sub-grantees to measure outcomes 

The Newark and Sacramento Title I grantees require sub-grantees to measure the 

outcomes of the services they are contracted to provide. However, sub-grantee interviews 

in these EMAs indicate that some sub-grantees do not routinely measure outcomes. The 

Newark and Sacramento grantees are taking steps to increase sub-grantee outcome 

measurement through greater standardization and increased reporting requirements. 

Starting in FY 1998, the Sacramento grantee required sub-grantees to report outcome 

measures on a quarterly basis. Due to a low level of sub-grantee compliance in the initial 

6 months, the grantee developed a standard format for outcome reporting. As of FY 

1999, outcome measures are a required component of the sub-grantee quarterly report. 

The Newark grantee requires all sub-grantees to outline expected outcomes and their 

means to measure the outcomes in a logic model in their annual grant proposal. The subgrantees are required to include the following elements in their logic model: 1) service 

gaps and consumer needs, 2) activities to address the needs, 3) outcomes defined as shortterm consumer benefits, 4) measures to quantify the outcomes, and 5) expected long-term 

impacts on the system or community. The logic model is subject to contract negotiations. 

Sub-grantees are not currently required to report outcome measures to the Newark 

grantee. However, the grantee requires the sub-grantees to measure the outcomes in their 

logic model and to conduct an internal evaluation documenting their progress. On site 

visits the grantee reviews the sub-grantees’ internal evaluation reports. Sub-grantee 

outcome measurement, documentation and achievement is scored and used as criteria in 

determining annual contract awards to continuing applicants. 

Ryan White Evaluation Systems: Title I 12 OEI-05-98-00392 
The other 10 Title I grantees visited are preparing to require sub-grantees to 

collect outcome measures in the next few years 

The remaining 10 grantees visited reported plans to require sub-grantees to measure client 

outcomes, however the grantees are at varying levels of planning and implementation. Six 

of the 10 grantees are currently developing outcome measures, half of them are using a 

consultant to draft suggested outcomes. The other four grantees are laying the 

groundwork for outcomes by developing standards of care and data collection 

mechanisms. Seven of the 10 grantees expect to require some or all providers to report 

outcomes by FY 2000. 

Most of the planning councils in the 12 sample eligible metropolitan areas (EMAs) are 

involved in the outcome measure efforts underway in their EMA. In six of the EMAs, the 

councils are involved in the actual determination of the outcome measures. In other 

EMAs, council involvement includes activities such as establishing broad outcome areas, 

participating in the development of service standards and reviewing suggested outcomes. 

Most of the grantees solicited sub-grantee input in developing outcomes. A few of the 

grantees cited the United Way’s manual on outcome measurement as the primary source 

of guidance they have relied upon to direct their efforts in this area. The United Way 

approach emphasizes developing outcomes reflective of an agency’s goals and 

recommends using the data sources readily available to agencies, such as client records, 

client and staff interviews and mechanical tests and measurements. 

Many of the sub-grantees we visited collect client information which could be used to 

measure program outcomes if aggregated. Sub-grantees often document client progress in 

a client’s chart or case-notes. Most of the sub-grantees we visited do not aggregate the 

client outcomes and are not able to compare outcomes across client groups, service 

strategies or over time. Nor do they report outcomes to the grantee or other funders. 

Majority of sampled Title I grantees have concerns regarding outcome measures 

Most of the Title I grantees we visited expressed concerns regarding outcome measure 

requirements. The primary area of concern cited is the lack of financial resources and the 

administrative spending cap. Other concerns include: the burden of additional reporting 

requirements on sub-grantees; a lack of sub-grantee and planning council expertise in 

collection, analysis and use of outcome measures; and the difficulty of isolating the 

influence of Ryan White services on client outcomes. A couple of grantees also expressed 

a hesitancy to advance outcome measurement efforts in the absence of more specific 

guidance from HRSA out of concern that local efforts may conflict with future 

requirements. All of the Title I grantees visited said that they would like more guidance 

from HRSA on how to proceed with establishing and collecting outcome measures. 

Ryan White Evaluation Systems: Title I 13 OEI-05-98-00392 
The majority of Title I grantees visited assess the needs of 

sub-populations in HIV/AIDS care but do not solicit needs 

information from those outside of HIV/AIDS care 

The grantees collect information on service and sub-population needs through annual or 

bi-annual comprehensive needs assessments or targeted assessments of client populations 

and service categories. All of the grantees use both qualitative and quantitative data 

sources in their needs assessment work, including public health data, consumer and 

provider surveys, focus groups, public forums and key informant interviews. Outside of 

the public health data, information on the needs of infected individuals not receiving 

HIV/AIDS care is usually not obtained. 

The majority of Title I grantees visited match the demographics of HIV cases to 

demographic data on the clients receiving Title I services to determine unmet need. These 

data matches do not provide information on the barriers individuals face to care or the 

unique service needs they may have beyond those implied through co-morbidity data. 

Furthermore, in the three EMAs visited which do not have unduplicated client counts, the 

service utilization data most likely count individual clients more than once, inflating the 

numbers of persons appearing to receive care. Hence, a higher proportion of the 

HIV/AIDS population is considered receiving care than do in reality. 

Only a few of the grantees have gone beyond data matches to obtain information on the 

needs of persons who are not in HIV/AIDS care. One planning council tried to solicit this 

information by sending trained peer outreach workers to homeless shelters, without much 

success. Two other councils were more successful in obtaining information from persons 

not in HIV/AIDS care, one through street outreach and the State substance abuse 

program, and the other through focus groups of current and former prison inmates. 

Assessing Unmet Need in New York City: The CHAIN Study 

Since 1995, the New York City planning council has contracted with the Columbia University 

School of Public Health to conduct a longitudinal study of persons living with HIV/AIDS 

(PLWH/A). Through the Community Health Advisory Information Network (CHAIN) study, 

the planning council collects data on a sample of 700 PLWHA, systematically drawn from 

over 40 agencies in New York City, including agencies which receive Ryan White funding and 

those which do not. The sample includes infected persons who are un-connected to care. The 

CHAIN study collects information on client demographics, risk factors, family and social 

supports, barriers to care, environmental impacts and client outcomes. Through the CHAIN 

study, the council has identified unmet needs of persons in and out of care. 

Ryan White Evaluation Systems: Title I 14 OEI-05-98-00392 
RECOMMENDATIONS 

The HRSA should promote the development of 

Metropolitan-wide data systems to further improve the 

program management activities of the sub-grantees 

Building on the client-level data demonstration projects underway,1 HRSA should provide 

grantees with technical assistance to facilitate the development of eligible metropolitan 

area-wide data systems. The sophistication and use of current EMA data systems vary 

widely by grantee. The HRSA should aid grantee efforts to build or improve unified data 

systems by disseminating a variety of technical information including information on model 

systems, effective system development strategies and specific elements to include to 

maximize benefits. This information should incorporate the lessons learned through the 

client-level data demonstration projects. 

Sub-grantee access to client-level data eliminates inefficient intake duplication and 

facilitates service coordination by informing providers of a client’s existing referrals and 

services received. Confidentiality concerns can be overcome using staff confidentiality 

agreements and unique identification numbers, rather than client name, to track clients. 

Sub-grantees can also use EMA-wide data systems to generate routine program and fiscal 

reports, reducing the time and resources needed by sub-grantees for reporting. 

In addition, sub-grantees can use the systems to track and report outcomes. One concern 

we heard from several grantees regarding outcome measurement is that they do not want 

to increase the reporting burden on the sub-grantees. If outcome fields are integrated into 

the EMA’s data systems as they are built, sub-grantees can track, aggregate and report 

client-level outcome measures through the existing system. Toward this end, HRSA is 

developing new software for the Annual Administrative Report (AAR) that will include a 

voluntary module on health-related client-level information. 

The development of client-level data systems requires a substantial financial investment. 

Based on the client-level data demonstration projects, HRSA estimates that it costs 

between $100,000 and $500,000 in start-up costs for each grantee to build a client level 

data system and between $50,000 and $335,000/year for each grantee to maintain the 

1 Under the client level data demonstration project, four EMAs and three States have been 

collecting and analyzing client-level data since 1994. Collected data includes client 

characteristics, health indicators and services received. The HRSA plans to expand the client 

level data set in 1999. 

Ryan White Evaluation Systems: Title I 15 OEI-05-98-00392 
system. Currently, grantees are restricted in the amount of funds they can invest in 

infrastructure by an administrative cap and by competing demands to fund services. 

Recognizing the current limitations on funds available for data collection and evaluation in 

the Ryan White CARE Act, HRSA should work in partnership with the Department to 

secure funds to facilitate the development of needed infrastructure in the CARE Act 

program. 

There are several potential sources of funds which could be used, either alone or in 

combination, to assist grantees in building client-level data systems including: 1) the one 

percent set-aside of program funds reserved for evaluation of programs authorized under 

the Public Health Service Act; 2) an appropriation specifically designated for data system 

development, similar to the funding allocated for such development within the child 

support enforcement program; and 3) a designation of funds for evaluation of the Ryan 

White programs included in the CARE Act’s FY’ 2000 re-authorization, carved outside of 

the administrative spending cap. 

Grantees could build on an initial infusion of funds, received through one or more of the 

above means, to streamline their tracking and reporting systems, potentially reducing their 

administrative costs over the long term. A reduction in administrative costs achieved 

through more efficient tracking and reporting processes could free up funds to support the 

improved data collection systems in the future. 

As HIV/AIDS has moved from an emergency epidemic to a chronic epidemic, it is critical 

for service providers to move from a crisis mode of operations to more long-term capacity 

building. Data systems are a vital ingredient for HIV/AIDS service agencies to improve 

the efficiency and effectiveness with which they provide their services, track their 

outcomes and inform their funding sources of their activities and results. Promoting the 

development of these systems is one practical step the administration can take in its 

initiative to strengthen the capacity of small, community-based organizations to serve the 

emerging HIV/AIDS population. Agencies with the most skill in serving the needs of 

disenfranchised populations often lack the resources for necessary infrastructure. 

In their data systems promotion and technical assistance efforts, the HRSA should urge 

Title I grantees to work with the other grantees in their State to develop a shared system 

so as to reduce multiple system requirements on sub-grantees receiving funding through 

more than one Ryan White Title. Recent HRSA efforts to develop a cross-Title AAR 

should facilitate the development of shared data systems. 

In addition, HRSA should encourage the CARE Act grantees to reach out to other public 

health programs serving persons with HIV/AIDS in their State to share services data. The 

1998 inter-agency agreement between HRSA, HCFA and CDC to promote the 

development of integrated public health information systems and the inter-departmental 

activities supporting this agreement are intended to foster such collaborations. 

Ryan White Evaluation Systems: Title I 16 OEI-05-98-00392 
The HRSA should implement a multi-year outcomes initiative 

including grantee requirements and technical assistance 

The HRSA should build upon their efforts to promote grantee outcome measurement by 

establishing a multi-year implementation plan. Outcome measurement should start on the 

sub-grantee level and lead through subsequent steps to grantee analysis of client outcomes 

across the EMA. Outcome measurement would foster program improvements and greater 

accountability for expenditures and highlight the value of Ryan White funds. 

Here is one option of how HRSA could proceed: 

< In stage one, HRSA could develop one or more core outcomes for each service 

category and require Title I grantees to develop three additional outcomes for each 

service category in collaboration with Title II Consortia where they are co-located. 

The core outcomes HRSA requires the grantees to measure should be aligned with 

HRSA’s performance goals outlined in their annual performance plan. 

< In stage two, HRSA could require grantees to collect the established outcome 

measures from the providers on a quarterly basis. Outcome measures should 

eventually be reported as client-level data in preparation for stage three. 

< In stage three, HRSA could require the grantees to analyze the reported outcome 

measures across services and report EMA-wide outcomes to HRSA annually. 

In establishing the core outcomes, HRSA should place a greater emphasis on primary 

health care outcomes since the ultimate goal of the Ryan White CARE Act is improved 

health status. However, at least one core outcome should be developed for each support 

service category. The outcomes for the support services should reflect the contribution 

the support services make to improved health status such as facilitating access to primary 

care or creating conditions conducive to following and benefitting from a structured health 

care regimen. Measuring these incremental outcomes would provide valuable information 

on the contribution of these services to health outcomes. This information could be used 

for accountability and program improvement purposes. Confidentiality concerns can be 

overcome using staff confidentiality agreements and unique identification numbers, rather 

than client name, to track clients. 

The outcome measurement initiative can serve as a means for HRSA to further promote 

continuous quality improvement (CQI) efforts. Through CQI, information on the 

effectiveness and quality of program delivery is fed back to program managers on a 

continuous basis, serving as a tool for managers to improve program delivery. While 

quality assurance and process indicators are used to gauge the quality of services 

delivered, outcome measures can be used to gauge service effectiveness. Service 

providers could use outcome measures along with quality of care indicators to improve 

their performance through program alterations and further self-evaluation under CQI. 

As the Ryan White CARE Act faces its second decade of providing care to persons 

Ryan White Evaluation Systems: Title I 17 OEI-05-98-00392 
infected with HIV/AIDS, it is time to ensure that the grantees have the capacity to 

evaluate the outcomes of the services they deliver. Outcome evaluation as a CQI tool is 

vital to strengthening the effectiveness of the services delivered and the efficiency of the 

funds expended to provide the services. This basic need for appropriate funding should be 

addressed in the re-authorization of the CARE Act in FY 2000. 

The HRSA should provide the grantees with technical assistance to implement the core 

outcome measures and to develop and implement their own additional outcome measures 

within a CQI framework. The technical assistance could emphasize soliciting provider 

input in the development of the additional outcomes, using the United Way approach. 

The United Way approach to outcome measurement emphasizes using outcome measures 

for program improvement purposes. To this end, the United Way emphasizes soliciting 

provider input in the development of outcome measures to make sure that outcome 

measures reflect program goals. The United Way also stresses the importance of 

considering the data resources already available to the providers when developing 

outcome measures so as not to impose too much additional administrative or cost burden. 

The United Way’s approach recognizes the limited capacity of health and human service 

providers to engage in rigorous outcome evaluation. Hence, the United Way model 

emphasizes contribution rather than attribution. Using outcome measures, providers can 

determine if clients achieve an expected result after receiving their service intervention. 

While the provider is not able to scientifically prove that the result is due solely to their 

intervention, they can see whether their service is contributing to positive outcomes on a 

regular basis. 

The HRSA should urge Title I grantees to work with co-located Title II consortia in 

establishing the outcome measures sub-grantees will be required to measure, so as to 

avoid imposing a greater measurement burden on sub-grantees funded under both Titles. 

The HRSA should also urge the grantees to require sub-grantees to report outcome 

measures as client-level and aggregate data. Grantees will need client-level outcome data 

in order to analyze client outcomes across the EMA. The HRSA should urge the grantees 

to consider this need in the creation or redesign of unified data systems. 

In recent years, HRSA’s application guidance to Title I grantees has stressed the 

importance of documenting outcomes and has asked grantees to include outcome data in 

their applications. All of our respondents have received the message from HRSA that 

outcome measures are important and that HRSA expects them to move ahead in this area. 

However, respondents in the 12 EMAs we visited stated that they would like more 

specificity from HRSA on what outcomes they are expected to measure and on HRSA’s 

implementation plans. 

Likewise, in response to a July 1998 survey sent out by HRSA’s technical assistance 

contractor, 81 percent of the 37 responding EMAs indicated that “clear written guidance 

Ryan White Evaluation Systems: Title I 18 OEI-05-98-00392 
from the HIV/AIDS Bureau on outcome evaluation expectations or requirements” would 

be among the “most useful” types of technical assistance they could receive. Seventyeight percent of the responding EMAs indicated that “lists of suggested outcome 

indicators for other (non-primary care) service categories” would be among the most 

useful assistance. 

The HRSA should provide technical assistance to improve 

the ability of grantees to estimate the needs of infected 

persons not in HIV/AIDS care 

In the FY 1999 Application Guidance, HRSA emphasizes the importance of obtaining 

information on unmet need, acknowledges the difficulty of obtaining this information and 

pledges to work with EMAs to address this area in the future. The HRSA should build on 

this effort by requiring the grantees to solicit needs information from persons not in 

HIV/AIDS care and by providing the grantees with technical assistance on how to do this. 

The public health data grantees are currently required to use to estimate unmet need 

provides useful information on the needs of infected persons. However, the input of 

persons outside of the care system would add a richness of understanding regarding their 

needs and the barriers to service they encounter. 

Grantee solicitation of the input of persons un-connected to care would assist the grantees 

in complying with the re-authorized CARE Act’s directive to prioritize a portion of funds 

for emerging populations. Grantee solicitation of this input would also contribute to the 

Department’s efforts to address the greater burden of HIV/AIDS on racial and ethnic 

minorities. 

In their attempt to fulfill the CARE Act’s requirement to prioritize funds for emerging 

populations, HRSA should urge the planning councils to strive to reach persons 

unconnected to HIV/AIDS care, in addition to efforts to reach persons who 

demographically reflect the emerging populations. The HRSA should urge the councils to 

recognize the complexity and disproportionate costs which may be involved in serving 

persons previously unconnected to HIV/AIDS care. Persons infected by HIV/AIDS are 

increasingly individuals who are traditionally not connected to the health care system for 

reasons such as poverty and historic mistrust of institutional health care. A focused 

outreach and planning strategy is required to address their needs. 

The HRSA’s technical assistance materials on conducting needs assessments urge grantees 

to solicit information from persons not in HIV/AIDS care. Although the materials suggest 

avenues for soliciting this information, they do not include any information on how to use 

these means. Several grantees indicated that they would like more guidance on how to 

proceed in this area. 

Ryan White Evaluation Systems: Title I 19 OEI-05-98-00392 
The HRSA could provide the grantees with a technical assistance document devoted to 

soliciting needs information from PLWH/A who are not in HIV/AIDS care, providing 

steps for conducting outreach and examples of effective strategies. The HRSA could 

highlight outreach strategies which have been used by other Ryan White grantees, CDC 

HIV prevention community planning groups and service providers in other arenas. 

Particular attention should also be paid to reaching persons who receive HIV/AIDS 

services but may not be receiving primary health care services. 

Ryan White Evaluation Systems: Title I 20 OEI-05-98-00392 
AGENCY COMMENTS 

The HRSA provided comments to the draft report. While HRSA concurred with the 

report’s findings and recommendations, they offered suggestions for clarifying the report 

and making other technical changes. Where appropriate, we changed the report to reflect 

their comments. The complete text of HRSA’s comments can be found in Appendix A. 

The HRSA pointed out several ongoing and planned activities in response to the report 

recommendation to promote the development of metropolitan-wide data systems. We 

commend their efforts to assist the grantees in the development of these data systems. 

The revisions to the Annual Administrative Report (AAR) to allow for cross-Title 

reporting and the addition of the voluntary module on health-related client-level 

information in the AAR software will provide the grantees with added flexibility to track 

client-level information across CARE Act Titles. We also commend HRSA on their 

plans to host a FY 1999 meeting of grantees with expertise in client-level reporting to 

launch a peer-based technical assistance effort. 

In response to our recommendation that HRSA provide technical assistance and financial 

resources to support the development of EMA-wide data systems, HRSA cautioned that it 

would “be unreasonable to expect technical assistance and financial resources from 

HRSA’s HIV/AIDS Bureau to be sufficient to fully develop and implement those systems 

in an appropriate manner.” The HRSA has made a great start in the provision of technical 

assistance through building expertise via the seven client-level data demonstration sites 

and the planned FY 1999 meeting to launch the peer-based technical assistance effort. 

These endeavors will facilitate an exchange of information on model systems and lessons 

learned, providing a knowledge base to support local system development efforts. We 

urge HRSA to continue down this path. 

We do not expect HRSA to provide all of the financial resources necessary to fully 

develop and implement metropolitan-wide data systems. We recommend that HRSA, in 

partnership with the Department, secure funds to assist in the development of such data 

systems. The FY 2000 re-authorization of the CARE Act provides an optimal opportunity 

to consider funding for infrastructure development vital to ensuring efficient and effective 

service delivery in the future. The grantees could build upon an initial infusion of funds to 

streamline their systems, potentially reducing their administrative costs and freeing up 

funds to maintain their systems over time. 

Furthermore, we agree with HRSA that the participation of non-Ryan White public health 

agencies is necessary to provide additional administrative resources, skills and data to 

develop client-level data systems which capture the full array of services used by HIVinfected persons. To this end, we recommend that HRSA build upon the HRSA-HCFACDC inter-agency data sharing agreement to promote the development of integrated 

Ryan White Evaluation Systems: Title I 21 OEI-05-98-00392 
public health data systems by encouraging Ryan White CARE Act grantees to work with 

the other public health programs in their State serving persons with HIV/AIDS. 

The HRSA briefly described a few activities planned or underway in support of our 

recommendation to implement a multi-year outcomes initiative. We commend HRSA on 

these activities and plans. Despite their efforts in this area, HRSA expressed a few 

concerns regarding their ability to carry out this recommendation: 1) limitations on 

accessibility and compatibility of data sets, 2) restrictions on administrative costs 

impacting a Grantee’s ability to use complex data sets for outcome studies, and 

3) confidentiality concerns. 

These limitations are to be expected in a large endeavor but are not insurmountable. For 

this reason we recommend an outcomes approach designed with feasibility as a 

cornerstone. We do not expect the outcomes initiative to lead to rigorous scientific 

evaluations conducted by every grantee. The HRSA is funding such projects on a limited 

basis where the capacity exists. It is not feasible, nor is it necessary, for every grantee to 

participate in this work. What is feasible and necessary for every grantee to undertake is 

an assessment of the effectiveness of the services they fund. By examining the 

contributions made by the services to client outcomes, the grantees can assess, in general 

terms, the effectiveness of these services. We highlight the United Way approach which 

emphasizes contribution, rather than attribution, for this reason. The United Way 

approach is designed for service providers, like those in the Ryan White program, who 

face data and cost limitations. 

Although we expect the costs involved to be minimized by emphasizing the use of data 

resources already available to the providers, many grantees may need additional financial 

resources to develop an outcomes system, depending upon the size of their grant and 

competing demands on limited administrative funds. The metropolitan-wide client-level 

data systems discussed in our first recommendation, would develop a sound infrastructure 

upon which to build the outcome measures initiative. We recommend that HRSA work 

with the Department to secure funds for these initiatives critical to the future of the Ryan 

White CARE Act programs. 

The HRSA’s third area of concern - assuring client confidentiality, can be addressed by 

using unique identifiers and staff confidentiality agreements. Grantees collecting clientlevel information have used these measures successfully to address confidentiality 

concerns. 

We commend HRSA’s efforts to address our recommendation to improve the ability of 

grantees to estimate the needs of infected persons not in HIV/AIDS care. We urge HRSA 

to disseminate technical assistance information to grantees in this area as soon as possible, 

as the population of infected persons outside of care continues to grow. 

Ryan White Evaluation Systems: Title I 22 OEI-05-98-00392 
APPENDIX A 

Agency Comments 

Ryan White Evaluation Systems: Title I 23 OEI-05-98-00392 
APPENDIX A 

Agency Comments 

Ryan White Evaluation Systems: Title I 24 OEI-05-98-00392 
APPENDIX A 

Agency Comments 

Ryan White Evaluation Systems: Title I 25 OEI-05-98-00392 
APPENDIX A 

Agency Comments 

Ryan White Evaluation Systems: Title I 26 OEI-05-98-00392 
APPENDIX A 

Agency Comments 

Ryan White Evaluation Systems: Title I 27 OEI-05-98-00392 
APPENDIX A 

Agency Comments 

Ryan White Evaluation Systems: Title I 28 OEI-05-98-00392 
APPENDIX A 

Agency Comments 

Ryan White Evaluation Systems: Title I 29 OEI-05-98-00392 
APPENDIX A 

Agency Comments 

Ryan White Evaluation Systems: Title I 30 OEI-05-98-00392 
APPENDIX B 

Related Office of Inspector General Reports

The Ryan White CARE Act: Local Implementation Issues (OEI-05-93-00336) 

The Ryan White CARE Act: Examples of Local Coordination (OEI-05-93-00335) 

The Ryan White CARE Act: Special Projects of National Significance (OEI-05-93-00332) 

The Ryan White CARE Act: Technical Report of 1992 Expenditures (OEI-05-93-00334) 

The Ryan White CARE Act: Funding Formulas (OEI-05-93-00330) 

Medicaid Managed Care and HIV/AIDS (OEI-05-97-00210) 

Audit of State AIDS Drug Assistance Programs’ Use of Drug Price Discounts (A-01-97-01501) 

Review of Maximizing Drug Discounts Under the Missouri Ryan White Program 

(A-01-97-00926) 

Audit of the Ryan White Comprehensive AIDS Resource Emergency Act of 1990, Title II, 

Administered by the Health Resources and Services Administration (A-01-97-01500) 

Audit of Eligibility Under Title I of the Ryan White Comprehensive AIDS Resource Emergency 

Act of 1990, New York Metropolitan Area (A-02-95-02517) 

Audit of the Ryan White Comprehensive AIDS Resource Emergency Act of 1990, Title II, 

Administered by the State of Connecticut (A-01-96-01501) 

Audit of the Ryan White Comprehensive AIDS Resource Emergency Act of 1990 in the Boston 

Metropolitan Area for Fiscal Year 1994 (A-01-95-01504) 

Ryan White Evaluation Systems: Title I 31 OEI-05-98-00392