United States General Accounting Office

GAO Report to the Honorable

Daniel Patrick Moynihan, U.S. Senate

July 1995 WELFARE TO WORK

State Programs Have

Tested Some of the

Proposed Reforms

GAO/PEMD-95-26

GAO United States

General Accounting Office

Washington, D.C. 20548

Program Evaluation and

Methodology Division

B-261315 

July 14, 1995

The Honorable Daniel Patrick Moynihan

United States Senate

Dear Senator Moynihan:

As the Congress has been considering various proposals to reform welfare,

questions have arisen about how best to reduce welfare dependency and

help recipients move from welfare to work. This report responds to your

request that we review the evaluations of the numerous state

welfare-to-work experiments completed since the reforms enacted in 1988

to learn (1) How do they resemble the welfare reforms currently being

discussed? and (2) What approaches have been effective in increasing

employment and earnings or reducing benefit receipt among welfare

clients?

Background The nation’s major cash assistance program to poor families, Aid to

Families with Dependent Children (AFDC), provides cash benefits to needy

families with children who lack support from one or both of their parents

because of unemployment, incapacity, absence, or death. Funded with

federal and state dollars, the program operates as an individual

entitlement—that is, everyone who meets the eligibility requirements is

entitled to receive benefits. In fiscal year 1993, AFDC benefits supported

5 million families and more than 9.5 million children each month and cost

over $25 billion in federal and state funds. The Family Support Act of 1988

created the Job Opportunities and Basic Skills Training (JOBS) program,

which requires the states to enroll an increasing proportion of their adult

AFDC recipients (primarily women) in the education, training, and

employment-related activities they need to get jobs and avoid long-term

welfare dependency. The states are permitted substantial flexibility in

designing and implementing their JOBS programs, but they are required to

provide participants with the support services deemed necessary, such as

child care and transportation. Federal funds to match state JOBS

expenditures are capped, but most states have not reached the limit of that

cap. However, as we reported last December, the share of AFDC recipients

active in JOBS is limited; only about one fourth of those required to

participate were served in an average month in fiscal year 1993.1

1

U.S. General Accounting Office, Welfare to Work: Current AFDC Program Not Sufficiently Focused on

Employment, GAO/HEHS-95-28 (Washington, D.C.: December 1994).

Page 1 GAO/PEMD-95-26 Welfare to Work
B-261315 

Rapid growth in the AFDC caseload since 1989 and concern about program

costs and beneficiaries’ long-term dependence have led to widespread

dissatisfaction with the AFDC program and to several congressional

proposals to reform it. Some provisions of current proposals represent

continuity with previous legislative efforts to strengthen the employment

focus of the program, such as requiring larger proportions of recipients to

participate in a work program. Other provisions propose dramatic changes

in AFDC’s structure, such as imposing time limits on the receipt of benefits

and replacing the individual entitlement to benefits with a block grant for

which federal funding would be fixed.

Concern about welfare dependency has spurred policy initiatives since the

1970s to encourage or assist welfare clients to get jobs. The states have

obtained waivers from existing federal statutes and regulations to test a

variety of welfare-to-work initiatives. One condition of the waivers is that

the states rigorously evaluate the effects of these initiatives. Evaluations

conducted under such waivers informed the formulation of the JOBS

program; others completed since 1988 can similarly inform the current

debate.

This report presents the results of our evaluation synthesis of nine

published high-quality studies, from eight states, of welfare-to-work

experiments for adult AFDC recipients.2

 We identified these studies by

conducting a systematic search and methodological review of all

evaluations published since the Family Support Act of 1988 that focused,

at least in part, on moving clients from welfare to work. All nine studies

used comparison groups, six of which were formed through random

assignment, making it possible to estimate the effects of a program by

comparing the outcomes for its participants with those for

nonparticipants.

To meet our first objective, we compared the approaches used in these

experiments with provisions of the proposed welfare reforms being

debated. Our list of provisions was derived primarily from the pending

House welfare reform bill, H.R. 4, but we also included a few provisions

from other bills introduced in the 104th Congress.3

 To meet our second

objective—to identify approaches that successfully moved AFDC recipients

2

The nine studies covered 10 programs; the Greater Avenues for Independence (GAIN) program in

Riverside, California, was sufficiently different from the GAIN programs evaluated in the other

California counties to treat it separately.

3

Our reference to H.R. 4 is to the March 27, 1995, version passed by the House. We excluded bills

introduced after April 1995, when we completed our review.

Page 2 GAO/PEMD-95-26 Welfare to Work
B-261315 

from welfare to work—we compared and contrasted the statistically

significant effects of similar and dissimilar programs on participants’

earnings, employment, and welfare receipt. (See appendix I for details on

our selection and analysis of these studies.) We conducted our work in

accordance with generally accepted government auditing standards

between December 1994 and April 1995. However, we did not

independently verify the information in the evaluation reports.

Results in Brief The state welfare-to-work experiments provide evidence on the states’

experience with some features of current federal welfare reform proposals

but not others. The features common to both address work programs for

recipients, stricter requirements for participation in work programs and

child support enforcement, and increasing work incentives. Some

proposal features, such as limiting the length of time a family can receive

benefits, are currently being tested by the states, but their evaluations are

not yet complete. Other proposed federal reforms have not been tested by

the states. In addition, some states have evaluated features of

welfare-to-work programs—such as providing a broad mix of employment

services—that go beyond the requirements of some of the current

proposals. For example, one proposal would not count some of these

activities toward meeting its work program participation standards,

thereby creating a potential disincentive for financing them. Thus, the

states’ positive experience with the broader mix of employment activities

is relevant to considering this proposal. Although the states’ experiences

provide information regarding some of these reform features, other

features—such as time-limiting benefits and turning the AFDC program into

a block grant—would so alter the environment that we cannot confidently

project the likely effects of the entire package of reform proposals.

The most successful programs—those that consistently showed better

employment and welfare-related outcomes for participants—combined a

broad range of employment-related activities and support services with

some form of participation mandate, and they had adequate funding to

fully serve their clients as intended. Each of these programs made a broad

mix of services available to their clients, such as intensive job search and

placement assistance, basic skills and secondary education, and

vocational training. Other programs using this approach were less

successful when funding shortfalls delayed full implementation or where

budget freezes caused services to be cut back. However, moving welfare

recipients into self-supporting employment has proved challenging. Even

the most successful program had modest effects; after 3 years, only one

Page 3 GAO/PEMD-95-26 Welfare to Work
B-261315 

fourth of its participants were self-sufficient in being both employed and

off welfare.

A quite different approach—increasing both work incentives (such as

allowing recipients to earn somewhat more income before their benefits

were reduced) and access to employment supports (such as child

care)—was also successful in one program at increasing employment and

earnings, but it did not reduce welfare receipt. However, two other states

that increased work incentives and mandated work program participation,

but did not expand their employment-related services, have not yet seen

clear improvements in either employment or welfare-related outcomes.

Principal Findings

The Completed State

Experiments Have Tested

Only Some of the Proposed

Welfare Reforms

The welfare-to-work experiments we reviewed tested many of the

provisions in welfare reform proposals (including H.R. 4), such as

conducting some form of work program that may provide support services

such as child care and requiring adult AFDC recipients to participate in that

work program and to cooperate with child support enforcement. (See

table 1.) In addition, some states experimented with extending medical

and child care benefits to families as they leave welfare for work and with

increasing the disregard of earnings while on welfare, both of which are

provisions in other current proposals. Of course, the states may not have

implemented these features in quite the same form as they appear in the

legislative proposals.

Page 4 GAO/PEMD-95-26 Welfare to Work
B-261315 

Table 1: Relationship Between

Features in Federal Proposals to

Reform AFDC and States’ Completed

Welfare Evaluations

Completed state evaluations

Federal welfare reform

proposals Included these features

Did not include these

features

Included these features Provide a work program

with support services

Require work program

participation 

Require cooperation with

child support enforcement

Extend transitional medical

and child care benefits 

Increase disregard of

earnings while on welfare

Limit lengths of stays on

welfare

Bar additional benefits for

children born while on

welfare

Bar aid to unwed teenage

mothers living on their own

and to noncitizens

Cap the amount of federal

funds available

End requirement for states

to match federal

expenditures

Did not explicitly address

these features

Enhance employment and

training activities

Consolidate Food Stamp

and AFDC programs

Not applicable

Proposed provisions not in the state experiments we reviewed include

limits on the length of welfare receipt, prohibition of additional benefits

for additional children born to families on welfare, and requirements that

unwed teenage mothers live with a parent or guardian. These are the

subject of ongoing or planned experiments and have not yet been

evaluated. Prohibiting aid to noncitizens, creating block grants with fixed

funding, and ending requirements that the states match federal

expenditures have not been options available to the states. Replacing the

current AFDC program with a block grant would basically repeal current

federal law prescribing state procedures for determining individuals’

eligibility for benefits and benefit levels. This change aims to increase the

states’ flexibility in managing their programs of assistance to needy

families and would provide the states with a fixed amount of funds each

year rather than matching (at federally specified rates) whatever their

expenditures had been.

The states have also tested several program features not explicitly

addressed in some of the legislative proposals, such as enhancing

employment and training activities and consolidating the AFDC and Food

Stamp programs. Some of these experiments were begun before the JOBS

Page 5 GAO/PEMD-95-26 Welfare to Work
B-261315 

program was enacted but tested features it currently requires, such as

providing a broad range of employment-related and support services.

Under H.R. 4, the states would be permitted but no longer required to

provide as broad a range of employment-related services and supports.

Indeed, the states might be discouraged from enrolling clients in some

types of education and training because these activities would not count

toward the bill’s work program participation requirements. The states

would face financial sanctions if they failed to meet minimum

participation levels. Thus, these state experiments are relevant to the

question of whether the more inclusive provisions of current law should

be retained.

A Range of Programs Had

Positive Results

All but three of the experiments had a statistically significant positive

effect on at least one of the following: participants’ employment, earnings,

receipt of welfare, and welfare payment amounts.4

 Four were successful

on all four outcomes, three others on only one or two. Effects were

positive more often on employment and earnings than AFDC receipt, but a

variety of approaches and their combinations had some success.

Program outcomes were often measured 1, 2, or sometimes 3, and in one

up to 5, years after clients had been enrolled. We scored them as “positive”

if a statistically significant effect in the intended direction was recorded at

any of these time points. The more complex scores for Florida’s Project

Independence (FPI) program are discussed below.5

Table 2 summarizes the major features being tested and does not include

features that applied to both the experimental and comparison groups. For

example, programs that did not test an employment and training program

(the last four rows) offered similar levels and kinds of employment

services to both the program and comparison groups, but only the

program participants were offered an increase in the earned income

disregard.

4

At the .05 level or better. All the programs wanted to increase client earnings and employment and

decrease AFDC receipt and payments.

5

We did not base our analysis on reported effect sizes because to do so would imply a false level of

precision in the comparison of very different, complex programs and evaluations. Few meaningful

conclusions could be drawn from comparing these effect sizes.

Page 6 GAO/PEMD-95-26 Welfare to Work
B-261315 

Table 2: Ten Treatments That Were Tested in Welfare-to-Work Evaluations and Their Outcomesa

AFDC

Treatment Outcome

Program Mandate

Income

disregard

Support

serviceb

Employment and

training

Merge

programs Earnings Employment Receipt Payments

Calif.

SWIM

Monthly

participation;

fixed

activity

sequence

No Child care Emphasis: job

search. Also CWEP,

education, and

training

No Positive Positive Positive Positive

Mass.

ET

Work

registration

only

No Child

care;

transitional

care

Emphasis:

voluntary education

and training. Also

CWEP and job

search

No Positive Positive Positive Positive

Calif.

GAIN

Work

registration;

activity

sequences

No Child

care;

transitional

care

Emphasis: basic

education and

training. Also job

search and

vocational

education

No Positive Positive Positive Positive

Riverside

GAIN

Work

registration;

activity

sequences

No Child

care;

transitional

care

Emphasis: job

search. Also basic

and vocational

education and job

training

No Positive Positive Positive Positive

Fla.

FPI

Orientation;

activity

sequences

No Child

care;

transitional

care

Emphasis:

independent job

search. Also basic

and vocational

education and job

training

No Positive

early;

none laterc

None early;

positive later

None

early;

none later

Positive

early;

positive

later

Ohio

TI

Education

and training

No None Emphasis:

education and

training. Also CWEP

and job search

No Nonec Positive Nonec Nonec

N.Y.

CAP

Child

support

cooperation

Yes Child care No Yes Positive Positive Nonec Nonec

Ala.

ASSETS

(AFDC)

Work

registration;

child

support

cooperation

Yes Reduced

child care

No Yes Not

available

Not available Nonec Nonec

(continued)

Page 7 GAO/PEMD-95-26 Welfare to Work
B-261315 

AFDC

Treatment Outcome

Program Mandate

Income

disregard

Support

serviceb

Employment and

training

Merge

programs Earnings Employment Receipt Payments

Mich.

TSMF

Education

and training

Yes None No No Nonec Nonec Nonec Nonec

Wash.

FIP

None Cash

bonus

Child

care;

transitional

care;

transitional

medical

No. But small cash

bonus for

participating in

education or training

Yes Nonec Nonec Negatived Negatived

aPrograms abbreviated are AFDC, Aid to Families with Dependent Children; ASSETS, Avenues to

Self-Sufficiency through Employment and Training Services; CAP, Child Assistance Program;

CWEP, Community Work Experience Program; ET, Employment and Training; FIP, Family

Independence Program; FPI, Florida Project Independence; GAIN, Greater Avenues for

Independence; SWIM, Saturation Work Initiative Model; TI, Transitions to Independence—Fair

Work; TSMF, To Strengthen Michigan Families.

bTransitional care and transitional medical are the extension of child care and medical care

benefits to ease clients’ transition to work after leaving AFDC.

cNone statistically significant at p = .05 or less.

dThe program increased AFDC receipt or payment rather than reducing them as intended.

Because the programs typically combined several features at once,

individually they do not provide clear tests of the effectiveness of single

program features. Therefore, we drew our conclusions about the success

of program approaches (including clusters of these features) both by

comparing the effects of programs that included and did not include the

same feature and by comparing the features of the more and less

successful programs. However, our sample of 10 studies is not large

enough to provide conclusive answers, because, of course, there are many

differences between the studies, some of which might have influenced

their outcomes.

Combining a Broad Range

of Employment-Related

Services and Supports

Yielded the Best, Though

Modest, Results

The most successful welfare-to-work programs—those with the largest

and most consistent effects—offered participants an expanded mix of

education, training, and employment services; increased child care

assistance; and mandated some form of client participation. Four

programs using this same general approach—San Diego’s Saturation Work

Initiative Model (SWIM), Massachusetts’ Employment and Training (ET)

program, and California’s Greater Avenues for Independence (GAIN)

Page 8 GAO/PEMD-95-26 Welfare to Work
B-261315 

program, both statewide and in Riverside county—were the only ones to

record statistically significant effects on all four outcomes.

These programs provided a mix of employment-related services, of which

clients could receive one or more. Education and training included

assistance in basic education, preparation for the high-school equivalency

examination (or GED), English-language training, and vocational classes.

Intensive job search included program staff working with employers to

develop job placements, assisting clients with their job search, or starting

clients with job searches immediately. In addition, some offered

community work experience (CWEP), which involves unpaid work in public

or nonprofit agencies aimed at increasing clients’ employability. Their

evaluations compared participants’ outcomes to those of AFDC clients who

received whatever the standard level of employment services was at the

time. Since some of these programs began operating before the JOBS

program was enacted, they typically offered either a lower level of service

than is currently required or nothing at all.

Child care assistance was increased to allow participation in employment

preparation activities and, during the first year of postwelfare

employment, to facilitate the transition off public assistance. Participation

mandates included requirements to register for job search and apply for

work, participate for a specified number of hours per month, or enroll in a

sequence of employment-related activities. However, this does not mean

that all clients actually participated; some could be exempted for personal

reasons, others for lack of program resources.

There were, however, some significant differences in the four successful

programs. Massachusetts’ ET allowed voluntary client participation and

selection of activities after a mandatory work registration, while

California’s SWIM enforced a fixed sequence of activities and GAIN allowed a

variety of sequences. ET put more emphasis on education and training,

while GAIN in Riverside put more emphasis on aggressive job search

support. The statewide program emphasized basic education more than

the other programs.

Two other programs in Ohio and Florida that took the same general

approach had mixed results, which could in part be explained by funding

problems that delayed or cut short the full experiment. Ohio’s economy

took a downward turn at the start of the Transitions to

Independence—Fair Work (TI) program evaluation period, causing an

influx of cases and lengthy backlogs. In fact, a majority of clients did not

Page 9 GAO/PEMD-95-26 Welfare to Work
B-261315 

even receive their employment and training assignments. TI achieved

effects on only one of the four outcomes. Florida’s FPI showed positive

effects for first-year participants on two outcomes, but an economic

downturn combined with a budget freeze led to program reductions in the

second year. This provided the opportunity to test the effects of the

changes—increases in caseloads and the elimination of child care

assistance. However, the contribution of these features is unclear because

both the early and later groups of participants achieved mixed results.

The effects of even the most successful program were modest. The

Riverside GAIN program is arguably the most successful of the

welfare-to-work programs. It increased the proportion of clients ever

employed in 3 years to 67 percent, or 14 percentage points over the

comparison group, but this means that 33 percent of clients in the best

program were never employed in 3 years. Of those who were employed at

the end of 3 years, only 24 percent made more than $5,000 per year. Thus,

Riverside GAIN participants averaged a 49-percent increase in earnings over

3 years compared to nonparticipants receiving only traditional AFDC, but

this amounted to only $3,113, or about $1,000 per year. The Riverside

program lowered average AFDC payments for all participants over 3 years

by 15 percent, or $1,983, and reduced the percentage who were receiving

AFDC payments after 3 years by 5 percent, compared to the

nonparticipants. However, after 3 years only one fourth of its participants

had achieved self-sufficiency by being both employed and off welfare.

That the successful programs only modestly reduced welfare dependency

has, no doubt, a variety of causes. Even when participation was mandated,

not all recipients were required to enroll in activities, some were exempt

for ill health or to care for an infant, and others had to wait for

assignments. In addition, some education and training programs had

participation and attendance problems that diminished their success.

These might reflect problems that clients had that support services like

those in these programs could address, or perhaps other interventions are

needed. Researchers also point to other barriers to moving welfare

recipients into self-supporting employment—in particular, their low skill

levels and the low wages and short tenure of low-skill jobs. In 1992,

45 percent of the single mothers receiving AFDC lacked a high school

diploma and another 38 percent had no schooling beyond high school.6

 Yet

occupations that accept limited schooling pay fairly low wages, have

limited fringe benefits (such as health insurance), and are characterized by

6

U.S. General Accounting Office, Families on Welfare: Sharp Rise in Never-Married Women Reflects

Social Trend, GAO/HEHS-94-92 (Washington, D.C.: May 1994).

Page 10 GAO/PEMD-95-26 Welfare to Work
B-261315 

high job turnover.7 Thus, relatively short-term training and job search

interventions may have a limited effect on recipients whose skill levels are

low.

Increasing Work Incentives

Also Succeeded When

Reinforced by

Employment Supports

Rather than enhancing work-related services, the New York Child

Assistance Program (CAP) took a different approach, providing an

incentive to work by increasing the amount of earned income working

recipients could keep. The program supported this incentive by lowering

barriers to reentering the job market; it provided child care stipends in

advance for clients to use during job search and training. New York’s

program successfully increased employment and earnings but did not

reduce welfare receipt.

In contrast, two programs that increased work incentives and mandated

work program participation without expanding employment-related

services or child care assistance have not yet succeeded. Michigan’s “To

Strengthen Michigan Families” (TSMF) program increased the amount of

the income disregard and also required participation in some form of work

program. AFDC clients were required to enter into “a social contract” in

which they had to complete 20 hours a week of broadly defined “useful”

activities of their own choice, such as education or job search. However,

no additional child care assistance was provided to assist them in keeping

this contract, and there were no significant effects in the first year. During

the second year, some small effects were achieved for both earnings and

welfare receipt for some subgroups but typically only in the final quarter

or month for those with 2 years of data. Evaluation of the effects on the

full sample and their stability will have to await future reports.8

Similarly, Alabama’s Avenues to Self-Sufficiency through Employment and

Training Services (ASSETS) program increased work incentives and

strengthened its work registration and child support cooperation

requirements. In addition to raising the amount of the basic earnings

disregard, ASSETS raised the limits on savings and other resources that

families were allowed to have while remaining eligible for AFDC. However,

it also reduced the amount that could be specifically deducted from

earnings for child care expenses. The implementation of their planned

employment and training component was delayed by 2 years, so available

7

Rebecca M. Blank, “Outlook for the U.S. Labor Market and Prospects for Low-Wage Entry Jobs,” in

Demetra Smith Nightingale and Robert H. Haveman (eds.), The Work Alternative: Welfare Reform and

the Realities of the Job Market (Washington, D.C.: Urban Institute Press, 1995).

8

Table 2 includes only the first year’s effects because data for the full sample were not provided in the

most recent report.

Page 11 GAO/PEMD-95-26 Welfare to Work
B-261315 

results do not fully reflect it. This program has had no significant effects

on welfare receipt or average payment so far, although the evaluation is

not yet complete.

Finally, like New York’s CAP, Washington’s Family Independence Program

(FIP) both provided economic incentives to encourage work and increased

child care assistance. It also aimed to increase participation in education

and training by offering small cash bonuses to the participants. However,

FIP’s plans became difficult to implement under budget restrictions, and

caseloads increased sharply without a corresponding increase in staff.

Several features were implemented minimally, such as improving a client’s

contact with a case manager and increasing resources to pay for education

and training. In addition, the comparison group began getting very similar

services in 1990, about a year and a half into the program, when JOBS was

implemented in Washington state. Thus, it is difficult to know how to

attribute the significant increase in AFDC receipt and payments

experienced by this program’s participants.

Conclusions Our review of state experiences suggests that the most successful

programs offered a broader package of employment-related services than

some proposed reform legislation encourages. The programs that

successfully increased employment and earnings and reduced welfare

receipt offered a broad mix of education, training, and employment-related

services and supports like those in the current JOBS program. However,

under H.R. 4, welfare recipients enrolled in some education and training

activities would not count toward meeting the work program participation

levels that are required in order to avoid financial sanctions.

Some provisions of the proposed reforms—like the time limit on benefit

receipt—have not yet been tested and thus we cannot confidently project

the future effects of either those individual provisions or the entire

package of reforms. For example, imposing a strict limit on the length of

time a family can receive benefits might influence participants’ work

behavior. This could influence the effectiveness of both types of work

programs, those offering either a broad or narrow package of services; we

simply do not have similar past experiences to draw upon.

The modest results of even the most successful programs implies that

(1) within the current program structure, even increasing investments in

employment and support services will not quickly reduce caseloads or

welfare dependency, and (2) additional research is needed to understand

Page 12 GAO/PEMD-95-26 Welfare to Work
B-261315 

the barriers to better program performance and to develop and test more

successful approaches. However, it should be recognized that some of

these barriers may reside outside the welfare program’s control, including

poor school preparation and the limited availability and low wages of

low-skill jobs.

Although federal funds for AFDC benefits have not been capped before, the

states have limited the funds available for their work programs. Our

review suggest that adequacy of funds can be a critical barrier to the

success of efforts to help clients move from welfare to work. Three states

in our review were unable to sustain or fully implement their planned level

of service because state budget constraints kept them from increasing

program capacity to match their growing caseloads. However, by reducing

federal prescriptions on the use of these funds, the reform proposals aim

to increase the states’ flexibility to manage such resource constraints.

Many of the program evaluations that we reviewed were conducted under

the requirement that waivers of federal regulations be rigorously

evaluated. The pending welfare reform legislation would reduce federal

regulation in order to foster further state experimentation, but it would,

thereby, effectively remove that evaluation requirement and thus possibly

reduce the incentive for future evaluations of state experiments.

Recommendations We are not making recommendations in this report.

Agency Comments

and Our Response

The U.S. Department of Health and Human Services (HHS) commented on a

draft of this report and generally agreed with our conclusions but argued

that (1) the differences between the programs studied and those that

would be offered under H.R. 4 are so substantial that one must conclude

that the proposed reforms have not been tested and (2) the report makes

too strong a case for individual factors explaining program success or

failure and should instead describe the “package” of services that may

have led to certain effects. On the first point, we agree that some features

of the proposed reforms have not been tested, but we believe that the

states’ experiences with the program features that would be included

under some of the current proposals, as well as with other features that

might be discouraged, are relevant to consideration of these reforms. The

text has been altered, as necessary, to clarify this distinction. On the

second point, our general approach was to focus on packages of services.

However, where appropriate we have made changes to clarify this. In

Page 13 GAO/PEMD-95-26 Welfare to Work
B-261315 

addition, HHS provided suggestions for clarifications that we have

incorporated, as appropriate, throughout the text. HHS’s comments are

reprinted in appendix II.

We will send copies of this report to the Chairman of the House

Subcommittee on Human Resources of the Committee on Ways and

Means, the Chairman of the Senate Finance Committee, the Secretary of

Health and Human Services, and others who are interested. Copies will

also be made available to others on request. If you have any questions

concerning this report or need additional information, please call me on

(202) 512-2900 or Robert L. York, Director of Program Evaluation in

Human Services Areas, on (202) 512-5885. Other major contributors to this

report are listed in appendix III.

Sincerely yours,

Joseph F. Delfico

Acting Assistant Comptroller General

 for Program Evaluation

 and Methodology

Page 14 GAO/PEMD-95-26 Welfare to Work
Page 15 GAO/PEMD-95-26 Welfare to Work
Contents

Letter 1

Appendix I 

Our Evaluation

Synthesis

Methodology

18

Search for and Selection of Studies 18

Quality Review of Evaluation Studies 19

Synthesis of Program Evaluation Results 20

Strengths and Limitations of Our Synthesis 21

Appendix II 

Comments From the

Department of Health

and Human Services

22

Appendix III 

Major Contributors to

This Report

27

Bibliography 28

Evaluations in This Report 28

Other Studies 29

Related GAO Products 32

Tables Table 1: Relationship Between Features in Federal Proposals to

Reform AFDC and States’ Completed Welfare Evaluations

5

Table 2: Ten Treatments That Were Tested in Welfare-to-Work

Evaluations and Their Outcomes

7

Page 16 GAO/PEMD-95-26 Welfare to Work
Contents

Abbreviations

AFDC Aid to Families with Dependent Children

ASSETS Avenues to Self-Sufficiency through Employment and

Training Services

CAP Child Assistance Program

CWEP Community Work Experience Program

ET Employment and Training

FIP Family Independence Program

FPI Florida Project Independence

GAIN Greater Avenues for Independence

GED General equivalency diploma

HHS Department of Health and Human Services

JOBS Job Opportunities and Basic Skills Training program

SWIM Saturation Work Initiative Model

TI Transitions to Independence

TSMF To Strengthen Michigan Families

Page 17 GAO/PEMD-95-26 Welfare to Work
Appendix I 

Our Evaluation Synthesis Methodology

We conducted an evaluation synthesis to identify approaches that have

successfully helped welfare clients achieve economic independence. That

is, we conducted a systematic review and analysis of the results of

previous evaluation studies of programs sharing this goal. Whereas some

evaluation syntheses examine studies of similar programs to learn whether

a treatment consistently has had the intended effect, we examined studies

of programs that used a range of different approaches toward the same

goal to learn which ones had been successful.

Our evaluation synthesis consisted of several steps. The first step entailed

locating state welfare-to-work experiments and screening them to identify

rigorous evaluation studies with reliable results in terms of the intended

outcomes. In the second step, we identified the commonalities and

differences among the programs and assessed whether these were related

to the programs’ demonstration of effects. We then drew conclusions from

the cumulative picture of existing research about what approaches have

helped AFDC clients move from welfare to work.

Search for and

Selection of Studies

We identified relevant, potentially high-quality studies by searching for as

many existing evaluation studies as possible of welfare-to-work programs

for adult AFDC clients. Our criteria were

• A program could have started before 1988 but its evaluation had to have

been reported after the passage of the Family Support Act of 1988.

• A study had to be testing, at least in part, the effect of welfare-to-work

initiatives on adult AFDC single parents.

• The study measured the effects of the program on employment or AFDC

receipt.

• The program’s effects were measured through a comparison group of

nonparticipants (not necessarily a control group).

We searched for references to terms such as Family Support Act, JOBS, and

welfare reform in on-line bibliographic databases, including CCRSP, ERIC,

Sociological Abstracts, the PAIS International index of the Public Affairs

Information Service, and the NIS index of the U.S. Department of

Commerce. From the resulting abstracts, we were able to screen the

hundreds of citations down to six promising evaluations.

In addition, we reviewed the bibliographies of research studies and

interviewed experts on welfare evaluation to identify other studies we

should consider. The experts identified an additional three studies that

Page 18 GAO/PEMD-95-26 Welfare to Work
Appendix I 

Our Evaluation Synthesis Methodology

had only just been published and therefore had not yet appeared in

databases or bibliographies. This gave us a total of nine potentially

high-quality evaluations of 10 different programs from eight states. (The

Riverside County GAIN evaluation included treatments and effects

sufficiently different from the rest of California’s GAIN evaluation that we

considered them as separate programs.1

) Finally, we confirmed this list of

nine evaluations with program and evaluation officials at HHS. They

suggested several studies that we might consider as background but no

additional impact evaluations.

We explicitly excluded programs focused exclusively on AFDC teenagers,

who may have very different needs. We also excluded unpublished studies,

implementation studies, evaluations of single program features rather than

complete programs, and many studies and reviews that did not examine

program effects. So, for example, we excluded the Utah Unemployed

Parents evaluation and the National Job Training Partnership Act study,

because they did not focus on single parents.

Quality Review of

Evaluation Studies

After identifying the 10 programs, we rated the quality of each study to

ensure that the research was rigorous and would produce reliable results.

We used six specific criteria, adapted from dimensions in The Evaluation

Synthesis, that together would reflect the rigor, consistency, and reliability

of an evaluation study:2

• similarity of the comparison group to the program’s clients,

• adequacy of the sample size for the analyses performed,

• standardization of data collection procedures,

• appropriateness of the measures used to represent the outcome variables,

• adequacy of the statistical or other methods used to control for threats to

validity, and

• presence and appropriateness of the methods used to analyze the

statistical significance of observed differences.

We rated each study on a three-point scale from “unacceptable,” because

the report provided no information on the dimension or the method was

1

Of course, Riverside GAIN also had much in common with the other GAIN programs in California;

while we considered Riverside separately, we were unable to exclude Riverside data from the larger

evaluation of the GAIN program in six counties.

2

See U.S. General Accounting Office, The Evaluation Synthesis, GAO/PEMD-10.1.2 (Washington, D.C.:

March 1992), p. 31. A similar set of dimensions was used in U.S. General Accounting Office, Teenage

Pregnancy: 500,000 Births Year But Few Tested Programs, GAO/PEMD-86-16BR (Washington, D.C.:

July 1986), p. 34.

Page 19 GAO/PEMD-95-26 Welfare to Work
Appendix I 

Our Evaluation Synthesis Methodology

so flawed that the data were probably wrong, to “acceptable,” indicating

that an appropriate method had been used or attempts had been made to

minimize problems.

Results of Quality Review Most of the 10 programs had well-designed and rigorously structured

quasi-experimental or experimentally based evaluations. Six of the 9

evaluations had comparison groups formed by random assignment. In

Alabama and Washington, the comparison groups were drawn from AFDC

clients in demographically similar jurisdictions; in Massachusetts, from a

random sample of clients who did not start a program activity within a

specified time period. The rigor of our first screening of programs was

reflected when all 10 programs met our standards. However, there were

problems with the implementation and execution of several of these

programs, rather than with their evaluation designs, that have to be kept in

mind when interpreting them. A weakness, or confounding factor, in 3

programs was the similarity in services received by the program

participants and the comparison group. (This was a serious problem in

Washington but only a minor problem in California’s SWIM and GAIN

programs.) This type of confounding factor means that the standard

measure of a program’s effect—the difference between outcomes for the

two groups—most likely underestimates the program’s potential effect.

Overview of Programs All 10 programs targeted single adult AFDC recipients, but 2 also included a

small number of unemployed couples in their results. The recipients were

overwhelmingly women. Some programs were statewide while others

were conducted in several counties or just one county. A few were

voluntary; most were mandatory. Some included mothers with children

younger than 6 but older than 3; others simply excluded mothers with

preschool children. Some delivered services directly; others provided

referrals or did nothing at all. Some programs included new AFDC

applicants, others included people already enrolled, and some used both.

Synthesis of Program

Evaluation Results

We focused on program effects on aspects of economic self-sufficiency:

employment, earnings, and public assistance receipt (any effects reported

on additional outcomes are not included here). For each outcome in each

study, we compared the participants receiving program services

(treatment group) with those of the control (or comparison) group;

statistically significant differences were deemed to be program effects.

The evaluation reports estimated the likelihood that these differences

Page 20 GAO/PEMD-95-26 Welfare to Work
Appendix I 

Our Evaluation Synthesis Methodology

stemmed from random chance by using standard tests of statistical

significance. For our interpretation, we used a common significance level

of 5 percent (.05) or less, which was stricter than that used by some of the

evaluations.

We used a structured approach to look for program features or

characteristics that might explain why some programs had positive effects

and others did not, for each of the desired outcomes. First, we

hypothesized how each of a program’s features might affect each of its

outcomes. Then we compared the results of the programs that had each of

those features and those that did not. We found mixed results, and we

found that programs tended to group in clusters of features, which we

examined for their successes.

We also examined features of the studies themselves that might have

influenced the reporting of statistically significant results, such as whether

the treatment and comparison groups received similar services. We

reviewed the comments of the evaluators about any problems they had

encountered in program or study implementation. We considered not only

what services were delivered and how but also how services might have

influenced the participants’ behavior.

Strengths and

Limitations of Our

Synthesis

Clearly, looking across the studies provided us with information not

readily seen by looking only at individual studies. Including several

program approaches in our review allowed us to see that while a

particular approach can be successful, this does not mean that it is the

only successful approach. Examining patterns across a group of studies

may allow inferences about which of the variety of a program’s

components were probably responsible for its effects; examining single

studies ordinarily does not. However, our sample of nine studies cannot

provide conclusive answers, since there are many potential differences

between studies that might be related to why one has significant results

and another does not.

Page 21 GAO/PEMD-95-26 Welfare to Work
Appendix II 

Comments From the Department of Health

and Human Services

Note: GAO comments

supplementing those in the

report text appear at the

end of this appendix.

Page 22 GAO/PEMD-95-26 Welfare to Work
Appendix II 

Comments From the Department of Health

and Human Services

See comment 1.

See comment 2.

Now p. 19.

Page 23 GAO/PEMD-95-26 Welfare to Work
Appendix II 

Comments From the Department of Health

and Human Services

See comment 3.

See comment 4.

Now p. 9.

See comment 5.

See comment 6.

See comment 7.

See comment 8.

Page 24 GAO/PEMD-95-26 Welfare to Work
Appendix II 

Comments From the Department of Health

and Human Services

See comment 9.

Page 25 GAO/PEMD-95-26 Welfare to Work
Appendix II 

Comments From the Department of Health

and Human Services

The following are GAO’s comments on the June 19, 1995, HHS letter.

GAO Comments 1. The text has been changed to more clearly highlight the differences in

employment and training programs between the proposals and the

successful programs we reviewed and to indicate that the states may not

have implemented features exactly as they appear in current bills. We have

also clarified issues relating to program design and environment

differences.

2. Our general approach was to focus on the package of features unique to

the successful programs, while also noting differences among them.

Characteristics such as the age of a mother’s youngest child, noted in

appendix I, did not distinguish the four successful programs from the

others. However, we have made changes to the text to remove the

impression that a single factor was claimed as responsible for program

failures.

3. The text has been changed to indicate study results that are not yet final.

4. The text has been changed to indicate that in Massachusetts, after

registering for work, clients could choose whether to engage in other

employment-related activities.

5. The text has been clarified to indicate our belief in the importance of the

package of services provided by the successful programs. Although some

of these programs resemble the current JOBS program, we do not believe

they offer sufficient evidence from which to draw conclusions about the

JOBS program per se.

6. The names of the programs not using random assignment are now noted

in appendix I.

7. Table 2 has been changed to denote the availability of child care in the

SWIM program.

8. The text has been changed to clarify that the evaluation of the statewide

GAIN program was limited to six counties.

9. The Florida groups have been explained in the text.

Page 26 GAO/PEMD-95-26 Welfare to Work
Appendix III 

Major Contributors to This Report

Program Evaluation

and Methodology

Division

Stephanie L. Shipman, Assistant Director

Daniel G. Rodriguez, Project Manager

Page 27 GAO/PEMD-95-26 Welfare to Work
Bibliography

Evaluations in This

Report

Fein, David J., Erik Beecroft, and John Bloomquist. The Ohio Transitions

to Independence Demonstration: Final Impacts for JOBS and Work Choice.

Cambridge, Mass.: Abt Associates, 1994.

Friedlander, Daniel, and Gayle Hamilton. SWIM: The Saturation Work

Initiative Model in San Diego: A Five-Year Follow-up Study. New York:

Manpower Demonstration Research Corporation, 1993.

Hamilton, William L., et al. The New York State Child Assistance Program:

Program Impacts, Costs, and Benefits. Cambridge, Mass.: Abt Associates,

1993.

Hargreaves, Margaret, and Alan Werner. The Evaluation of the Alabama

Avenues to Self-Sufficiency Through Employment and Training Services

(ASSETS) Demonstration: Interim Implementation and Process Report.

Cambridge, Mass.: Abt Associates, 1993.

Kemple, James J., Daniel Friedlander, and Veronica Fellerath. Florida’s

Project Independence: Benefits, Costs and Two-Year Impacts of Florida’s

JOBS Program. New York: Manpower Demonstration Research

Corporation, 1995.

Kemple, James J., and Joshua Haimso. Florida’s Project Independence:

Program Implementation, Participation Patterns, and First-Year Impacts.

New York: Manpower Demonstration Research Corporation, 1994.

Long, Sharon K., and Douglas A. Wissoker. The Evaluation of the

Washington State Family Independence Program: Final Impact Analysis

Report. Washington, D.C.: Urban Institute Press, 1993.

Long, Sharon K., Demetra Smith Nightingale, and Douglas A. Wissoker.

The Evaluation of the Washington State Family Independence Program.

Washington, D.C.: Urban Institute Press, 1994.

Nightingale, Demetra Smith, et al. Evaluation of the Massachusetts

Employment and Training (ET) Program. Washington, D.C.: Urban Institute

Press, 1991.

Riccio, James, Daniel Friedlander, and Stephen Freedman. GAIN: Benefits,

Costs, and Three-Year Impacts of a Welfare-to-Work Program. New York:

Manpower Demonstration Research Corporation, 1994.

Page 28 GAO/PEMD-95-26 Welfare to Work
Bibliography

Werner, Alan, and Robert Kornfeld. The Evaluation of “To Strengthen

Michigan Families.” Second annual report. First-Year Impacts. Cambridge,

Mass.: Abt Associates, 1994.

Werner, Alan, and Robert Kornfeld. The Evaluation of “To Strengthen

Michigan Families.” Third annual report. Second-Year Impacts. Cambridge,

Mass.: Abt Associates, 1995.

Werner, Alan, and David Rodda. Evaluation of the Alabama Avenues to

Self-Sufficiency Through Employment and Training Services, (ASSETS)

Demonstration. Interim impact report. Cambridge, Mass.: Abt Associates,

1993.

Other Studies Bloom, Howard S., et al. The National JTPA Study Overview: Impacts,

Benefits, and Costs of Title II-A. Cambridge, Mass.: Abt Associates, 1994.

Brock, Thomas, David Butler, and David Long. Unpaid Work Experience

for Welfare Recipients: Findings and Lessons from MDRC Research. New

York: Manpower Demonstration Research Corporation, 1993.

Burghardt, John, and Anne Gordon. The Minority Female Single Parent

Demonstration: More Jobs and Higher Pay—How an Integrated Program

Compares with Traditional Programs. New York: Rockefeller Foundation,

1990.

Burghardt, John, et al. The Minority Female Single Parent Demonstration.

Vol. 1. Summary Report. Princeton, N.J.: Mathematica Policy Research,

1992.

Friedlander, Daniel. The Impacts of California’s GAIN Program on Different

Ethnic Groups: Two-Year Findings on Earnings and AFDC Payments. New

York: Manpower Demonstration Research Corporation, 1994.

Greenberg, David, Robert Meyer, and Michael Wiseman. “When One

Demonstration Site Is Not Enough.” Focus, 16:1 (Spring 1994), 15-20.

Gueron, Judith M., and Edward Pauly. From Welfare to Work. New York:

Manpower Demonstration Research Corporation, 1991.

Hamilton, Gayle. The JOBS Evaluation: Early Lessons from Seven Sites.

New York: Manpower Demonstration Research Corporation, 1994.

Page 29 GAO/PEMD-95-26 Welfare to Work
Bibliography

Hargreaves, Margaret, et al. Illinois Department of Public Aid: Community

Group Participation and Housing Supplementation Demonstration. Fourth

interim report. Cambridge, Mass.: Abt Associates, 1994.

Levin-Epstein, Jodie, and Mark Greenberg. The Rush to Reform: 1992 State

AFDC Legislative and Waiver Actions. Washington, D.C.: Center for Law and

Social Policy, 1992.

Manski, Charles F., and Irwin Garfinkel (eds.). Evaluating Welfare and

Training Programs. Cambridge, Mass.: Harvard University Press, 1992.

Nightingale, Demetra Smith, and Robert H. Haveman (eds.). The Work

Alternative: Welfare Reform and the Realities of the Job Market.

Washington, D.C.: Urban Institute Press, 1995.

O’Neil, June E. Congressional Budget Office Cost Estimate of H.R. 1214,

The Personal Responsibility Act of 1995. Washington. D.C.: Congressional

Budget Office, 1995.

Porter, Kathryn H. Making JOBS Work: What The Research Says About

Effective Employment Programs for AFDC Recipients. Washington, D.C.:

Center on Budget and Policy Priorities, 1990.

U.S. Department of Labor. What’s Working (and What’s Not): A Summary

of Research on the Economic Impacts of Employment and Training

Programs. Washington, D.C.: 1995.

U.S. Department of Labor, Employment and Training Administration.

“American Poverty: The Role of Education, Training, and Employment

Strategies in the New Anti-Poverty Struggle.” Evaluation Forum, 10

(Summer 1994).

Zambrowski, Amy, and Anne Gordon. Evaluation of the Minority Female

Single Parent Demonstration: Fifth Year Impacts at CET. Princeton, N.J.:

Mathematica Policy Research, 1993.

Page 30 GAO/PEMD-95-26 Welfare to Work
Page 31 GAO/PEMD-95-26 Welfare to Work
Related GAO Products

Welfare to Work: Most AFDC Training Programs Not Emphasizing Job

Placement (GAO/HEHS-95-113, May 19, 1995).

Welfare to Work: Participants’ Characteristics and Services Provided in

JOBS (GAO/HEHS-95-93, May 2, 1995).

Welfare to Work: Current AFDC Program Not Sufficiently Focused on

Employment (GAO/HEHS-95-28, Dec. 19, 1994).

Child Care: Current System Could Undermine Goals of Welfare Reform

(GAO/T-HEHS-94-238, Sept. 20, 1994).

Families on Welfare: Sharp Rise in Never-Married Women Reflects Societal

Trend (GAO/HEHS-94-92, May 31, 1994).

(973796) Page 32 GAO/PEMD-95-26 Welfare to Work
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. Orders for 100 or more copies to be mailed to a

single address are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (301) 258-4097 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Mail

Postage & Fees Paid

GAO

Permit No. G100
GAO/PEMD-95-26 Welfare to Work