Pre-Design Study for the 

Evaluation of Recovery Act 

Green Jobs, Health Care, and 

Other High Growth 

Competitive Grants 

White Paper 

April 6, 2010 

Karen Needels 

Annalisa Mastri 

 

Contract Number: 

DOLQ091A20941/DOLU101 

Mathematica Reference Number: 

06709.200 

Submitted to: 

U.S. Department of Labor 

Employment and Training 

Administration 

200 Constitution Avenue, NW 

Room N-5641 

Washington, DC 20210 

Project Officer: Garrett Groves 

Submitted by: 

Mathematica Policy Research 

P.O. Box 2393 

Princeton, NJ 08543-2393 

Telephone: (609) 799-3535 

Facsimile: (609) 799-0005 

Project Director: Karen Needels

Pre-Design Study for the 

Evaluation of Recovery Act 

Green Jobs, Health Care, and 

Other High Growth 

Competitive Grants 

White Paper

April 6, 2010

Karen Needels 

Annalisa Mastri 

Pre-Design Study of the ARRA Grants Mathematica Policy Research 

iii 

CONTENTS

 EXECUTIVE SUMMARY ...................................................................................xi

 I INTRODUCTION .............................................................................................1

A. Overview of the ARRA Grants...................................................................1

B. Purpose of the White Paper......................................................................2

C. Organization of the White Paper ..............................................................3

 II FEATURES OF THE ARRA GRANTS ...................................................................5

A. ETP Grants...............................................................................................5

1. Goals of the Grants...........................................................................8

2. Size and Duration .............................................................................8

3. Eligible Grantees and Grant Recipients..............................................8

4. Partnerships to Be Formed................................................................9

5. Target Clients...................................................................................9

6. Services and Activities Allowed with Grant Funds ............................10

7. Suitability for an Evaluation ............................................................11 

B. Pathways Out of Poverty Grants .............................................................12

1. Grant Awards, Funding, and Grantee Types ....................................12

2. Partners..........................................................................................13

3. Target Communities and Clients.....................................................13

4. Services and Activities Allowed with Grant Funds ............................14

C. State Energy Sector Partnership Grants ..................................................15

1. Size and Duration of the Grants, Eligible Grantees, and Partners.....15

2. The SESP-Guided Energy Sector Strategy .........................................15

3. Participants Eligible to Receive Training..........................................16

D. Health Care Sector and Other High-Growth and Emerging Industries 

Grants ...................................................................................................16

1. Goals of the Grants.........................................................................17

2. Size and Duration ...........................................................................17

3. Eligible Grantees.............................................................................18

4. Partnerships to Be Formed..............................................................18

5. Target Clients.................................................................................18
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

iv 

Contents (continued) 

 III POTENTIAL OBJECTIVES OF AN EVALUATION ................................................19

A. Potential Objectives of an Evaluation .....................................................19

1. Learning about the Microeconomic Effects of Training on 

Workers..........................................................................................19

2. Understanding the Macroeconomic Effects of the Temporary 

Funding Stream ..............................................................................20

3. Examining the Implementation and Operational Issues Involved 

in Deploying ARRA Grants...............................................................20

B. Considerations for a Study of the Effect of Training on Individuals’ 

Outcomes..............................................................................................21

1. Need for an Understanding of the Counterfactual ...........................21

2. Factors Influencing What the Grants Can Accomplish......................23

3. Potential Outcomes to Be Examined................................................23

 IV RANDOM ASSIGNMENT IMPACT DESIGNS......................................................25

A. Overview of Design Features..................................................................25

1. Basic Approach and Strengths of the Method..................................25

2. Important Components for a Random Assignment Evaluation .........26

B. Possible Approaches to Using Random Assignment for ARRA Grants .....29

1. Randomly Assign All Individuals at All Grantees..............................29

2. Randomly Selecting a Subset of Grantees and then Randomly 

Assigning Individuals at Them ........................................................30

3. Focusing on a Nonrandom Subset of Grantees and then 

Randomly Assigning Individuals at Them........................................30

C. Details of the Random Assignment Process ...........................................31

1. The Point at Which to Insert Random Assignment ...........................31

2. Restriction of Control Group Members’ Access to Services ..............33

D. Scheduling Considerations that Influence All of the Random 

Assignment Designs..............................................................................33

1. Assumptions About the Time Lines of the Grant and Evaluation 

Activities ........................................................................................34

2. Implications about the Portion of the Grantees’ Clients Who 

Could Participate in Random Assignment........................................36

3. Implications about the Follow-Up Period for the Observation of 

Post-Training Outcomes..................................................................37
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

v 

Contents (continued) 

E. Example Minimum Detectible Impacts for Random Assignment 

Studies of the ARRA Grants....................................................................38

1. Types of Variation Taken into Account in the MDI Calculations .......39

2. Example MDIs Based on Different Random Assignment Scenarios 

and Assumptions............................................................................41

 V OTHER TYPES OF METHODOLOGIES..............................................................53

A. Regression Discontinuity with Individuals in Winning Grantees or 

Grant Applicants....................................................................................53

1. Overview of RD Designs..................................................................54

2. RD Designs for an Evaluation of ARRA Grants .................................55

B. Comparison Group Designs...................................................................58

1. Overview of Comparison Group Designs .........................................58

2. Comparison Group Designs for an Evaluation of ARRA Grants.........59

C. Interrupted Time Series Design..............................................................64

1. Overview of Interrupted Time Series Designs ..................................64

2. Strengths and Weaknesses of Interrupted Time Series Designs .......65

 VI SUMMARY OF CONSIDERATIONS WHEN PROCEEDING WITH AN 

 EVALUATION OF THE ARRA GRANTS.............................................................67

 REFERENCES.................................................................................................75

 APPENDIX A: KEY FEATURES OF AWARDED GRANTS .....................................77

Pre-Design Study of the ARRA Grants Mathematica Policy Research 

vii 

TABLES 

Table II.1. Overview of ARRA Grants in the Pre-Design Study ................................... 6

Table IV.1. Minimum Detectible Impacts on the Percentage Employed and 

 Average Quarterly Earnings, Energy Training Partnership Grants........... 42

Table IV.2. Minimum Detectible Impacts on the Percentage Employed and 

 Average Quarterly Earnings, Pathways Out of Poverty Grants ................ 47

Table IV.3. Minimum Detectible Impacts on the Percentage Employed and 

 Average Quarterly Earnings, State Energy Sector Partnership Grants ..... 49

Table VI.1. Overview of Research Questions and Considerations for the 

 Evaluation ............................................................................................ 68

Table VI.2. Overview of Potential Methodologies to Evaluate the ARRA Grants ........ 69

Table A.1. Energy Training Partnership Grant Recipients ....................................... 79

Table A.2. Pathways Out of Poverty Grant Recipients............................................. 84

Table A.3. State Energy Sector Partnership Grant Recipients .................................. 90

Pre-Design Study of the ARRA Grants Mathematica Policy Research 

ix 

FIGURES 

Figure IV.1. Impact Estimation from a Random Assignment Design ......................... 26

Figure IV.2. Hypothetical Time Lines for 3-Year Grant and 5-Year Random 

 Assignment Evaluation ......................................................................... 34

Figure V.1. Visual Interpretation of a Regression Discontinuity Design ................... 55

Figure V.2. Impact Estimation from a Comparison Group Design............................ 59

Figure V.3. Visual Interpretation of a Difference-In-Differences Design ................... 62

Figure V.4. Visual Interpretation of an Interrupted Time Series Design ................... 65

Pre-Design Study of the ARRA Grants Mathematica Policy Research 

xi 

EXECUTIVE SUMMARY 

The recent economic crisis that began in December of 2007 with a housing crisis and a 

contraction in available credit has led to a steady decline in jobs and rising unemployment rates 

across the country (Bureau of Labor Statistics [BLS], January 2010). From December 2007 to 

December 2009, the national unemployment rate rose from 5.0 percent to 10.0 percent. Although 

the number of unemployed persons in the nation rose from 7.7 million to 15.3 million during this 

time, still more individuals are either underemployed or have given up looking for work. 

In February 2009, President Obama signed the American Recovery and Reinvestment Act 

(ARRA) into law to address the employment challenges facing America’s workforce. With a total 

value of $787 billion, the Act’s purposes include preserving and creating jobs, assisting people most 

affected by the recession, and promoting economic efficiency and long-term economic benefits 

(U.S. Congress 2009). A key aspect of the Act was its urgency, as reflected by its enactment shortly 

after the start of the new Congress and administration. 

A major goal of ARRA is to train and assist workers for successful employment in high-growth 

and emerging industries, including the efficient energy and renewable energy sectors. As part of this 

goal, the U.S. Department of Labor (DOL) has created six grant-providing programs, four of which 

have been under DOL’s consideration for an evaluation of their effectiveness: (1) Energy Training 

Partnership (ETP) Grants; (2) Pathways Out of Poverty (Pathways) Grants; (3) State Energy Sector 

Partnership (SESP) and Training Grants; and (4) Health Care Sector and Other High-Growth and 

Emerging Industries (Health Care) Grants.1

 Although these grants share many similar 

characteristics, they differ by providing funding to different types of grantees and, in some cases, by 

targeting different industries. They also have different target populations, such as workers who are 

unemployed and in need of basic skills development, workers dislocated from other failing 

industries, or those who are already in high-growth and emerging industries but in need of updated 

skills. DOL is considering conducting an objective and rigorous evaluation of the ARRA grants. 

To gain insights about how best to proceed with evaluations of these grants, DOL has 

contracted with Mathematica Policy Research to conduct the Pre-Design Study for the Evaluation of 

Recovery Act Green Jobs, Health Care, and Other High-Growth Competitive Grants. The study 

aims to provide DOL with information about different types of methodological approaches for 

conducting the evaluations. 

The purpose of this white paper, provided as the final deliverable product of the pre-design 

study, is to provide guidance to DOL as it assesses potential methodologies for evaluating each 

grant. First, we discuss the features of the grants that might influence the type of evaluation design 

that might be appropriate. We base our discussion on information that is available through the 

Solicitation of Grant Applications (SGAs), as well as the announcements of grant recipients for the 

three sets of grants that were awarded in January 2010. Second, we explore possible objectives of an 

evaluation of the ARRA grants and the types of research questions that DOL might be interested in 

asking. These include learning about the microeconomic (individual-level) effects on workers of 

 1 In addition to these four, the ARRA created two additional grant types, State Labor Market Information 

Improvement Grants and Green Capacity Building Grants. Although these two sets of grants have some similarities to 

the four listed grants, the Solicitation of Grant Applications (SGAs) for those grants did not include information 

regarding a potential evaluation. Therefore, they are not discussed in the remainder of this paper. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

xii 

training funded by ARRA grants, learning about the macroeconomic effects of the grants on 

communities or industries, and learning about implementation issues associated with the grants. 

Third, given the high level of methodological rigor that can be achieved with a random assignment 

design and DOL’s special interest in this evaluation approach, we explore three different types of 

random assignment designs that could be used for an evaluation of the ARRA grants. We assess the 

strengths and weaknesses of these approaches, as well as how features of the grants, such as when 

grantees might begin serving participants, might influence their feasibility. We conclude that, 

although random assignment is the most analytically rigorous of the possible methodologies, it also 

is more likely than other approaches to interfere with normal program activity and will likely take 

longer to implement. We include some estimates of the sizes of program impacts that could be 

detected based on different types of random assignment studies and subsets of grantees, concluding 

that approaches based on a subsample of grantees might be feasible options for DOL to consider. 

Fourth, we explore other types of methodologies, such as regression discontinuity, that could 

be used should a random assignment approach be deemed either inappropriate or infeasible. 

Although not as rigorous as random assignment, and therefore less convincing, these approaches 

still could provide information of considerable value to policymakers. Finally, we summarize the 

relative strengths and weaknesses of all of the methodological approaches that we examined, so that 

their relative merits can be compared. 

The purpose of this white paper is not to recommend a single methodological approach as the 

best way to evaluate these grants, and it does not make such a recommendation. Rather, it explores 

the advantages and disadvantages of different methodological approaches so that DOL can consider 

its options for proceeding with an evaluation of these grants. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

1 

I. INTRODUCTION 

The recent economic crisis that began in December of 2007 with a housing crisis and a 

contraction in available credit has led to a steady decline in jobs and rising unemployment rates 

across the country (U.S. Department of Labor, Bureau of Labor Statistics [BLS] January 2010). 

From December 2007 to December 2009, the national unemployment rate rose from 5.0 percent to 

10.0 percent. Although the number of unemployed persons in the nation rose from 7.7 million to 

15.3 million during this time, still more individuals are either underemployed or have given up 

looking for work. 

In February 2009, President Obama signed the American Recovery and Reinvestment Act 

(ARRA) into law to address the employment challenges of America’s workforce. With a total value 

of $787 billion, the Act’s purposes include preserving and creating jobs, assisting people most 

affected by the recession, and promoting economic efficiency and long-term economic benefits 

(U.S. Congress 2009). A key aspect of the Act was its urgency, as reflected by its enactment shortly 

after the start of the new Congress and administration. 

A major goal of ARRA is to train and assist workers for successful employment in high-growth 

and emerging industries, including the efficient energy and renewable energy sectors. As part of this 

goal, the U.S. Department of Labor (DOL) has created six grant-providing programs, four of which 

have been under DOL’s consideration for an evaluation of their effectiveness: (1) Energy Training 

Partnership (ETP) Grants; (2) Pathways Out of Poverty (Pathways) Grants; (3) State Energy Sector 

Partnership (SESP) and Training Grants; and (4) Health Care Sector and Other High Growth and 

Emerging Industries (Health Care) Grants.2

 Although these grants share many similar 

characteristics, they differ by providing funding to different types of grantees and, in some cases, by 

targeting different industries. They also have different target populations, such as workers who are 

unemployed and in need of basic skills development, workers dislocated from other failing 

industries, or those who are already in high-growth and emerging industries but in need of updated 

skills. DOL is considering conducting an objective and rigorous evaluation of the ARRA grants. 

To gain insights about how best to proceed with evaluations of these grants, DOL has 

contracted with Mathematica Policy Research to conduct the Pre-Design Study for the Evaluation of 

Recovery Act Green Jobs, Health Care, and Other High-Growth Competitive Grants. The predesign study aims to provide guidance as DOL assesses different types of methodological 

approaches for conducting the evaluations. 

A. Overview of the ARRA Grants 

Although few industries were untouched by the current financial crisis, some, including the auto 

industry, were dealt devastating blows. This has resulted in many dislocated workers with few 

opportunities for rejoining industries that were once robust. At the same time, those who are 

already unemployed have even fewer opportunities to join the workforce. In recognition of this, 

ARRA has appropriated substantial funds for training and employment services for workers to join 

high-growth and emerging industries. To promote America’s long-term prosperity, these industries 

 2 DOL also recently has provided State Labor Market Information Improvement Grants and Green Capacity 

Building Grants. Although these two sets of grants have some similarities to the four grants described, the SGAs did not 

include information regarding an evaluation and are not included as part of this study. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

2 

include the efficient energy and renewable energy sectors, as well as health care and others. In 

addition to helping new workers join high-growth industries, training will also be available for 

workers already in these industries to improve their skills. 

To help with the dual objectives of aiding worker transitions to high-growth fields and 

stimulating the economy, DOL issued four solicitations to provide worker training in efficient 

energy, renewable energy, health care, and other high-growth sectors.3

 Taken together, there will be 

about 150 grants totaling $660 million. DOL is positioning itself to conduct a rigorous evaluation of 

the ARRA grants to ensure the efficient and effective allocation of funds for similar types of services 

in the future. Initial steps to perform such evaluations are clearly established in the four program 

solicitations by requiring that “grantees must agree to participate in such an evaluation.” 

B. Purpose of the White Paper 

The purpose of this white paper is to provide guidance to DOL as it assesses potential 

methodologies for evaluating each grant. We discuss the types of research questions that can be 

answered with different methodologies, as well as technical and analytical issues and tradeoffs 

associated with different methodological approaches. We also discuss other considerations for an 

evaluation, such as possible types of data sources that could be used. However, because the purpose 

of the paper is to compare and contrast a wide range of possible approaches, and because the 

information available about the grants is very limited, the discussion of each potential approach is 

not as comprehensive as would be expected during the design phase of an evaluation. 

In developing our guidance, we take as a basic premise that, all else equal, more 

methodologically rigorous approaches are preferred to less rigorous ones. However, the reality is 

that not all else is equal. Different approaches have advantages and disadvantages. For example, 

some approaches can answer certain types of research questions but not others, and some require 

certain assumptions or conditions that might not be feasible or desirable in the case of these 

particular grants. 

The purpose of this white paper is not to recommend a single methodological approach as the 

best way to evaluate these grants, and it does not make such a recommendation. Rather, it explores 

the advantages and disadvantages of different methodological approaches so that DOL can consider 

its options for proceeding with an evaluation of these grants. 

Our discussion is based primarily on two types of information. The first is publicly available 

information about the grants, which comes from the announcements of the availability of the grant 

funding and, where available, the grant recipients. This information provides insights about DOL’s 

goals for the grants, the main characteristics of the grants, and contractual issues that are important 

to take into consideration when designing an evaluation. Our second source of information is our 

individual and corporate experience conducting a range of evaluations of the workforce investment 

system for DOL, as well as evaluations of programs and grants for other clients. Many of our 

studies have used a randomized controlled design; others have used quasi-experimental designs 

when randomization is not the best option. Although each evaluation is distinctive, in part because 

each program or set of grants is distinctive, we hope that this experience provides a useful 

 3 The main features of the grants are described in Chapter II. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

3 

perspective on the issues that need to be considered as DOL determines how best to move forward 

with its plans for the evaluations. 

C. Organization of the White Paper 

This rest of this white paper is organized into five chapters. In Chapter II, we describe the 

main features of the grants, focusing especially on the features that are relevant for comparing and 

contrasting different approaches to evaluating the grants. Doing so lays an important foundation 

for the discussion in later chapters of particular methodologies. Chapter III continues the 

foundation-building, but with a focus on issues that would influence any type of approach, such as 

the objectives and types of research questions that could be asked. Our discussion of specific 

methodologies, and the considerations that might influences their appropriateness for use with these 

four ARRA grants, begins in Chapter IV. That chapter focuses on random assignment experiments, 

which are generally thought to be the most rigorous designs, and therefore provide the most 

defensible answers to research questions. In Chapter V, we discuss other types of methodologies, 

which are important to consider if the disadvantages of random assignment are weighed as too large 

for that approach to be desirable. In Chapter VI, we tie together the main points from prior 

chapters in a brief summary. 

Pre-Design Study of the ARRA Grants Mathematica Policy Research 

5 

II. FEATURES OF THE ARRA GRANTS 

In the solicitations for grant applications (SGAs) for the four ARRA-funded grants that are part 

of this pre-design study, DOL provided descriptions of the important features of the grants. These 

features include the goals of the grants; the necessary characteristics of successful grant applicants; 

the target populations for services; the types of services and other activities that could be proposed; 

and others, such as the criteria by which grant applications would be judged. 

Understanding the characteristics of the grants lays a critical foundation for the discussions in 

later chapters, because these characteristics will have a strong influence on the types of questions 

that DOL might want to ask about the grants, the relative merits of different types of approaches 

for evaluating the grants, and, ultimately, which approach or approaches are most appropriate for 

DOL to consider. Therefore, in this chapter, we describe key features of the grants, emphasizing 

those that are most pertinent to designing a sensible evaluation.4 Because information on the grants 

currently is limited, this description is inherently general. Nevertheless, it will provide a foundation 

for understanding the strengths and limitations of different methodological approaches for 

evaluating the grants. 

An overview of the four types of grants is provided in Table II.1. Although the four sets of 

grants share many similar characteristics, they differ by providing funding to different types of 

grantees and, in some cases, by targeting different industries. They also have different target 

populations, such as workers who are unemployed and in need of basic skills development, workers 

dislocated from other failing industries, or those who are already in high-growth and emerging 

industries but in need of updated skills. Because of these differences, we discuss each of them in 

turn, even though they share many characteristics. We begin by describing the ETP grants (Section 

A), followed by the Pathways grants (Section B), the SESP grants (Section C), and the Health Care 

grants (Section D). 

A. ETP Grants 

Of the four types of grants that are part of this study, the ETP grants were the first to be 

awarded. Therefore, available information includes information from the SGAs and additional detail 

provided in DOL’s January 6, 2010, announcement of grant awards.5 (Table A.1 in Appendix A 

contains a list of the ETP grant winners and a few aspects of their grant plans.) In this section, we 

describe the main goals of the ETP grants; their size and duration; the types of organizations that 

could apply for and receive the grants; the service components and allowable activities; the 

partnerships that are expected; the target clients; and expectations of the grantees that might increase 

their suitability for an evaluation. 

 4 Most of the information about the grants comes from the SGAs for each of the grant types. When possible, we 

supplement this information from DOL announcements of awards of grants. 

5 Information on the grant awards is at http://www.doleta.gov/ETA_News_Releases/20091526.cfm and 

http://www.doleta.gov/pdf/ETP_SGA_Award_Summaries_120409.pdf, accessed January 27, 2010. 
Pre-Design Study of the ARRA Grants 

6 Table II.1. Overview of ARRA Grants in the Pre-Design Study Grant Name Goal of Grants Number and Amount of Grants Duration of Grants Entities Eligible for Grants Target Populat

Energy Training Partnership Grants To prepare workers for careers in the seven energy efficiency and renewable energy industries, and related occupations 25 grants awarded, ranging from $1.4 to $5 million each; about $100 million in total Up to 24 months Private nonprofit organizations that are either (1) national labor- management organizations with a local network or (2) statewide or local partnerships Workers affected 

national energy aenvironmental poneeding updated training about enefficiency and renewable energyunemployed work

Pathways Out of Poverty Grants To supply training and placement services providing pathways out of poverty and into employment for seven energy efficiency and renewable energy industries 38 grants awarded, ranging from $2.1 to $8 million each; about $148 million in total Up to 24 months (1) National nonprofit entities with networks of local affiliates, coalition members, or other established partners; or (2) local entities Adults who are unemployed, hig

school dropouts, those who have a criminal record odisadvantaged individuals living areas of high pov

State Energy Sector Partnership and Training Grants To invest in workforce sector strategies that target green industries by encouraging a strategic planning process aligning the governor’s workforce vision, state energy policies, and local and regional training activities leading to employment in energy efficiency and renewable energy industries 34 grants awarded, ranging from $2.5 to $6 million, about $187 million in total Up to 36 months State WIBs, where the term “state” includes the District of Columbia and the U.S. territories Workers who are affected by natio

energy and environmental por who need updtraining related the energy efficiand renewable energy industrieveterans, the unemployed, thowith a criminal record, or indivisuch as at-risk ywho are seeking employment pathways out of poverty 
Pre-Design Study of the ARRA Grants M

Table II.1 (continued) 

 7 Grant Name Goal of Grants Number and Amount of Grants Duration of Grants Entities Eligible for Grants Target Populat

Health Care Sector and Other High-Growth and Emerging Industries To provide training and placement services for workers in certain industries, including health care sector (nursing, allied health, long-term care, health information technology), and other high- growth emerging industries (information technology, advanced manufacturing, wireless and broadband deployment, and biotechnology) 45 to 65 grants are expected to be awarded, ranging from $2 to $5 million, about $220 million in total Up to 36 months Public entities or private nonprofit entities, such as LWIBs, tribal organizations, labor organizations, health care providers, education or training providers, and faith- based community organizations Workers who are unemployed; dislocated; or nee

training to secure time employmentadvance in their careers, or retain current occupatioThis last categoryincumbent workeincludes low-wagpart-time worker

Sources: Solicitations for grant applications (SGAs) for each grant type (DOL 2009a, DOL 2009b, DOL 2009c, DOL 2009d, DOL 2009

DOL 2009h). ARRA = American Reinvestment and Recovery Act of 2009; LWIB = Local Workforce Investment Board; WIB = Workforce Investment Board. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

8 

1. Goals of the Grants 

In addition to providing economic stimulus to the economy, the ETP grants are intended to 

prepare workers for careers in any of the seven energy efficiency and renewable energy industries 

and related occupations.6

 These industries include: (1) the energy efficient building, construction, 

and retrofit industries; (2) the renewable electric power industry; (3) the energy efficient and 

advanced drive train vehicle industry; (4) the biofuels industry; (5) the deconstruction and materials 

use industries; (6) the energy efficiency assessment industry serving residential, commercial, or 

industrial sectors; and (7) manufacturers that produce sustainable products using environmentally 

sustainable processes and materials. 

The green occupations for which workers can be prepared might be in a range of different 

areas. Specific examples given in the SGA include transportation, green construction, 

environmental protection, sustainable agriculture, including healthy food production, forestry, and 

recycling and waste reduction. Furthermore, grantees can propose to provide training for emerging 

green occupations that were not explicitly listed in the grant announcement. 

2. Size and Duration 

The SGA specified DOL’s intention to award 20 to 30 grants, ranging from $2 to $5 million 

each for about $100 million in total. DOL expected that about $25 million of the total funding for 

this type of grant would be reserved for projects serving communities that were included in a list of 

312 communities that have been affected by auto industry restructuring. The grant period could be 

up to 24 months. 

Based on the announcement of grant winners, 25 grants have been awarded, with total funding 

of $99,760,688. The value of each grant ranges from $1.4 to $5 million. Sixteen of the grant 

recipients included service provision to communities that are affected by auto restructuring as part 

of their plans. 

3. Eligible Grantees and Grant Recipients 

Eligible ETP grant applicants must be private nonprofit organizations that fall into one of two 

categories: (1) national labor-management organizations with local networks or (2) state or local 

nonprofit partnerships. Neither type of grantee can receive both an ETP grant and a Pathways out 

of Poverty grant, although an organization that receives an ETP grant may participate as a partner in 

a Pathways out of Poverty grant. 

National Labor-Management Organizations. A national labor-management organization is 

a nonprofit entity, such as a training fund, training trust fund, or an education trust fund, for which 

both employers and labor organizations participate on the executive board or comparable governing 

body. In most cases for this category of grant recipient, a national labor-management organization 

would be the official grant recipient. However, the official grant recipient can be a labor 

 6 In the SGA, DOL provided broad definitions of both ‘‘energy efficiency’’ and “renewable energy.” The former 

pertains to programs aimed at mitigating the use of energy, reducing harmful emissions, and decreasing overall energy 

consumption. The latter pertains to electric energy generated from solar; wind; biomass; landfill gas; ocean; geothermal; 

municipal solid waste; or new, efficient hydroelectric generation capacity. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

9 

organization in cases in which a national labor-management organization is not a distinct legal entity. 

For all grantees of this type, however, strong involvement by both the employer(s) and labor 

group(s) is expected in both the administration and governance of the ETP grant. 

National labor-management grantees are required to offer grant services through state, local, or 

regional networks affiliated with the grant recipient. DOL expected that each grantee in this 

category would fund from two to five subgrants or subcontracts to the affiliates that would deliver 

training and other grant services to participants. 

Statewide and Local Entities. According to the SGA, grant recipients in this category of 

grantee would be statewide or local private nonprofit entities, including private nonprofit entities 

that serve American Indian reservations, with a joint partnership of labor organizations, employers’ 

or industry organizations, Workforce Investment Boards (WIBs), and One-Stop Career Centers.7

 A 

grant recipient in this grantee category cannot receive a subgrant or subcontract funding from a 

grant recipient under the national labor-management category. 

Grant Recipients. Of the 25 grant awards made in early January 2010, 7 are to national labormanagement organizations and the remaining 18 are to state or local organizations. Collectively, the 

grants include 30 states and the District of Columbia, and many of the grantees plan to provide 

services throughout an entire state or to a large number of cities or counties within the state. 

4. Partnerships to Be Formed 

 In structuring the grants, DOL placed a strong emphasis on ensuring that grantees 

developed cohesive partnerships of stakeholders in the community, thus fostering a greater 

likelihood of success in the grant endeavors. As such, the SGA specified mandatory partners for the 

grants, as well as suggested additional partners. In addition to including labor organizations and 

employers’ or industry organizations, both types of grantees needed to demonstrate partnerships 

with the local WIBs and One-Stop Career Centers. They also were encouraged to include in their 

grant activities (1) the education and training community; (2) apprenticeship offices, such as DOL’s 

Office of Apprenticeship or the state apprenticeship agency; (3) a state office that focuses on energy, 

weatherization, environmental protection, utility boards, and other agencies with experience in the 

renewable energy and energy efficiency industries; and (4) faith- and community-based 

organizations. Furthermore, grantees were encouraged to include in their grant plans other 

organizations implementing projects funded by ARRA to create jobs in the energy efficiency or 

renewable energy industries, regardless of the Federal agency through which the funding was 

delivered. 

5. Target Clients 

DOL listed several categories of workers that would be eligible for ETP-funded training and 

other services. These include workers affected by national energy and environmental policy, 

workers needing updated training about energy efficiency and renewable energy, and unemployed 

workers. In addition to these, grantees would be allowed to serve individuals with criminal records, 

 7 Throughout this chapter, the term state means each of the 50 states of the United States, the District of Columbia, 

and U.S. territories. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

10 

individuals with disabilities, and groups not traditionally employed in construction and skilled trades, 

such as women and minorities. 

Taken together, grant winners proposed providing services to the full range of allowable target 

groups and to serve at least 34,000 participants (Table A.1 in Appendix A).8

 For example, the 

United Auto Workers-Labor Employment and Training Corporation (UAW-LETC) plans to 

provide services to 430 veterans, ex-offenders, individuals with disabilities, or women in Missouri. A 

grant provided to the State Labor Management Cooperation Committee for the International 

Brotherhood of Electrical Workers and the National Electrical Contractors Association (LMCCIBEW-NECA) will serve 2,292 unemployed and underemployed electricians in California, and a 

grant provided to Blue Green Alliance will provide services to dislocated workers, women, 

minorities, veterans, incumbent workers, and unemployed steelworkers in Minnesota. The Oregon 

Manufacturing Extension Partnership will provide services to 1,670 unemployed and dislocated 

workers and employed workers at local renewable energy industries, such as the renewable electric 

power and biofuels fields. Other grant recipients include workers with a wide range of backgrounds 

and characteristics. 

6. Services and Activities Allowed with Grant Funds 

Grant applicants were encouraged to propose program models that were already identified 

through prior evidence as effective with the grantee’s target populations, including strategies for 

recruiting and training participants, providing supportive services to help them complete grantsponsored training, placing them in jobs, and helping them to retain their employment.9

 

One potential benefit of using existing, promising approaches for service delivery, rather than in 

the development of new approaches, presumably is a greater likelihood of improved post-training 

outcomes for participants. In addition, it could help to ensure a shorter start-up period before 

participants could begin receiving services, thus bolstering the economic stimulus benefits of the 

grants. 

The training activities are to provide workers with the skills and competencies needed by the 

targeted industries and to support participants’ advancement along a career pathway. Occupational, 

on-the-job, and customized training in the seven energy efficiency and renewable energy industries 

listed previously could be funded by the grant. Although wage subsidies are not allowed, grantees 

could include as part of their plans paid work experience activities, registered apprenticeship 

activities, or pre-apprenticeship strategies so that participants could gain occupational skills while 

having earnings, if this experience will lead to permanent employment. Because DOL wanted to 

ensure that the training is accessible to participants, grantees could use distance learning or 

technology-based learning. Finally, the training is to lead to an industry-recognized degree or 

certificate. 

 8 The information that is available from the grant announcement does not allow a full tally of the number of 

participants that grantees plan to serve. In some cases, the announcement provides the number of participants that 

grantees expect will participate in training. In other cases, the announcement provides information on the number of 

participants that grantees expect will be placed in jobs after training is completed. Thus, if all grantees meet their 

projected targets of enrollments and outcomes, the number of participants served is likely to be larger than 34,000, 

which was calculated as a lower bound given the available information. 

9 The SGA specified that grantees could use up to five percent of grant funds for supportive services. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

11 

In addition to providing training, grantees could propose using a portion of grant funds for 

activities that support the provision of training and attainment of credentials in the target industries. 

Examples given in the SGA include education or training for staff that will provide training or 

registered apprenticeships, the development or modification of curricula to deliver training, and the 

development or modification of apprenticeship standards and procedures for defining and issuing 

credentials within the energy-efficiency occupations. Because of DOL’s desire for grant services to 

be provided quickly, a grantee needed to demonstrate why these developmental activities were 

valuable for the success of the grant. 

Finally, successful grant applicants will need during the grant period to develop a plan for 

sustaining the activities after the grant is over. 

7. Suitability for an Evaluation 

The SGA specified that one component of the criteria used to determine grant recipients is the 

applicants’ suitability for evaluation, and grant applicants were asked to demonstrate their 

suitability.10 To receive points in this area, grantees needed to describe their recruitment plan that 

could yield a large number of qualified applicants for the program, including more applicants than 

the number of positions available. This consideration was included in the SGA to allow DOL the 

option of using a random assignment evaluation design to study the impacts of the grant activities 

on participants’ outcomes, because the presence of excess demand for services would allow the 

formation of a control group in this type of evaluation. Grant applicants also were to describe (1) 

their ability to collect participant-level information on who applies to participate; (2) their retention 

strategies that would minimize client attrition and help researchers track participants; and (3) their 

willingness to work collaboratively with outside researchers conducing rigorous, independent 

research. Furthermore, grant applicants were to provide an explanation for why funding of their 

proposed plans would enhance knowledge about ways to effectively serve individuals and 

communities, including those not directly affected by the proposed program. These aspects of grant 

applicants’ plans would be important regardless of the type of evaluation design that might be used. 

Grantees are required to collect participant-level data, which will be useful both for the 

monitoring of the program by DOL staff and a potential evaluation of the grants. Although the list 

of data items about participants who receive services through the ARRA grants is still being 

developed, the broad categories of information to be collected include (1) demographic and 

socioeconomic characteristics, (2) employment history, (3) services provided, and (4) outcomes 

achieved. This individual-level information could be used to support evaluation efforts to 

understand who participates in the grant activities and to assess what participants’ post-training 

outcomes are.11 Furthermore, because grantees will need to collect Social Security numbers from 

clients, it will be possible to link the grant-specific individual-level data to unemployment insurance 

 10 The suitability for evaluation counted as five points out of a total of 100. 

11 This information will form the basis of aggregate statistics that are reported to DOL on a quarterly basis. Grant 

winners are required to track outcomes about the number of participants (1) that are served; (2) that begin training; (3) 

that complete training; (4) that complete training and receive a degree/certificate; (5) that complete training and are 

placed into unsubsidized employment; (6) that complete training and are placed into training-related unsubsidized 

employment; and (7) that are placed in unsubsidized employment and who continue to be employed in the first and 

second quarters after the initial job placement. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

12 

(UI) administrative wage data, which can serve as another source of information about participants’ 

pre- and/or post-training employment and earnings. 

B. Pathways Out of Poverty Grants 

The Pathways Out of Poverty grants are similar in many ways to the ETP grants (see Table 

II.1). Both types of grants focus on training workers for the seven energy-efficient and renewable 

energy industries and occupations listed above. Both are for two years and have two categories of 

grantees. Applicants to both grant types were encouraged to collaborate with other efforts that are 

intended to foster the “greening” of the economy, regardless of whether they are funded by ARRA 

or another source. Furthermore, the SGAs for each grant type had a similar emphasis that the 

grantees demonstrate their suitability for, and willingness to cooperate with, an evaluation. 

However, there are some important differences between the two sets of grants, including their 

funding, the size of the grants, and the types of eligible grant applicants; the partners to be included 

in the grants; the target communities and clients, and the expectations for services that will be 

offered. Information discussed in this section comes both from the SGA and DOL’s January 13, 

2010, announcement of grant winners.12 A summary of key features of grant recipients is in Table 

A.2 in Appendix A. 

1. Grant Awards, Funding, and Grantee Types 

Whereas the approximate total funding for the ETP grants is $100 million, the SGA for the 

Pathways grants specified that about $150 million in funding would be available. The 38 grant 

awards, announced January 13, 2010, totaled close to this amount, at $147,757,701. 

Two types of grantees were specified in the SGA: (1) national nonprofit entities with networks 

of local affiliates or partners or (2) local entities. Grants awards for the first type, the national 

entities, range from $3 to $8 million; grant awards for the second type, local entities, range from $2 

to $4 million. Of the 38 awards, 8 went to national grantees (with a total grant value of about $48 

million) and 30 to local ones (with a total value of about $100 million). Collectively, the planned 

activities of the grant recipients are expected to take place in 27 states, the District of Columbia, and 

Puerto Rico. 

National entities are defined as private nonprofit organizations that deliver services through 

networks of local affiliates, coalition members, or other affiliated partners, such as community- or 

faith-based organizations. Furthermore, the local affiliates must have the ability to provide services 

in four or more states. National grantees are expected to implement projects that serve 

communities located in at least two states and a range of three to seven communities served in 

total.13 Similar to the ETP national labor-management grantees, Pathways grantees who receive a 

 12 Information about DOL’s announcement about grant winners can be found at http://www.doleta.gov/ETA 

News_Releases/20100039.cfm, and http://www.doleta.gov/pdf/Pathways_Poverty grants.pdf, accessed January 27, 2010. 

13 A community was defined as a geographic area located within one or more contiguous Public Use Microdata 

Areas (PUMAs), which are geographic statistical areas designated by the U.S. Census Bureau. For urban areas, these will 

be neighborhoods within a city rather than the entire city. For rural applications, it is likely that designated communities 

will be from one to three counties, American Indian areas, Alaska Native areas, or Hawaiian homelands. Although 

communities do not have prespecified sizes, it was expected that they would contain between 10,000 and 100,000 

people. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

13 

grant under the category of national entities are required to fund subgrants or subcontracts to local 

affiliates and their partners, each of which will implement the project in its designated community. 

Of the eight grantees that are national grantees, most chose to serve communities in three or more 

states, with one planning to serve communities in six states. 

The second type of grantee, local entities, are public organizations or private nonprofit 

organizations whose service area is limited to a single substate geographic area, such as a 

neighborhood, city, county, substate region, or interstate region comprised of multiple substate 

regions (such as Kansas City). According to the SGA, examples of potentially eligible local entities 

include entities such as community colleges, WIBs, community- or faith-based organizations, and 

tribal governments. The range of grant winners reflects this diversity. Local grantee applicants must 

propose serving a single, substate community.14 

Neither type of recipient of a Pathways grant can receive an ETP grant. However, a Pathways 

grant recipient may participate as a partner in an ETP grant. 

2. Partners 

The list of mandatory partners for the Pathways grants is slightly different from that for the 

ETP grants. In each community served, regardless of grantee type, the partnership must include at 

least one entity from each of the following five categories: (1) nonprofit organizations, such as 

community- and faith-based organizations; (2) the public workforce investment system; (3) the 

education and training community; (4) public and private employers and industry-related 

organizations; and (5) labor organizations. By requiring that all of these types of organizations are 

represented in a partnership, DOL expected that grantees could take advantage of the existing 

resources within the community and that participants will be able to access a wide range of services 

that would help them to complete training, overcome barriers to employment, obtain jobs, advance 

along career pathways, and achieve economic self-sufficiency. 

In addition to the required types of partners, grant applicants were strongly encouraged to 

include other partners that could contribute to the project, such as public housing agencies; 

community action agencies implementing the weatherization assistance programs; other 

organizations implementing ARRA-funded projects to support jobs in the energy efficiency or 

renewable energy industries; and foundations and other social service organizations that assist likely 

project participants. 

3. Target Communities and Clients 

Projects funded through the Pathways grants are to provide an integrated set of training and 

supportive services so that the target populations can exit poverty and gain economic self-sufficiency 

through employment in the targeted energy efficiency and renewable energy industries. Pathways 

grant recipients are planning to serve adult participants who have a range of disadvantages that 

might affect their labor market success. These disadvantages include (1) unemployment or 

underemployment, (2) the lack of a high school diploma or General Educational Development 

(GED) certificate, (3) the presence of a criminal record, (4) limited English proficiency or recent 

 14 A local entity that receives an award under this SGA may not receive subgrant or subcontract funding through a 

grant awarded to a national entity under this SGA. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

14 

immigration or refugee status, and (5) residence in an area of high poverty. Other target groups 

include individuals who are dislocated from their prior employment, who receive public assistance 

recipients, or who are homeless. At least 90 percent of clients must live within the target 

communities to be served, which need to have poverty rates of at least 15 percent. However, up to 

10 percent of clients may live outside of the target communities if they live in another high-poverty 

area. 

Taken together, the Pathways grant recipients proposed providing services to the full range of 

allowable target groups described earlier (Table A.2 in Appendix A). Although the precise number 

that they plan to recruit and serve is not available, the number is likely to exceed 20,000.15

4. Services and Activities Allowed with Grant Funds 

The services and activities that are expected to be offered with the Pathways out of Poverty 

grants are similar, but not identical, to those to be offered through the ETP grants. The activities 

will include recruitment and referral; training for careers in the energy-efficiency and renewable 

energy sectors; and job search, placement, and retention services. As with the ETP grants, 

applicants could propose occupational, on-the-job, and customized training activities, as well as 

registered apprenticeship and pre-apprenticeship programs. In addition, Pathways grantees can 

propose using grant funds to update training curricula if doing so is necessary to the effective 

provision of training. However, because of the different types of target populations for the two 

types of grants, these grantees—compared with the ETP grantees—are more likely to need to 

provide basic skills training to their participants as part of the package of services. Pathways 

grantees are allowed to provide basic skills training, such as adult basic education, English as a 

second language (ESL), and job readiness training; initial assessments of skill levels, aptitudes, 

abilities, and supportive service needs; and case management services. In addition, they will be 

allowed to use up to 10 percent of their grant funds for the provision of supportive services, which 

can help participants to complete training and overcome barriers to employment. In contrast, ETP 

grantees will be able to use up to five percent only for supportive services. 

As with the ETP grants, Pathways grant applicants were encouraged to propose program 

models that have already been shown to be promising. DOL encouraged Pathways grantees to 

consider integration of occupational skills training with basic skills and work-readiness training, 

when needed, for low-income young adults. Grant applicants who intend to propose serving high 

school dropouts were encouraged to consider program models that link work-based learning and 

occupational skills training in the targeted industries with basic literacy and mathematics skills 

training to help participants obtain a high school diploma or GED. Those who intend to propose 

serving ex-offenders were encouraged to consider programs that, by integrating services both before 

and after release from prison or jail, have had positive impacts on employment outcomes. Finally, 

applicants intending to serve other types of disadvantaged individuals were encouraged to consider 

providing on-the-job training with a specific employer who agrees to hire individuals who 

successfully complete the training. 

 15 See footnote #7 for an explanation of why the precise number is not available. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

15 

C. State Energy Sector Partnership Grants 

Like the ETP and Pathways grants, the SESP grants have a strong emphasis on investments in 

strategies that target energy efficiency, renewable energy, and other green industries (see Table II.1). 

And, like the other grants, an important component of the SESP grant is the provision of training to 

provide workers with the skills and credentials necessary to obtain and retain employment, and to 

progress along career pathways in these industries. 

However, the SESP grants were designed to highlight and foster the important role that states 

can play in these workforce sector investments and the transition to a more green economy. In this 

section, we describe some of the distinctive features of the SESP grants, which might influence the 

way in which an evaluation of these grants might be designed. 

Information discussed in this section comes from the SGAs and DOL’s January 20, 2010, 

announcement of grant awards.16 A summary of key features of grant recipients is in Table A.3 in 

Appendix A. 

1. Size and Duration of the Grants, Eligible Grantees, and Partners 

On January 20, 2010, DOL announced the provision of $186,908,818 for the SESP grants. The 

34 grants range from $2.5 million to $6 million. When the SGA was issued in June 2009, DOL 

expected to reserve some funds for communities affected by auto restructuring. (The SGA 

provided a list of 312 such communities.) In fact, 21 of the 34 grants included at least one 

community affected by auto restructuring. Although these grants total more than $120 million, it is 

not clear what portion of the grant funds will be targeted to these communities or to other 

communities within the states. Unlike the ETP and Pathways grants, which are expected to last up 

to two years, the SESP grants are for three years. 

Eligible grantees consisted of the state-level WIBs of the 50 states, the District of Columbia, 

and the U.S. territories, although in practice all grant recipients were states and not nonstate 

jurisdictions. Grantees are required to partner with the state workforce agency, local WIBs or 

regional consortia of WIBs, and the One-Stop Career Centers within the state. 

2. The SESP-Guided Energy Sector Strategy 

Compared with the other ARRA-funded grants included in this pre-design study, the SESP 

grants have a distinctive component that focuses on the development or refinement, and ultimate 

implementation, of a statewide energy sector strategy, which is a workforce development approach 

that targets the needs of the energy sector. Furthermore, as part of their grant applications, state 

WIBs needed to include a charter for an SESP. The charter is to describe the SESP’s purpose, 

goals, and functions; broadly speaking, however, the SESP’s mandate will be to serve as a steering 

committee for the state’s energy sector strategy. Ultimately, the SESP also will be responsible for 

ensuring that the specific SESP grant endeavors are successful.17 

 16 DOL’s announcement of grant recipients can be found at http://www.doleta.gov/ETA_

News_Releases/20100078.cfm and http://www.doleta.gov/pdf/SESP_Summaries.pdf, accessed January 27, 2010. 

17 SESP grant recipients will need to submit to DOL a sustainability plan prior to the end of the grant. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

16 

In encouraging the development and implementation of a state energy sector strategy through 

the SESP, DOL stressed the importance of having a comprehensive set of organizations and groups 

involved. The SESP is to be structured so that all of the grant partners will have an integral role in 

the strategic planning process that the SESP undertakes. Furthermore, the energy sector strategy is 

to be designed and implemented by a range of organizations and groups working collaboratively, 

including community- and faith-based organizations, business and industry groups, educational 

institutions such as community and technical colleges, the public workforce system, labormanagement partnerships, and others. This statewide planning is to align with the governor’s 

overall workforce vision, state energy policies, and feasible training activities that lead to 

employment in the targeted energy sectors. 

In addition to the SESP charter, grant applicants had to include a list of local and regional 

project teams in their applications for funding. In addition to providing input about the strategy at 

the local or regional level, the purpose of these teams is to provide services to individuals as part of 

the grant efforts. Specific activities of these teams include the recruitment of training participants; 

the development and use of linkages with employers and training providers; and the provision of 

other services (such as case management, job retention, and supportive services) to support the 

success of the grant endeavors. As with the other types of grants, training can be in a variety of 

forms, such as on-the-job training blended with classroom training, customized training with an 

existing registered apprenticeship program or labor-management partnership, technology-based 

learning, or other appropriate training strategies. 

3. Participants Eligible to Receive Training 

Although much of the grant-funded provision of services directly to workers will be guided by 

the SESP energy sector strategy, ETA provided guidance in the SGA about which groups of 

workers are most appropriate for grant-funded activities. The local and regional teams that are to 

provide training and other services funded by the SESP grants must give priority to the following 

target populations: (1) workers affected by national energy and environmental policy; (2) individuals 

in need of updated training related to the energy efficiency and renewable energy industries; (3) 

veterans; (4) unemployed individuals; (5) individuals, including at-risk youth, seeking employment 

pathways out of poverty and into economic self-sufficiency; and (6) individuals with criminal 

records. Other individuals, such as entry-level and incumbent workers and individuals with 

disabilities, also may be served through the SESP grants. 

Collectively, the SESP grant winners plan to provide services to this wide range of target groups 

(Table A.3 in Appendix A). Based on their proposals, the grantees are likely to serve at least 54,000 

participants.18

D. Health Care Sector and Other High-Growth and Emerging Industries 

Grants 

As the title of this set of grants clearly indicates, these grants focus on training workers in the 

health care sector and other high-growth and emerging industries (see Table II.1). This industry 

focus makes this set of grants distinctive among the four sets that are part of this pre-design study. 

 18 See footnote #7 for an explanation of why the precise number of participants is not available. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

17 

Despite this difference, however, many of the basic features of the Health Care grants are 

similar to other grants. The Health Care grants’ allowable activities and services, for example, have a 

similar focus as the Pathways grants, because both types of grants might be especially likely to serve 

participants in need of comprehensive services. Grantees will be expected to work with participants 

from the time they are recruited, through training, job placement, and job retention. Grantees were 

strongly encouraged to propose providing training services that would lead to portable employer- or 

industry-recognized certificates or degrees and enable participants to progress along a career 

pathway or lattice. Furthermore, as with other grant types, the training could be in a variety of types 

(such as through Registered Apprenticeships, classroom training, on-the-job training, or 

customized). Applicants were encouraged to use existing training curricula but could propose 

modifying existing curricula if a strong justification for doing so was given. As needed, grantees 

could offer initial assessments, basic skills training, English as a Second Language (ESL), job 

readiness training, and case management. In addition, they could use up to 10 percent of grant 

funds for supportive services to assist participants as they receive other services. And, grant 

applicants were encouraged to propose program models that had been found to be successful for 

participants who are similar to the applicants’ target groups. 

In addition to the similarities with other grant types in terms of the allowable activities and 

services, Health Care grantees were encouraged to try to coordinate with other ARRA programs, 

and applications are being assessed, at least in part, for their suitability for evaluation. 

As with our discussions about other grants, therefore, we focus the rest of our discussion in this 

section on the distinctive aspects of this set of grants. 

1. Goals of the Grants 

As noted earlier, the Health Care grants are intended to help workers receive training and job 

placement services in certain industries, including the health care sector and other high-growth 

emerging industries. Examples of health care fields for which individuals might be trained include 

nursing, allied health, long-term care, and health information technology. If an applicant is 

interested in providing services for a high-growth or emerging industry other than health care, DOL 

encouraged grant applicants to define the industry within the context of their state or regional 

economy; to be considered a high-growth or emerging industry in a local area, the grant applicants 

would need to demonstrate that the industry is projected to add a large numbers of new jobs to the 

economy, that technology and innovation require new skill sets for workers, or more generally that it 

is a new and emerging industry projected to grow. As examples of emerging industries, the SGA 

mentioned information technology, advanced manufacturing, wireless and broadband deployment, 

and biotechnology. 

2. Size and Duration 

As of the SGA release date, DOL expected to provide a total of about $220 million in funding, 

through 45 to 65 grants that would range from $2 to $5 million. About $125 million of the funds 

were to be reserved for projects dedicated to the health care sector. About $25 million was expected 

to be reserved for projects serving communities affected by auto industry restructuring. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

18 

3. Eligible Grantees 

In contrast to the ETP and Pathways grants, each of which specified more than one category 

for grant applicants, the Health Care grants did not distinguish among grant applicants on the basis 

of their organizational structures. Eligible applicants for the Health Care grants include public 

entities or private nonprofit entities, such as LWIBs, tribal organizations, labor organizations, health 

care providers, education or training providers, and community- and faith-based organizations. 

4. Partnerships to Be Formed 

As with other grant types, DOL emphasized the importance of strong partnerships that would 

ensure that grant winners would be able to provide a comprehensive set of services to participants. 

Winning grant applicants need to include at least one entity each from (1) the public workforce 

investment system; (2) public and private employers, such as health care providers when appropriate, 

and industry-related organizations; and (3) the education and training community. DOL also 

encouraged applicants to include other partners, such as nonprofit organizations, labor 

organizations, entities implementing related projects funded by ARRA, foundations, and state and 

local social service agencies. 

5. Target Clients 

Health Care grantees will be allowed to serve individuals who qualify for services based on a 

wide range of characteristics. Target groups of participants include unemployed workers; dislocated 

workers; and incumbent, low-wage workers or those who need training to secure full-time 

employment, advance in their careers, or retain their current occupations. In addition, grantees can 

target individuals on public assistance, high school dropouts, individuals with disabilities, veterans, 

and individuals with limited English proficiency. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

19 

III. POTENTIAL OBJECTIVES AND CONSIDERATIONS OF AN EVALUATION 

The design of an evaluation is intricately linked to the study objectives and research questions 

of interest. As such, it is useful to discuss broadly the potential objectives and research questions of 

an evaluation of the four sets of ARRA grants that are included in this pre-design study before 

turning to the details of the designs, because the design that is most appropriate will depend on the 

research question(s) of interest. 

In order to identify the objectives of an evaluation, we first recall the goals of the ARRA grant 

funding. Each grant type is designed with specific goals in mind: to provide workers in need of 

career development or retooling with the skills and credentials necessary to obtain and retain 

employment. With the ETP, Pathways, and SESP grants, the focus is on employment in seven 

energy efficiency and renewable energy sectors; the remaining grant focuses on health care and other 

high-growth and emerging sectors. The grant types also have target populations in mind. For 

instance, three grant types (ETP, SESP, and Health Care) have specific monies set aside to target 

individuals affected by auto industry restructuring. Pathways grants are designed to move a highly 

disadvantaged population (including high school dropouts and individuals with criminal records) out 

of poverty and into employment. 

The grants may also provide macroeconomic benefits, such as economic stimulus during a 

recessionary period and aiding in the expansion of target occupations and industries, particularly 

those in the energy efficiency, renewable energy, and health care fields. Although the grants support 

these efforts, their main focus is training workers for jobs, and grant funds cannot be used for direct 

job creation. 

With the main grant goals in mind, in this chapter we present a series of potential objectives 

that may be considered in the evaluation of ARRA grants (Section A). Because the grants are 

primarily intended to train and obtain employment for individual participants, we then discuss some 

of the considerations that must be taken into account for any design that aims to assess participantlevel outcomes (Section B). 

A. Potential Objectives of an Evaluation 

We consider three possible objectives of an evaluation that could be of interest to DOL: (1) 

learning about the microeconomic effects of training on workers, which examines outcomes at the 

individual level, (2) understanding the macroeconomic effects of the temporary funding stream in 

funded communities, which examines outcomes at the community level, and (3) examining 

implementation and operational issues involved in deploying ARRA grants. Although DOL has 

expressed special interest in the first of these (evaluating the microeconomic effects of training on 

workers), we also discuss the other possibilities because they could provide valuable information and 

could be done either in concert with an analysis at the individual level or instead of such an analysis. 

1. Learning about the Microeconomic Effects of Training on Workers 

If the evaluation is to determine whether training has an effect on the outcomes of workers, an 

approach in which data are gathered and analyzed at the individual-worker level is appropriate. In 

the context of ARRA grants, such an evaluation could answer the following research question: 

“What was the impact of access to ARRA grant funding on participants’ outcomes?” The results of 

this type of evaluation will need to be put in the context of the unique environment in which ARRA 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

20 

grants are awarded, including high unemployment and severe economic dislocation in some parts of 

the country. However, in addition to answering the specific question of what impact the ARRA 

grants had given the current economic circumstances, the results from a study that aims to assess the 

effect of training on workers can more generally provide guidance about how policy initiatives can 

be shaped in the future, even after the economy recovers. The issues related to the selection of 

outcomes and potential sources of data for a study conducted at this level are discussed in detail in 

Section B. 

2. Understanding the Macroeconomic Effects of the Temporary Funding Stream 

Although DOL has expressed interest primarily in the effects of ARRA-funded training on 

workers, examining the effects of the ARRA grants in funded communities may be informative, 

given that the grants support two macroeconomic goals: economic stimulus and development of the 

green economy through training of workers. Such an evaluation would answer the following 

research question: “What was the impact of ARRA grant funding on the macroeconomic conditions 

in communities that received grants?” 

If the evaluation is to determine whether ARRA grants more generally had an effect on 

macroeconomic health, then an approach in which data are gathered and analyzed at the community 

or local labor market level is appropriate. An analysis of effectiveness at the macroeconomic level 

would examine broader trends in the local economies that received grants. For instance, the 

evaluation could examine outcomes such as the unemployment rate and the job creation rate in 

green industries or health care industries, depending on the focus of each grant type. Given the 

diversity in the specific types of industries and occupations that the grants target, care would be 

needed to ensure that the outcome measures that are chosen for examination are pertinent and 

available for each grantee. Chapter V discusses some methodologies that could be used to examine 

these types of outcomes. 

3. Examining the Implementation and Operational Issues Involved in Deploying ARRA 

Grants 

Learning about the experiences of grantees as they implement ARRA grants could provide 

valuable information both to policymakers and service providers nationwide as future efforts build 

upon the work of ARRA grantees. Even seasoned service providers in the workforce investment 

system are likely to be challenged as they implement programming funded by ARRA grants; the 

grants offer states and communities the opportunity to provide training and other services to help 

workers transition to careers in high-demand, emerging industries not previously targeted within the 

workforce investment system. Among these challenges are establishing or enhancing their 

partnerships and processes; developing or modifying curricula and certification standards in 

emerging occupations; recruiting and serving participants; or other facets of grant operations. 

In addition to examining the general challenges that ARRA grantees are likely to face in 

implementing their grant activities, two facets of the grants are especially well suited for an 

implementation evaluation. The first facet is grantees’ development or modifications of products to 

assist in training activities. Because many of the grant types target emerging industries, some 

grantees plan to develop or modify training models and the curricula used in training to 

accommodate the focus on these new industries. Pathways grantees, which target a highly 

disadvantaged population, may also develop innovative remediation techniques to improve the skills 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

21 

of individuals they serve. An implementation study would be an appropriate way to examine 

grantees’ efforts to develop, modify, and implement these new curricula. 

The second facet of the grants that is well suited to an implementation evaluation pertains 

exclusively to the SESP grants. Examining operational issues might be especially important for 

these grants, which have a strong emphasis on the development and implementation of a statewide 

energy sector steering committee. The purpose of this committee is to guide the development (or 

refinement) of the state’s energy sector strategy both during and after the end of the ARRA grants. 

The techniques that are used in an implementation study are the most appropriate ones to assess the 

effects of this steering committee. 

Even if an evaluation of the ARRA grants does not have these implementation issues as its 

primary focus, an implementation study could be an important component of any evaluation of the 

grants. A detailed implementation study could collect data on services received by participants, 

where they were received, the duration of service receipt, and so on. This information could be used 

to interpret any findings about the effects of the grants. For example, if no micro- or 

macroeconomic effects are found, then one could use information about the services received by 

participants to try to gain insights about why that is the case. Even when beneficial effects are 

found, implementation information is useful for gaining insights about activities that might need to 

be duplicated in the future to achieve similar results, and which aspects could be enhanced. 

An implementation study typically involves the collection and analysis of data through 

discussions with or surveys of grantee staff, observations of grant activities, discussions with and/or 

surveys of participants, or other methods. Although an implementation study is likely to be at least 

one component of an evaluation of the ARRA grants, focusing on this type of analysis is outside of 

the scope of this pre-design study. The details of an implementation study are likely to depend 

greatly on whether or how DOL pursues the other objectives for an evaluation, which we discuss in 

this white paper. 

B. Considerations for a Study of the Effect of Training on Individuals’ 

Outcomes 

Because DOL is likely to desire an approach examining outcomes at the individual level, we 

focus here on two key considerations that guide the design and interpretation of this approach: (1) 

understanding the types of activities in which individuals in the control or comparison groups 

engage and (2) recognizing factors that may influence what the grants can accomplish. In this 

section, we discuss each of these in turn. We conclude by discussing some outcomes that could be 

examined, given these important considerations. 

1. Need for an Understanding of the Counterfactual 

Answering a question about the impact of the grant on individual outcomes relies on some type 

of comparison of the outcomes of individuals who receive ARRA-funded training with those who 

do not. Regardless of the specific design used (for example, random assignment or comparison 

group), it is essential that an evaluation includes a firm understanding of the types of activities in 

which individuals in the control or comparison group (known as the “counterfactual”) engage. For 

this particular evaluation, it is important to note that a great deal of funding is currently available for 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

22 

training through a number of sources. In addition to the four ARRA grant types, sources such as 

One-Stops may have additional funding for training.19

The activities the nonparticipants engage in strongly influence the research questions that can 

be examined. To illustrate this, consider two extremes. In the first, individuals in the comparison 

group receive no training from other sources; this would enable an evaluation of the impact of 

ARRA-funded training compared with no treatment at all. The second is the opposite case, in 

which all members of the comparison group received the same amount of training as ARRA 

participants but through a different source, such as regular WIA formula funding. Then an 

evaluation of the impact of ARRA-funded training on employment outcomes, such as employment 

status and quarterly earnings, would actually be an evaluation of the impact of ARRA-funded 

training relative to other sources of training funds. 

Depending on how similar these alternative training services are to the training funded by 

ARRA grants, the evaluation might not be capable of identifying the differential effects of ARRAfunded training. If it is likely that members of the comparison group can easily seek comparable 

training elsewhere, one could potentially mitigate this by applying a more narrow definition of the 

outcomes of interest. For instance, one could examine employment status in ARRA-targeted 

industries specifically, rather than employment status in any industry. 

Because the types of individuals targeted by the grants varies considerably, it is important to 

recognize that the counterfactual may vary by the types of individuals receiving services. For 

example, One-stop Career Center customers might get regular (formula-funded) WIA services if the 

ARRA grants are not available. However, the counterfactual for incumbent workers might simply 

be no training at all if the grants are not available. The counterfactual for treatment group members 

recruited from a community-based organization might be some other type of training service 

altogether. 

The considerable variation in the populations and industries targeted by ARRA grants, and the 

services the participants in the comparison or control condition can receive could affect the 

interpretation of the study’s findings, as well as the ability to detect significant impacts. Thus, it is 

important to explore these factors in the design phase of an evaluation in order to construct suitable 

comparison or control groups. 

Finally, it is worth mentioning that members of the counterfactual group face economic 

circumstances that almost certainly differ from those they would face if the grants were deployed in 

a more stable economic environment. This may influence the extent to which different types of 

people seek job training services and, perhaps, which types seek ARRA-funded training versus 

training funded through other sources. Although this limits the extent to which the impact of 

ARRA funding can be generalized to a time of relatively better economic circumstances, it does not 

necessarily indicate that it will be more difficult to detect impacts. 

 19 For example, since October 2009, DOL has announced grants of $447 million in Trade Adjustment Assistance 

funds and $55 million in Green Capacity Building Grants, in addition to many state-specific grant awards. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

23 

2. Factors Influencing What the Grants Can Accomplish 

Any evaluation of ARRA grants will have to be conducted in light of additional factors that 

could influence what the grants can accomplish. First, certain grant types, namely Pathways, target a 

highly disadvantaged population. It might be more difficult and more costly to effect change within 

this group of participants than within other groups of workers. To some extent, all of the grants 

target job training to be used in emerging industries. It is possible that there simply are not enough 

available jobs in these industries to provide employment for many ARRA-funded training 

participants. Moreover, because it is grantees’ first experience working with these industries, it may 

be more difficult to place individuals in those jobs that are available. 

However, the evaluation designs proposed in Chapters IV and V of this paper mitigate 

concerns of these influences on the estimation of program impacts on participants’ outcomes; 

impacts are estimated by comparing the outcomes of individuals who receive ARRA-funded training 

with the outcomes of those who do not, holding all else equal. The control or comparison groups 

are created so that, to the extent possible, their members have similar characteristics and face similar 

sets of job openings as participants. Thus, estimation of program impacts will not be biased by 

these influences. In addition, as discussed in Section A, a qualitative component could supplement a 

quantitative analysis of the ARRA grants’ impacts on participants’ outcomes, providing insights 

about these contextual influences on grant activities and outcomes. 

3. Potential Outcomes to Be Examined 

The outcomes of interest to the evaluation must be selected according to the research questions 

of interest, taking into account the issues surrounding the counterfactual and the targeted individuals 

and industries, as previously described. Studies of the effects of training or employment services 

often include as outcomes employment status and earnings, measured for a period of one to several 

years after enrollment in a program. However, different outcomes for an ARRA grants evaluation 

might be warranted because the grants are specifically targeted toward occupations in certain 

industries and to specific populations of individuals. Therefore, we highlight two types of outcomes: 

(1) general labor market outcomes and (2) outcomes tailored to the evaluation of ARRA grants. 

1. Quarterly earnings and employment status are labor market outcomes commonly 

examined in evaluations of training programs. These outcomes can be available through 

unemployment insurance (UI) wage records linked to participant data using participants’ 

Social Security numbers (SSN); as noted in Chapter II, ARRA grantees are required to 

collect SSNs so that participant information can be linked to these records. 

Although grantees are required to collect some information on participants’ engagement 

in training and other services, it is often the case with studies of the effects of training on 

workers’ outcomes that the utility of the administrative data would be enhanced by the 

use of a follow-up survey; this could be a source of rich, detailed information about 

individuals’ participation in ARRA-funded training and other services as well as the 

characteristics of their post-training jobs. For both the survey and administrative data, 

the follow-up period must be long enough after participants become involved in grant 

activities to observe meaningful training and employment outcomes. Follow-up periods 

for studies that Mathematica has conducted to evaluate training programs often range 

from one to several years. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

24 

2. Targeting grants to occupations in certain industries may warrant examining more 

specific outcomes than those traditionally examined in these types of evaluations. For 

instance, rather than looking at the employment status of individuals who received 

training regardless of the industry in which they found employment, it may be more 

meaningful (and more indicative of program effectiveness) to examine employment and 

earnings in the industries targeted by the grant. For example, this approach would 

suggest examining the employment and earnings in health care and related industries for 

those individuals who received training through Health Care grants; for Energy Training 

Partnership Grants one could examine the employment status in the seven targeted 

energy efficiency and renewable energy industries. 

Targeting grants to certain groups of individuals may also influence the selection of 

outcomes of interest. For example, the Pathways grants are targeted to adults who are 

unemployed, high school dropouts, have criminal records, or are disadvantaged 

individuals and living in areas of high poverty. For these individuals, changes in receipt 

of social services or rearrest rates may be outcomes of interest. For other grant types, 

progression along a career pathway as indicated by job promotion or salary increases may 

be important. 

As mentioned previously, the individuals with whom ARRA grant participants are being 

compared may also receive some types of training. This argues for adopting one or both types of 

tailored outcomes in order to identify program effectiveness. Both of these types of outcomes 

would likely have to be collected through a survey of participants. Some pertinent information 

relating to these outcomes might be available through administrative data or a grant-specific data set. 

However, because survey questions can be tailored to the specific topics of interest, it is likely that 

the quality of the information available through a survey would be higher. 

Regardless of the source of the outcomes data, as well as the methodological approach for 

conducting the evaluation, it will be important to limit the number of outcomes of interest to a small 

number. The likelihood of finding a statistically significant impact estimate when none exists 

increases as the number of examined outcomes increases. Although statistical techniques can be 

used to adjust for the increased likelihood of falsely detecting a statistically significant impact, the 

generally preferred approach is to limit the number of outcomes by which the success of an 

intervention is judged to a small number prior to data collection and analysis. 

 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

25 

IV. RANDOM ASSIGNMENT IMPACT DESIGNS 

DOL has as its goal a rigorous evaluation of the ARRA grants. Although evaluations can take 

different forms, a random assignment evaluation is often referred to as the gold standard in 

evaluation design. When designed and implemented well, random assignment evaluations can 

provide very convincing evidence about the impacts of the intervention under study, because its 

results are based on the highest available degree of statistical rigor. 

In examining different types of methodological approaches to evaluating the ARRA grants, we 

focus in this chapter on random assignment; other possible approaches are discussed in Chapter V. 

We take as a premise that a random assignment evaluation would be intended to estimate the 

impacts of the grants on participants’ outcomes. That is, we assume in our discussion that 

community-wide outcomes, which were discussed in Chapter III, are not a primary focus of a 

random assignment evaluation design.20

In this chapter, Section A lays a foundation for the chapter by providing an overview of key 

components of a random assignment study.21 We then describe three possible ways in which a 

random assignment study of the grants could be designed (Section B). In Section C, we describe 

some considerations related to the details of the random assignment process. In Section D, we 

describe considerations about the timing of the grants and random assignment. Finally, in Section 

E, we present estimates of the minimum detectible impacts (MDIs) that might be feasible to achieve 

with two of the random assignment designs in which DOL has expressed the most interest. 

A. Overview of Design Features 

At a conceptual level, random assignment is an extremely simple method for evaluating the 

impacts of programs or services, such as those offered by the ARRA grants. At the same time, it is 

an extremely difficult approach to implement well. In this section, we provide a description of how 

random assignment works and some of the components that are most pertinent for consideration 

when planning an evaluation of the ARRA grants. 

1. Basic Approach and Strengths of the Method 

The goal of any random assignment evaluation is to yield precise, unbiased estimates of the 

effectiveness of the intervention under examination. Random assignment experiments involve 

randomly assigning individuals (or other units of analysis) into two or more research groups, with 

each group offered the option to receive a different set of services. When implemented carefully, 

 20 It is theoretically possible to use random assignment to assess community-wide outcomes. However, doing so 

would likely involve the random assignment of either (1) grant applicants into recipient or non-recipient status or (2) 

communities that grant recipients would want to serve into grant-funded communities or non-funded communities. 

Each of these approaches would require a major deviation from the current direction of the grants. The first of these 

approaches seems infeasible because the awarding of grants has already begun. The second of these approaches does not 

seem feasible for grantees that plan to serve only one community, and it would require a major restructuring of the way 

in which multi-community grantees will unfold their grant plans. 

21 Many of the conceptual points described for this type of study would also be pertinent for the other types of 

methodologies, which are discussed in Chapter V. However, given DOL’s focus on random assignment as a possible 

approach, we provide details about them in this chapter and avoid repetition in the later chapter. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

26 

random assignment creates groups that are almost identical in their characteristics prior to the 

random assignment process; the only systematic difference between them is in whether they are 

exposed to the intervention, such as ARRA grants-funded training, after being randomly assigned to 

a group. As a result, we can confidently conclude that the treatment caused any observed 

differences in the average outcomes between the two groups (Figure IV.1). Furthermore, the 

simplicity with which results can be interpreted provides additional appeal to this design. 

2. Important Components for a Random Assignment Evaluation 

Despite the simplicity of interpretation of the results from a well-designed and wellimplemented random assignment study, the underpinnings of the study can be complex. A random 

assignment study needs to have certain features, and other features are highly desirable even though 

technically not necessary.22 Here, we focus on four of the most important and common 

components of this type of study design. 

Figure IV.1. Impact Estimation from a Random Assignment Design 

There must be excess demand for intervention services. As implied earlier, a random 

assignment design of the ARRA grants would require that people who express an interest in 

 22 Chapter III contains a discussion of three important considerations when developing a design for an evaluation: 

(1) the importance of defining the research questions to meet policymakers’ areas of interest; (2) the need for a clear 

understanding of the counterfactual; and (3) the need to focus the outcomes of interest to a small number, and what 

those outcomes might be. Although these issues are important aspects of a random assignment design, we do not 

discuss them again in this chapter. 

0

200

400

600

800

1,000

1,200

1,400

1,600

1,800

2,000

Control Treatment

Outcome

(such as Average

Quarterly Earnings, in dollars)

Estimated

Impact

Note: This figure uses hypothetical data to demonstrate a random assignment design with an outcome such as 

average quarterly earnings on the Y-axis.
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

27 

receiving grant services can be randomly assigned either to a treatment group that is eligible to 

receive services or to a control group that is not. Thus, grantees will need to have excess demand 

for their services—that is, more eligible individuals requesting services than can be served—so that 

all program slots are filled and that the sample sizes of the treatment and control groups are 

adequate for the impact analysis. As explained in Chapter II, grant applicants were encouraged to 

demonstrate their suitability for an evaluation, including their ability to generate excess demand for 

their services. DOL indicated its intention to take this characteristic into consideration when 

determining which applicants would win grants; five points of a possible 100 grantee evaluation 

points were given for suitability for an evaluation. If ARRA grantees are asked to be part of a 

random assignment evaluation, therefore, they must either naturally have excess demand for their 

grant-funded services, be able to generate the extra demand through aggressive marketing of their 

services to their target groups, or modify their grant plans to include a more expansive set of target 

groups (which will lead to more applicants who are eligible for services). 

Regardless of the approach chosen to yield excess demand for services, the grantees that are 

part of a random assignment evaluation would be required to deny services to members of the 

control group.23 Based on our experience conducting random assignment evaluations for other 

studies for DOL and other clients, grantees are often reluctant to do this, even though this method 

for restricting service access can be perceived as a fair way to determine access to limited resources. 

Although we do not know for sure, it is possible that the current weak economy also might 

affect how ARRA grantees respond to a suggestion about using random assignment. On one hand, 

they may recognize already that they cannot possibly serve all individuals who might need their 

services; thus, they might be more amenable than is often the case to the prospect of using random 

assignment to determine who is eligible for services. On the other hand, they may find it especially 

hard and unpalatable to deny services to individuals who express interest in receiving services, 

because the weak economy might make it harder for the individuals to obtain jobs without 

assistance. 

All random assignment studies must have a control group, but one way to mitigate the potential 

concerns of ARRA grantee staff—if a random assignment design of the ARRA grants is used—is to 

limit the portion of study sample members who are assigned to the control group. Typically, 

random assignment of half of the sample members to the treatment group and half to the control 

group is the most efficient allocation and gives an evaluator the best chance of detecting impacts. 

However, a study design that assigns fewer than half of sample members to the control group might 

be more palatable to grantees, while only slightly reducing the ability to detect impacts. 

Random assignment designs should have samples of sufficient size to yield adequate 

statistical power and policy-relevant minimum detectible impacts. A rigorous evaluation of an 

intervention’s impacts requires a sample size that allows the detection of impacts that are large 

enough to be policy relevant. Minimum detectable impacts (MDIs) are the smallest true impacts on 

outcomes for which there is a high probability (80 percent) of detecting them; the smaller the MDI, 

 23 In some random assignment studies, control group members are given access to treatment group services, but 

only after the study has completed data collection. Thus, technically, they are delayed in their receipt of services and not 

fully denied access to these services. This distinction is unlikely to be useful for an evaluation of ARRA grants, because 

the grants are time-limited and because control group members are unlikely to want to wait until after the study to 

pursue training. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

28 

the greater the statistical power of the design. For example, an MDI of $200 in average quarterly 

earnings means that if the quarterly earnings for the treatment group are $200 more than those of 

the control group, then the study will be able to detect this difference with a high probability. In 

addition to certain technical assumptions used for calculating a study’s MDIs, key factors influencing 

a study’s power include (1) the study’s sample size; (2) whether the results are to be generalized to a 

population that is broader than the study sample members; (3) the standard deviation in the 

outcome (or outcomes) of interest; and (4) the extent to which the outcome(s) vary across sample 

members within clusters (typically, sites) or across clusters.24 Typically, assumptions about the last 

two of these factors are made on the basis of evidence from prior research. 

To help assess the MDIs that might be achieved with different types of random assignment 

designs for an ARRA grants evaluation, we present results of a power analysis in Section E. We 

focus on two types of outcomes, employment and quarterly earnings, because these are often the 

primary outcomes of interest in evaluations of DOL-funded training and reemployment services. It 

is likely that results would differ somewhat if other types of outcomes, such as employment in 

industries targeted specifically by the grants, are used. 

Baseline and services data are important for strengthening a random assignment study. 

In addition to the outcomes data that were discussed in Chapter III, random assignment studies 

often collect two types of data on study participants: (1) baseline data describing the characteristics 

of customers prior to random assignment and (2) service receipt data. The baseline data can be used 

to verify that random assignment created groups that have similar characteristics. Furthermore, 

baseline data can enhance the impact analysis, by describing the groups served by the program under 

study (such as the ARRA grants) and by defining subgroups of interest in the impact analyses. Such 

data also can be used to improve the precision of impact estimates. Although baseline data are often 

collected through the completion of forms developed specifically for the evaluation, some of the 

relevant data may come from forms completed as part of individuals’ applications for services or 

through administrative records. 

Data on the receipt of services by both treatment and control group members also is important 

to collect. These data can be used to ensure that control group members do not receive intervention 

services (in this case, ARRA grant services) and to assess the extent to which treatment group 

members receive these services. In addition, analysis of these data can help to interpret impacts on 

outcomes. In the case of the ARRA grants, the grantee-collected data will cover service receipt only 

for treatment group members. One source of data on services received by both the treatment and 

control group members might be WIA files maintained by states, coupled with financial data 

maintained by LWIAs to provide information on the value of individual training account (ITA) 

amounts. However, these data files will not cover receipt of most types of services that are not 

funded by WIA. Another source that could provide data that are more comprehensive, as well as 

 24 When sample members are clustered within a grantee site, they are likely to face similar economic conditions and 

have similar demographic characteristics to each other; therefore, their outcomes are likely to be more similar to each 

other than to sample members in other sites. This correlation in outcomes within a site would increase the variances of 

the impact estimates relative to those from a simple random sample of the same size because the variance formulas must 

include a term representing the between-site variance in the outcome measures. Thus, when sample members are 

clustered, a larger sample size is needed to achieve a certain MDI compared to what would be the case with a simple 

random sample. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

29 

standardized across study sites, could be a survey of the treatment and control groups sample 

members. 25 

Another type of data that is often, but not always, collected as part of a random assignment 

study is data on the benefits and costs of the program. As a useful supplement to an impact 

evaluation, these data could provide answers about the net benefits (or costs) of the services that 

participants receive, measured in monetary terms. 

The analysis approach can provide answers to a range of questions of interest. As noted 

earlier, an advantage of a random assignment study design is its simplicity. When executed properly, 

simple comparisons of the differences in outcomes between the treatment and control groups can 

give unbiased estimates of the impacts of the intervention. However, using the data described 

earlier, and possibly qualitative data collected through an implementation component to the study, 

enhancements to this basic approach could improve the quality of information that is gained from 

the study and provide additional insights about issues of interest to policymakers. As alluded to 

previously, additional analyses in an evaluation of ARRA grants could be used to describe who 

participated in grant-funded services and what types of services they received. These analyses could 

provide information about the quality of implementation and whether some types of participants 

benefitted from grant services more than others. And, depending on how nearly complete followup data are, it might be appropriate to assess whether any biases arise from different rates of 

response to follow-up data collection. 

B. Possible Approaches to Using Random Assignment for ARRA Grants 

In this section, we discuss three types of approaches to conducting a random assignment 

evaluation of ARRA grantees. For all approaches, we assume that study sample members at a 

participating grantee site will be assigned to either a treatment group or a control group.26 The 

approaches are (1) randomly assigning all individuals at all grantees; (2) randomly selecting a subset 

of grantees and then randomly assigning all individuals; and (3) focusing on a nonrandom subset of 

grantees and then randomly assigning all individuals. 

1. Randomly Assign All Individuals at All Grantees 

One possible approach for conducting a random assignment study of the ARRA grants would 

be to include all grant participants in the evaluation. This approach has considerable strengths and 

weaknesses to it. Its main strength is its ability to provide answers to questions about the impacts of 

the grants nationwide, with the most statistical power of any of the approaches. Furthermore, if 

control group assignment rates are high enough, it might be possible to provide answers to 

questions about impacts for specific sets of grantees with a high degree of statistical precision. 

However, the scope of the grants in terms of both the number of participants to be served and 

their geographic locations means that a study that includes all grantee participants would be 

 25 If a follow-up survey is to be conducted, identifying and contact information collected at the point of random 

assignment can help in the process of locating participants for the survey. 

26 A refinement of these approaches would involve the subsampling of grantee participants at a study site to 

determine inclusion in the evaluation. We do not explore this issue in this paper, given its complexity and the breadth of 

approaches that are discussed. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

30 

extremely expensive. Although the precise target numbers of participants that grantees expect to 

serve is not currently available, our calculations based on available information suggest that the 

numbers are in the tens of thousands for each grant type. The 25 ETP grantees are likely to strive to 

serve at least 34,000 participants; the 38 Pathways and 34 SESP grantees are likely to strive to serve 

at least 20,000 participants and 54,000 participants, respectively. Furthermore, the geographic 

coverage of the grants is quite large, a factor that would add to the cost of this type of 

comprehensive approach to random assignment. The ETP, Pathways, and SESP grants are each

expected to operate in at least 29 states or other jurisdictions, such as the District of Columbia or 

Puerto Rico. Thus, conducting random assignment at all grantees (for even one of the grant types) 

would likely require setting up random assignment in hundreds of sites, training staff and monitoring 

procedures at these sites, processing tens of thousands of forms to collect baseline data, and 

collecting administrative data from numerous agencies if this type of data are used for measuring 

participants’ outcomes. 

Because of the great cost of including all grantees from one or more grant types in an 

evaluation, we do not explore this approach in greater detail in the rest of this chapter. 

2. Randomly Selecting a Subset of Grantees and then Randomly Assigning Individuals at 

Them 

One way to mitigate the cost disadvantages of the first approach to random assignment, while 

preserving some of its’ advantages, is to randomly select a subset of grantees for inclusion in the 

study and then randomly assigning sample members at those study sites only. Through the random 

selection of sites, this approach—like the first approach—will be able to answer questions about the 

impacts of the grants nationwide. Importantly, focusing on a subsample of grantees could reduce 

the heavy cost burden associated with random assignment of sample members at all grantees. In 

addition, it might still be possible to obtain impact estimates for subgroups by stratifying the 

grantees along dimensions of interest prior to random selection. 

However, a drawback of this approach is that it introduces a clustering effect, which reduces the 

precision of the impact estimates compared to a situation in which all grantees or a purposive 

selection of grantees are included in the study. Furthermore, given the variance across grantees in 

the per-grantee number of participants to be served, this approach runs the risk of a “bad draw” 

leading to the random selection of a set of grantees with very small target numbers of participants. 

Stratifying the grantees by size before random sampling and/or sampling sites proportional to size 

would help to avoid this problem and improve statistical power without introducing large design 

effects. 

3. Focusing on a Nonrandom Subset of Grantees and then Randomly Assigning 

Individuals at Them 

An alternative approach to random selection of grantees is to select intentionally a subgroup of 

grantees on which to focus the evaluation. After doing so, study sample members would be located 

at these sites only and—as with the other approaches—would be randomly assigned to either the 

treatment or the control group. 

Purposeful selection of grantees could be based on any of a number of policy-related or 

logistical reasons. For example, policymakers might want to focus an evaluation on a subset of ETP 

or SESP grantees that are serving communities affected by auto restructuring. Or, they might be 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

31 

particularly interested in looking at the Health Care and Other High-Growth and Emerging 

Industries grants that are focusing on the health care industry only. In addition, policymakers might 

be interested in the subset of grantees that plan to serve a particular type of target group, such as 

dislocated workers or high school dropouts, or that plan to use a particular service delivery method, 

such as online learning or apprenticeship programs. Logistical considerations, such as the timing of 

when grantees are expected to serve participants or a grantee’s conduciveness to participating in the 

evaluation, might also be important considerations for selecting grantees. 

Compared with the approach of randomly sampling grantees from among all grantees, this 

approach provides more statistical power because uncertainty due to the site sampling is not 

introduced into the study design. Furthermore, to the extent that policymakers have special interest 

in some grantees more than others, this approach would enable the evaluation to focus on the 

grantees of interest, although the ability to generalize to other grantees would be lost. (In technical 

terms, this approach would “lose external validity.”) Conclusions from the study would pertain to 

those grantees only, and not all grantees. Finally, it also might involve a smaller investment in both 

time and resources to secure the cooperation of sites that have been purposefully selected, compared 

with an approach of random selection of grantees, even if grantees are contractually required to 

participate in this type of study. 

C. Details of the Random Assignment Process 

A thorough consideration of the details of conducting the random assignment process at study 

sites is necessary for an experimental evaluation of grant services to be successful. These details 

include the point at which random assignment should be inserted in the flow of participant intake, 

the methods for the collection of baseline data and the assignment of individuals to treatment and 

control groups, monitoring of random assignment procedures, and restriction of control group 

members’ access to services. 

Although a comprehensive investigation of these issues requires more information about the 

grantees’ plans than is currently available, we discuss two considerations of primary importance for 

the design of a random assignment evaluation of the ARRA grants: (1) the point at which to insert 

random assignment and (2) restriction of control group members’ access to services. 

1. The Point at Which to Insert Random Assignment 

A very stylized and simplistic model of an intake and random assignment process begins with 

an individual’s expression of interest in receiving intervention services. After that, their eligibility 

for services is determined. Those who are not eligible for services are excluded from the study. 

Those who are eligible are asked to provide baseline data, which is used to further confirm the 

individual’s eligibility. Only after that step does the individual go through random assignment. 

However, our experience in conducting random assignment evaluations for DOL and other clients 

suggests that many details of this process need to be considered before a random assignment study 

can be successful. 

In this section, we focus on a few important issues related to the intake and random assignment 

process for grant services. We begin by describing potential implications for random assignment of 

the number of locations that could serve as intake points. We next focus on implications of the 

types of locations. Finally, we discuss issues regarding how the point of random assignment might 

influence the ability to estimate the impacts of the grants on participants’ outcomes. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

32 

An important requirement of the random assignment process is that random assignment occurs 

at all places where grant participants can go through the intake process for grant services. This is so 

that there is no circumvention of the process.27 Grantees of the four ARRA grant types are likely to 

have a different number of locations through which intake can occur. Two of the SESP grantees, 

for example, plan to provide services in more than 40 counties in their states. In contrast, some of 

the Pathways Out of Poverty grants plan to serve one or a few communities within a city. A wide 

geographic dispersion of grant services will likely complicate and increase the expense of the training 

of staff on random assignment and the monitoring of the procedures. 

In addition to the number of locations, the types of locations through which intake could occur 

is important, since random assignment procedures will need to be implemented at each type of 

location. The range in types of locations at which intake could occur is likely to vary across the four 

grant types since the four SGAs had different requirements about the types of partners that had to 

be included in grant efforts. Although many grantees will likely recruit at least some of their 

participants through the One-Stop Career Center system, they might differ in the other sources from 

which participants are recruited. For example, compared to the Pathways Out of Poverty grantees, 

some ETP, SESP, and Health Care grantees might be more likely to obtain participants through 

their partnerships with employer and union organizations, since these grants have more of an 

emphasis on serving incumbent workers. These cases might warrant special considerations or 

procedures because individuals assigned to the treatment and control groups might work in close 

proximity to each other or come in contact with each other in other ways. In contrast, the Pathways 

Out of Poverty grant participants might not be as closely affiliated with each other through official 

organizational ties, which might influence the way in which the procedures for random assignment 

can be designed. 

Finally, the four different grant types might differ from each other in terms of expected impacts 

of the intervention, given a particular point of random assignment. If an evaluation’s primary 

research question pertains to the impacts of access to any grant-funded services, a natural point for 

random assignment would be as soon as an individual is found eligible for grant-funded services and 

before receipt of those services. Because of the grants’ focus on the provision of training, many 

grantees are likely to place individuals in grant-funded training immediately upon establishing 

eligibility. However, in the case of the Pathways grants especially, the first type of grant-funded 

service might be something besides training. Because Pathways grantees are more likely than other 

ARRA grantees to serve participants with low levels of basic skills and less attachment to 

employment, their participants might be less likely to receive any services, including training services, 

even when they are assigned to the treatment group during a random assignment study. While this 

expected pattern might or might not influence a decision about where in the flow of intake random 

assignment should be inserted, it would be important for an evaluation to take into account the 

influence of this potential dilution of intervention services on impact estimates. For a given 

evaluation sample size, a higher portion of treatment group members who do not receive grant 

services leads to a lower ability to detect meaningful impacts. 

 27 As discussed in both Sections B and D, DOL could choose to conduct a random assignment study that does not 

include all grant participants. However, the point here is that any exclusion of some grant participants must be an 

intentional, carefully-assessed part of the design, rather than an omission based on poor execution of the design. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

33 

2. Restriction of Control Group Members’ Access to Services 

Another consideration related to the details of the random assignment process pertains to the 

types of services that might be denied to control group members. Again, the issues here are 

complex and warrant further investigation during the design phase of a random assignment 

evaluation. But, we raise two issues for further investigation. 

First, the evaluation would need to determine whether control group members, who would be 

denied access to training provided directly through ARRA funds, would be allowed to use nonARRA funds to participate in training that only exists as a result of ARRA grant-funded activities. 

Under some circumstances, DOL is allowing grantees to use grant funding for (1) the development 

or modification of training curricula, (2) the education or training of instructors who will provide 

training, and/or (3) the development or modification of standards and procedures for defining and 

issuing credentials. Thus, these ARRA-funded activities could have a broad influence on the types 

of training that are available in a grant community. 

Second, the evaluation of the ARRA grants will need to determine whether control group 

members are prevented from accessing any and all services funded through other ARRA grants 

deployed in the same local area. DOL imposed few restrictions upon whether organizations could 

apply for or participate in more than one of the four ARRA grants discussed in this pre-design 

study. One restriction was that a state or local ETP grant recipient could not receive a subgrant or 

subcontract from an ETP national labor-management grant recipient. Another was that a Pathways 

grant recipient could not receive an ETP grant, although it may participate as a partner in an ETP 

grant. Given the limited scope of these restrictions, and in some cases the breadth of geographic 

coverage of the grants, it is likely that more than one ARRA grantee (from either the same or 

different grant types) will provide services to a particular community included in an evaluation. As 

noted above, some of the SESP grants include plans to cover a wide geographic area in a state. But, 

even for more narrowly focused grants, such as two local Pathways grants serving St. Louis, 

Missouri, there is overlap in the PUMAs in which the grant recipients are targeting. 

D. Scheduling Considerations that Influence All of the Random Assignment 

Designs 

As discussed in Chapter II, three of the four types of ARRA grants were awarded in January 

2010, and grantees are currently developing their plans to begin enrolling and serving participants. 

Although DOL is eager to begin an evaluation of the ARRA grants and is striving to do so as 

quickly as possible, an important factor that influences the feasibility or desirability of different types 

of evaluation methodologies pertains to the schedule of both the grants and any evaluation that 

DOL undertakes. Issues related to the timing of grant and evaluation activities are particularly 

important for a random assignment design, because—compared with other types of evaluation 

methodologies—development and implementation of this type of design is especially timeconsuming. 

In this section, we discuss issues related to the timing of the grants and a random assignment 

evaluation. After specifying the assumptions that we have made for our analysis, we discuss two 

implications. The first implication is that, under certain assumptions, plausible time schedules for 

the grants and evaluations are likely to reduce considerably the number of clients who could be part 

of a random assignment evaluation of the three-year grants and might even make random 

assignment infeasible for the two-year grants. The second is that, under the same assumptions, it is 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

34 

likely that an evaluation time schedule can include a sufficiently long follow-up period to measure 

outcomes of interest to DOL. 

1. Assumptions About the Time Lines of the Grant and Evaluation Activities 

Figure IV.2 presents two hypothetical time lines: one for the three-year grants and one for the 

evaluation. For the purposes of our analysis of the time lines for the grants and an evaluation, we 

have made four assumptions about the grant activities. First, we have assumed the grants are threeyear grants (as is the case for the SESP and Health Care grants) and start on February 1, 2010. We 

have chosen this start date for illustrative purposes, but a start date that is earlier by a few weeks 

would not greatly influence the points discussed here.28 

Figure IV.2. Hypothetical Time Lines for 3-Year Grant and 5-Year Random Assignment 

Evaluation 

 28 DOL announced the award of the ETP grants on January 6, 2010; the Pathways Out of Poverty grants on 

January 13, 2010, and the SESP grants on January 20, 2010. 

Grant 

Start

4-Month 

Planning 

PeriodFirst Participant 

Starts Training

26-Month 

Enrollment Period

Last 

Participant 

Starts 

Training

6-Month 

Training

for Last 

Participant

Contract 

Start 15-Month 

Design Period

First Sample 

Member 

Randomly 

Assigned

18-Month 

Random 

Assignment 

Period

Last Sample 

Member 

Randomly 

Assigned

18-Month 

Follow-Up 

Interview Period

Last Follow-Up 

Interview 

Completed

9‐Month Analysis

and Reporting

Period

Last Participant 

Ends Training 

and Grant Ends

Contract

Ends

6/1/2010

7/31/2012 1/31/2013 6/30/2015

10/1/2011 3/31/2013 9/30/2014 2/1/2010

7/1/2010

Time Line for 3-Year Grant

Time Line for 5-Year Random Assignment Evaluation

2/1/2010

6/30/2015
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

35 

Second, we have assumed that grantees take four months for planning prior to enrollment of 

the first participant. This time will be used for start-up activities, such as hiring of staff, 

coordination among grant partners, and—in some cases—the development or modification of 

training curricula or certification standards. In Figure IV.2, the left side of the time line for the 

grants shows that, given a start date of February 1, 2010, and a four-month planning period, the 

grantees will be ready to enroll their first clients around June 1, 2010. 

Third, we have assumed that the typical length of training is six months. The Individual 

Training Account (ITA) experiment found that, at the 15-month follow-up interview, the average 

number of weeks that participants spent in training within five quarters after random assignment 

was between 25 and 29, although 20 to 25 percent of the ITA sample members were still in training 

at the time of the follow-up interview. The lack of full follow-ups for the ITA sample members 

suggests that the average length of training funded by the ARRA grants might be longer, but a 

mitigating factor is the duration of the grants. A better estimate of the length of training is likely to 

depend on the details of the grantees’ plans. 

Finally, we have assumed that grantees will complete their participant enrollment in time for the 

last enrolled participants to complete their training at the end of the grant period. Implicitly, this 

means that the last enrollments will occur six months prior to the end of the grant period, because 

we have assumed that the average length of training is six months. This also means that, for the last 

enrolled participants, any post-training services that the grantee or its partners provide, as well as any 

follow-up data collection, will need to be funded through sources other than the ARRA grants.29 As 

a result of this assumption, the right side of the grants time line in Figure IV.2 shows that, given a 

contract end date of January 31, 2013, the grantees would need the last clients to start training 

around July 31, 2012. 

For the evaluation schedule, we have made three key assumptions. The first is that the 

evaluation starts on July 1, 2010, and is a five-year contract. 

The second assumption is that it takes 15 months from when the contract starts until the first 

sample member goes through the random assignment process. Between the time an evaluation 

contract is awarded and when random assignment of participants can begin, several steps must be 

completed, including the development of the evaluation design, baseline and consent forms, and the 

database that will be used to conduct random assignment, as well as the Office of Management and 

Budget (OMB) approval process. This process will include the submission of a draft OMB package 

for DOL’s review and revisions to the draft based on DOL’s feedback; the publication of and 

waiting period required with a Federal Register notice; and the period during which OMB reviews the 

package and the package is revised before approval is given. Typically, the time between the initial 

submission of the draft package to DOL for review and the receipt of clearance is six to nine 

months. With a start date of July 1, 2010, for an evaluation contract and a 15-month design period 

prior to the start of random assignment, the first sample member could go through random 

assignment on October 1, 2011 (Figure IV.2). 

 29 Based on email communication with DOL on January 26, 2010, DOL expects that some participants might still 

be enrolled in training when a grant ends. In addition to participant follow-up, which grantees are expected to do 

throughout the grant period of performance, DOL will be able to calculate the “common measures” outcomes for 

participants through the use of unemployment insurance (UI) wage records data. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

36 

Based on our prior experience, allowing 15 months for the design phase of a random 

assignment evaluation would be a tight schedule, but it could be feasible. For example, 

Mathematica’s ITA and Job Corps experiments included an 18- to 19-month time frame and a 16- to 

17-month time frame, respectively, before random assignment began. However, it is possible that 

an evaluation of the ARRA grants might not need as long a pre-random-assignment design period as 

these two studies did. In the case of the ITA study, a considerable amount of time was spent 

designing the three categories of treatment services that would be compared to each other; it is 

unlikely that the need to develop and define the intervention exists to this extent with ARRA grants. 

For Job Corps, considerable time was needed because the applicants of many locations 

nationwide were to go through random assignment. If DOL were to choose either the second or 

the third approaches to random assignment (described earlier) for an evaluation of the ARRA 

grants, then the evaluation would take place in a much smaller number of locations than was the 

case for the Job Corps study. Furthermore, to the extent that ARRA grantees agreed as a condition 

of grant receipt that they might be required to participate in a random assignment study, the 

evaluation of these grants might not need as much time as is often required to convince grantees or 

sites to participate. 

Our third assumption is that the evaluation needs nine months before the contract end date for 

the analysis of data, preparation of the draft final report, and revision of the report in response to 

DOL’s comments. 

Of course, changes in these assumptions about the grant or evaluation time lines will change the 

conclusions of this analysis. For example, grantees might begin enrolling clients at a slower pace 

than we have assumed if they spend a considerable amount of time developing training curricula 

before enrollments begin. DOL might be able to have the evaluation started sooner than July 1, 

2010, and it might desire a longer or shorter evaluation period. Furthermore, should DOL choose 

to use a random assignment design that purposively selects grantees (that is, the third of the random 

assignment approaches discussed in Section B), part of the selection process could include 

consideration of grantees’ time lines for serving participants. 

Although our analysis of grant and evaluation time lines is inherently dependent on the 

assumptions described here, and therefore inherently subject to imprecision, the analysis can serve as 

a starting point for thinking about two key issues: (1) the portion of the grant participants that could 

be included in a random assignment study and (2) the length of the follow-up period in the 

evaluation. We discuss these issues in the next two subsections. 

2. Implications about the Portion of the Grantees’ Clients Who Could Participate in 

Random Assignment 

Based on the assumptions about the grants time line, it would be possible for the grantees to 

have a 26-month enrollment period, from June 1, 2010, to July 31, 2012. The hypothetical 

evaluation time line (shown as the bottom time line in Figure IV.2) indicates that this 26-month 

enrollment period overlaps only partly with a plausible random assignment period. The October 1, 

2011, start date for random assignment is 16 months into the grantees’ 26-month enrollment period, 

leaving only a 10-month period of overlap between the grantees’ enrollment and the evaluation’s 

random assignment time window. The overlap is for 38 percent (= 10/26) of the grantees’ 

enrollment period. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

37 

If grantees enroll clients at a steady pace, then only about 38 percent of their clients would be 

subject to random assignment. Based on our observations of grant enrollments for other studies, it 

is often the case that grantees have a slow start-up period. If that is the case for the ARRA grants, 

then it is possible that a higher percentage of clients would enroll after the start of random 

assignment. However, because grantees are likely to be eager to distribute grant funds, especially 

given the current high unemployment rates, they might have a strong incentive to enroll participants 

as quickly as possible. Grantees that are successful in responding to this incentive might enroll 

much more than 62 percent of their clients prior to the start of random assignment, with 

considerably fewer than 38 percent available for random assignment. 

Although the Health Care and SESP grants are for three years, the ETP and Pathways grants 

are for two years. For these grants, the hypothetical time line would look very similar to that shown 

in Figure IV.2, except that the grant end date and the last client’s enrollment date would need to be 

one year earlier. This suggests that the grants would need to enroll clients in about 14 months—not 

26 months—and enrollment would end around July 31, 2011, which is prior to when random 

assignment could be expected to start. In order for there to be overlap between the grantees’ 

enrollment period and the evaluation’s random assignment period, one or more of the assumptions 

about the time lines would need to be adjusted. For example, a four-month overlap could be 

possible if (1) the evaluation start-up period is 13 months instead of 15 months and (2) the average 

length of training is two months instead of six months. In this case, random assignment would start 

August 1, 2011, and enrollment would end November 30, 2011; the four-month overlap would 

allow 24 percent (= 4/17) of the participants to go through random assignment. 

For both the two- and three-year grants, the percentage of grant participants who can be 

included in the random assignment process will have a strong influence on the MDIs that can be 

detected through a random assignment evaluation. We discuss this issue further in Section E. 

3. Implications about the Follow-Up Period for the Observation of Post-Training 

Outcomes 

The hypothetical grant and evaluation time lines also indicate that a random assignment 

evaluation of three-year grants would have an ample follow-up period. Given an evaluation contract 

end date of June 30, 2015, and the need for nine months for the analysis and reporting periods, the 

last follow-up data collection would need to occur by September 30, 2014 (Figure IV.2). For 

administrative records data, such as UI wage records, this would be the date by which the data 

would need to be provided to the evaluator. (UI wage records data extracts provided by this date 

would probably contain information through June 30, 2014, given the time lag necessary for states to 

provide an evaluator with the data after processing employers’ reports of their employees’ earnings.) 

With the last clients of a three-year grantee having started training around July 31, 2012, the 

schedules would allow a follow-up period from July 31, 2012, to September 30, 2014 (26 months for 

interviews and probably about 23 months for administrative data). 

If DOL wanted to include as part of an evaluation a follow-up survey to measure participants’ 

post-training outcomes, September 30, 2014, would be the latest feasible date for the last follow-up 

survey to be completed. However, it appears there would be adequate time in the evaluation 

schedule to include this type of follow-up survey. Determining the best length of time after random 

assignment for a follow-up survey to be conducted involves balancing two often-competing factors. 

One is the desire to ensure that the follow-up period is sufficiently long so that participants could 

complete their activities (such as grant-funded training). The other is ensuring that the survey is not 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

38 

so long after the activities and outcomes of interest occur that survey respondents have difficulty 

recalling information accurately.30 Some evaluations mitigate the tension between these factors by 

including two follow-up surveys, one of which collects information on short-term outcomes and the 

other of which collects information about longer-term outcomes. However, this approach adds to 

the cost of the evaluation compared with use of a single follow-up survey. In the case of an 

evaluation of the ARRA grants, it is likely that DOL would have flexibility for determining how to 

balance these issues to best meet its needs for either short-term or long-term outcomes information. 

For illustrative purposes in Figure IV.2, we have shown an 18-month random assignment period 

and an equally long follow-up interview period. However, these lengths could be adjusted based on 

DOL’s preferences for how long the follow-up period should be. 

E. Example Minimum Detectible Impacts for Random Assignment Studies 

of the ARRA Grants 

Should DOL choose to use a random assignment design for an evaluation of the ARRA grants, 

it will be important to ensure that the design selected will yield adequate levels of precision for the 

estimation of program impacts on participants’ outcomes. As explained in Section B, an MDI is the 

smallest true impact for which there is a high probability of detection (typically, assumed as an 80 

percent probability); the smaller the MDI, the greater the statistical power of the design. For 

example, an MDI of 7.5 percentage points in the probability of employment means that if the 

employment rate for the treatment group is at least 7.5 percentage points higher than that of the 

control group, then the study will be able to detect this difference with a high probability. 

To provide guidance to DOL about the statistical precision that might be expected under 

different approaches to random assignment, and to assess the tradeoffs involved, we present a set of 

illustrative MDI calculations under different scenarios. Although this analysis provides several 

insights for planning an evaluation of the ARRA grants, we caution that making a sound decision 

about proceeding with a particular evaluation design would require more precise analysis, based on 

additional information and further investigation. 

For all scenarios, we have calculated MDIs separately for the three types of grants (ETP, 

Pathways, and SESP) awarded as of the end of January 2010 using publicly available information 

about the target number of participants that grantees plan to serve.31 We have not provided MDI 

calculations for the Health Care grants because information about the grant winners was not 

available for inclusion in this analysis. We begin by explaining the types of variation that we take 

into account in the MDI calculations. Then, we discuss patterns in the MDIs that have implications 

for a random assignment evaluation design. 

 30 For example, Mathematica’s original ITA study used a 15-month follow-up period. Although this length was 

adequate for observing the post-training outcomes of many sample members, DOL subsequently contracted with 

Mathematica to conduct a longer-term follow-up study, in part because some sample members were still in training at 

the time of the 15-month follow-up interview. 

31 As explained in Chapter II, the publicly available information on grant winners’ plans is limited. In some cases, 

we cannot tell the target number of participants to be served by grantees, because this information is unavailable or 

unclear. In these cases, we have made assumptions that facilitate our analysis. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

39 

1. Types of Variation Taken into Account in the MDI Calculations 

We have calculated the MDIs for a range of different circumstances to illustrate how they might 

vary based on different assumptions and strategies that DOL might want to use in an evaluation of 

the ARRA grants. In this section, we discuss the dimensions that we have allowed to vary. 

a. The Random Assignment Approach that Is Used 

In Section B, we presented three possible approaches for conducting a random assignment 

evaluation. The first (Approach # 1) includes all grantees in the evaluation. The second (Approach 

# 2) involves a random sampling of a subset of grantees to yield impact estimates that are 

representative of all grantees. The third (Approach # 3) involves purposeful selection of a subset of 

grantees. In all three approaches, individuals at study grantee sites are randomly assigned to either a 

treatment or control group. We have restricted our MDI analysis to Approaches # 2 and # 3 

because cost considerations are likely to make Approach # 1 very unappealing to DOL. 

b. The Percentage of the Evaluation Sample Members that Are Assigned to the Control 

Group 

As discussed in Section A, randomly assigning fewer than half of all study sample members to 

the control group can make participation in a random assignment study more appealing to grantees, 

because they have to deny services to fewer individuals deemed eligible for grant services. However, 

a 50–50 treatment-control group assignment rate provides the greatest degree of statistical precision 

for a given sample size. We have calculated MDIs based on two scenarios: The first is that half of 

sample members (50 percent) are assigned to the control group; the second is that one-quarter (25 

percent) are assigned to the control group. 

c. The Percentage of Grantees’ Participants that Can Be Included in Random Assignment 

As explained in Section D, timing issues related to when the grantees are likely to begin serving 

participants compared with when an evaluation is likely to be able to start random assignment might 

dramatically reduce the portion of participants that could be included in this type of evaluation. 

Based on the time line analysis in Section D, we have calculated MDIs under different assumptions 

about the percentage of grant participants that can be included in random assignment. For the 

three-year SESP grants, we have assumed that either 30 percent or 40 percent of participants are 

included. For the two-year ETP and Pathways grants, we have assumed participation rates of either 

10 or 20 percent. 

d. The Source of the Outcome Data 

Two sources of outcome data that are frequently used in evaluations of training and 

employment services initiatives are follow-up surveys and UI administrative wage records (according 

to the SGAs, DOL has already expressed interest in using the latter for the ARRA grants). The 

source of the data is important because of differential rates of availability of the data for the impact 

analysis. For this study, we have assumed there will be an 80 percent response rate to a survey of 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

40 

5,000 sample members and a 100 percent “response rate” for the administrative records data (that is, 

all sample members will have administrative records data).32 

e. The Number and Types of Grantees that Are Included 

DOL has expressed interest in learning about how sensitive the MDIs are to the number of 

grantees that might be included in a random assignment evaluation. The ETP grantees can be 

distinguished on the basis of whether they include auto restructuring communities and whether they 

are national or local grantees. The Pathways grantees also can be distinguished on the basis of 

whether they are national or local grantees. The SESP grantees can be distinguished on the basis of 

whether they include auto restructuring communities. Based on the number of types of grantees, we 

have devised several different subsets of grantees as possible candidates for consideration for an 

evaluation. However, we recognize that the subsets that we have examined are not exhaustive. In 

order to derive hypothetical sample sizes for the calculations, we generated an estimate of the 

average grantee size by grant type and subtype of interest (such as grantees serving auto 

restructuring communities for ETP grants) using publicly available reports of the number of 

participants each grantee intended to serve. 

f. Factors Not Taken into Account 

For our MDI calculations, we have not taken into account two factors that may prove 

important for further consideration. First, we have not adjusted our MDIs for the higher likelihood 

of detecting statistically significant, but spurious, impacts when more than one outcome of interest is 

examined. Adjusting for this increased likelihood would increase the MDIs. 

Second, we also have not taken into account the possibility of varying the number of sample 

members that are included from each study grantee. For example, instead of assigning all eligible 

individuals who express interest in receiving grant services to either a treatment or control group, 

one could randomly assign them to one of three categories: a treatment group, a control group, and 

a group that is not part of the study. Data for individuals assigned to the third group would not be 

collected or analyzed. The desirability of this approach would depend on cost considerations, the 

total number of potential sample members given participants’ enrollment patterns over time (as 

discussed in Section D), and the effect of the third group on statistical power. However, we do not 

explore this option because Section D indicates that, even with two groups only, the potential 

sample sizes for an evaluation might be considerably smaller than the total number of participants 

that grantees will serve. 

Another way to vary the number of sample members who are included in the study would be to 

subsample sample members at study grantees based on the communities in which the sample 

members are located: for grantees that plan to serve more than one community, one could 

randomly sample which community or communities are to be included in the evaluation. This 

approach has the potential to reduce evaluation costs when study grantees plan to serve a wide 

 32 More precisely, we assumed a survey sample size of the lesser of 2,500 sample members in each of the treatment 

and control groups and the number available based on our other assumptions. The number of sample members in each 

group was not necessarily the same for a given set of assumptions. In addition, for administrative data analysis, we have 

used the typical assumption that sample members for whom administrative records data are not found have “records” of 

zero earnings. Thus, we have assumed that all sample members have records even if a match is not found between his 

or her Social Security number and the administrative records data files. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

41 

geographic area. However, examining this issue is not feasible at this point, given the number of 

options that are already explored in this analysis and the information that is currently available about 

the grantees. A design that uses subsampling within grantees also would increase the MDIs. 

2. Example MDIs Based on Different Random Assignment Scenarios and Assumptions 

We begin our presentation of illustrative MDIs using the ETP grants. Table IV.1 shows a range 

of MDIs based on variations in the dimensions discussed earlier, such as the percentage of sample 

members that are assigned to the control group and the percentage of grant participants that are 

included in the study sample. For example, as shown in the first row of numbers in Table IV.1, an 

evaluation would be able to detect a $181 impact on average quarterly earnings, under the following 

assumptions: (1) the 16 auto restructuring grantees are included in the study, (2) they are 

purposefully selected (Approach # 3), (3) 10 percent of grant participants are part of the random 

assignment process, (4) survey data are used as the source of the outcome, and (5) 50 percent of 

sample members are assigned to the treatment group and 50 percent to the control group. 

Because of the many different assumptions and scenarios for which we calculate MDIs, it is 

hard to draw an overarching conclusion about whether the MDIs are “big” or “small” compared to 

what might be desirable for an evaluation of the ARRA grants. However, having in mind some 

MDIs from other studies that Mathematica has conducted for DOL might be useful for assessing 

whether these example MDIs for the ARRA grants are, relatively speaking, big or small.33 For 

example, for the current impact evaluation of the WIA program that Mathematica is conducting, we 

calculated MDIs using samples of adult and dislocated workers (Bellotti et al. 2009). For the full 

sample, the MDIs were reported as $157 using survey data and $123 using administrative data. For 

a subgroup of 50 percent of sites, the MDIs were reported as $195 and $153 based on survey and 

administrative data, respectively. For a study of the Trade Adjustment Assistance Program, for 

which Mathematica currently is serving as a subcontractor, we estimated MDIs of $242 and $122 in 

quarterly earnings for the full sample using survey data and administrative data, respectively 

(Schochet 2005). Finally, for a study of a program designed to promote entrepreneurship, for which 

Mathematica served as a subcontractor, the evaluation design report showed MDI estimates of 

$1,326 for annual earnings from any job (= $332 per quarter) for a sample of 2,000 treatment and 

2,000 control group members (McConnell et al. 2004). In practice, a slightly larger number of 

participants (4,198) went through the random assignment process than was reflected in the MDI 

calculations, which means that the actual MDIs might have been slightly smaller than those reported 

in the design report (Benus et al. 2008). Despite the perspective that the MDIs from other studies 

gives for assessing whether the example ARRA grant MDIs are big or small, a final assessment of 

whether or not an MDI for a particular design is adequate would depend on the hypothetical impact 

to be expected. 

 33 The interventions planned for the ARRA grants differ from the interventions that have been or are being 

assessed through the other studies cited here. Therefore, one might have different expectations about the impacts that 

are induced by the different programs. For example, a program that is very costly and intensive might be more likely to 

generate large impacts than one that is less costly and less intensive. Thus, DOL might deem acceptable an MDI for one 

study but not for another study. 
Pre-Design Study of the ARRA Grants 

42 Table IV.1. Minimum Detectible Impacts on the Percentage Employed and Average Quarterly Earnings, Energy 

Grants Random Assignment Approach # 2 (Randomly Subsampling Grantees) [Survey, Administrative Data] Random Assignm

(Purposefully Sel

[Survey, Admin

Percentage Employed Average Quarterly Earnings ($) Percentage Employed 

50–50 Treatment-Control Group Assignment Rate 10% of Grant Participants Included in Random Assignment The 16 auto restructuring granteesa n.a. n.a. .069, .062 

10 grantees (5 national and 5 local) .091, .085 238, 221 .077, .069 All 7 national grantees n.a. n.a. .090, .08 5 national grantees .125, .114 325, 298 .114, .102 All 18 local grantees n.a. n.a. .059, .053 10 local grantees .094, .086 244, 224 .084, .075 5 local grantees .158, .147 412, 383 .133, .119 

20% of Grant Participants Included in Random Assignment The 16 auto restructuring granteesa n.a. n.a. .049, .044 

10 grantees (5 national and 5 local) .074, .070 192, 182 .054, .049 All 7 national grantees n.a. n.a. .063, .057 5 national grantees .097, .090 252, 234 .080, .072 All 18 local grantees n.a. n.a. .042, .038 10 local grantees .073, .068 190, 177 .059, .053 5 local grantees .128, .121 333, 316 .094, .084 

75–25 Treatment-Control Group Assignment Rate 10% of Grant Participants Included in Random Assignment The 16 auto restructuring granteesa n.a. n.a. .080, .071 

10 grantees (5 national and 5 local) .101, .093 264, 243 .089, .080 All 7 national grantees n.a. n.a. .104, .093 5 national grantees .141, .128 366, 334 .132, .118 All 18 local grantees n.a. n.a. .069, .061 10 local grantees .105, .096 274, 250 .098, .087 5 local grantees .175, .162 457, 421 .155, .138 
Pre-Design Study of the ARRA Grants M

Table IV.1 (continued) 

 43 Random Assignment Approach # 2 (Randomly Subsampling Grantees) [Survey, Administrative Data] Random Assignm

(Purposefully Sel

[Survey, Admin

Percentage Employed Average Quarterly Earnings ($) Percentage Employed 

20% of Grant Participants Included in Random Assignment The 16 auto restructuring granteesa n.a. n.a. .057, .051 

10 grantees (5 national and 5 local) .08, .075 209, 196 .063, .057 All 7 national grantees n.a. n.a. .074, .066 5 national grantees .107, .099 278, 257 .093, .083 All 18 local grantees n.a. n.a. .051, .043 10 local grantees .080, .075 209, 194 .069, .061 5 local grantees .139, .130 362, 339 .109, .097 

Source: Author’s calculations based on publicly available information about grantees’ plans for serving partiNote: Because of the likelihood that the cost of conducting an evaluation using Random Assignment Ap

prohibitive, we do not include in the table the MDIs using this approach. 

 Survey sample sizes assumed in calculations are the lesser of 2,500 or the total number of expecte

each group. 

 The MDI formula used for the calculations in Approach 2 is as follows: 

2 2 2 11 (1 )(1 ) (1 )(1 ) # site ind T c FPC factor R c R sites r N r N σρ ρ ⎛ ⎞ × ⎛ ⎞ × − − +− − + ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ ⎝ ⎠ × ×

 where σ is the standard deviation of quarterly earnings ($1,250) or employment status (0.48) b

previous similar studies, FPC is the finite population correction, r is the response rate (0.80 for 

administrative records), R2 is 0.20 both at the site and individual levels, the intraclass correlation ρ i

of treatment and control groups within sites c is 0.70, NT and NC are pertinent sample sizes for t

groups, and #sites is the total number of sites selected in the approach being considered. The MDtwo-tailed tests. We assume 80 percent power and a 5 percent significance level. factor varies as a fuof sites being considered, but generally falls in the range of 2.8 to 3.19. Using variables that are deand with similar values, the MDI formula used for the calculations in Approach 3 is as follows: 
Pre-Design Study of the ARRA Grants M

Table IV.1 (continued) 

 44 2 1 1 (1 ) ind T c factor R rN rN σ ⎛ ⎞ ×− + ⎜ ⎟ ⎝ ⎠ × × . aA grantee is designated as an auto restructuring grantee if it has any component that is targeted to a

community. Not all grantees that plan to serve a community affected by auto restructuring will devote all of th

these communities. MDI = minimum detectable impact. n.a. = not applicable. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

45 

Across the wide range of design strategies that we considered, several general patterns in the 

MDIs emerge. For simplicity, we focus our discussion on the average quarterly earnings as an 

outcome using survey data, although the same patterns are found using the percentage employed 

and/or administrative data. 

First, including a larger number of grantees greatly helps to reduce the MDIs. This finding is to 

be expected given the influence of the clustering of the sample on the MDIs. With random 

assignment Approach # 3, a 10 percent rate of inclusion of grant participants and a 50–50 

treatment-control assignment rate, for example, an increase of the number of local grantees from 5 

to 10 decreases the MDI from $349 to $219. A further increase in the number of local grantees to 

18 decreases the MDI further, to $155. In a similar way, under the same assumptions, an increase in 

the number of national grantees from 5 to 7 leads to a decrease in the MDI from $298 to $235. 

Second, increasing the percentage of the grant participants that are included in the study 

reduces the MDIs, but not as much as does increasing the number of grantees. Take the scenario in 

which 5 local grantees and 10 percent of grant participants at these grantees are included in the 

study. As noted earlier, the average quarterly earnings MDI is $349. One way to double the sample 

size would be to include 10 grantees rather than 5. Another way would be to include 20 percent of 

grant participants rather than 10 percent. The first approach—doubling the number of grantees—

drops the MDI to $219. In contrast, doubling the percent of participants who are included from 

among the 5 grantees would reduce the MDI to $247. All else equal, therefore, inclusion of more 

grantees is likely to be a better approach for improving the precision of the impact estimates than 

would be increasing the sample size within grantees. (Of course, cost and other considerations 

might need to be taken into account, because increasing the number of grantees will be a more 

costly approach.) 

Third, for a given number of grantees, the MDIs are lower when national grantees are included 

instead of local grantees.34 For example, using Approach # 3, with a 50-50 treatment control group 

assignment rate and 10 percent of grant participants included in the evaluation, the MDI for average 

quarterly earnings is $298 when 5 national grantees are selected and $349 when 5 local grantees are 

selected. This is because, based on the information available through DOL’s announcement of 

grant winners, national grantees are larger on average than are local ones. 

Fourth, Approach # 2 yields larger MDIs than does Approach # 3 for a given number or type 

of grantees. For example, a 10 percent rate of inclusion of grant participants and a 50–50 treatmentcontrol assignment rate, random selection of 10 grantees (5 national and 5 local) through Approach 

# 2 yields an MDI of $238. Purposeful selection of the same number and type of grantees through 

Approach # 3 yields a $202 MDI. This pattern occurs because Approach # 2 introduces an 

additional level of statistical uncertainty in the analysis compared with Approach # 3. 

Fifth, as expected, assigning only 25 percent of the sample members to the control group 

increases the MDIs, although in most cases the increase is not dramatic. A 75–25 treatment-control 

assignment rate, in which 10 grantees (5 national and 5 local) and 10 percent of grant participants are 

 34 Because we are not taking into account different weighting or stratification schemes that could be employed 

when selecting a mix of national and local grantees, our analysis does not provide definitive insights about the tradeoffs 

in benefits to the MDIs of selecting a mix of national and local grantees versus selecting all of one or the other type. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

46 

included in the study through Approach # 3 leads to a $233 MDI, in contrast to the $202 MDI 

when a 50–50 assignment rate is used. 

These patterns in the MDIs also are found for the Pathways and SESP grantees (Tables IV.2 

and IV.3, respectively).35 However, when results are compared across grant types, several additional 

insights emerge. Again, for discussion purposes, we focus on the average quarterly earnings 

outcome based on survey data, but the patterns also hold for the employment status outcome 

and/or administrative data. 

First, the MDIs are larger for the Pathways grants than they are for the ETP grants, all else 

equal. For example, using random assignment Approach # 2, a 20 percent rate for participants’ 

inclusion in the evaluation, a 50–50 treatment-control assignment rate, and the selection of 10 local 

grantees, the MDI for the ETP grants is $190 (Table IV.1). In contrast, it is $304 for the Pathways 

grants (Table IV.2). This pattern emerges because, based on the publicly available information 

about grantees’ plans, the target number of participants for Pathways grantees is smaller than that 

for ETP grantees. On average, the ETP grants are about $4 million and include plans to serve about 

1,380 participants; in contrast, the averages for the Pathways grants are about $3.9 million and 540 

participants. 

Second, the MDIs for the SESP grants are relatively small compared with the MDIs shown for 

the ETP and Pathways grants. For example, an average quarterly earnings MDI of $111 can be 

obtained if 40 percent of SESP grant participants go through random assignment, if the treatmentcontrol group assignment rate is 50–50, and if 10 grantees are purposefully selected for inclusion in 

the study through Approach # 3. This MDI is much smaller than the MDIs for the ETP and 

Pathways grantees except for cases in which a large number of ETP or Pathways grantees is selected. 

The low MDIs for the SESP grants occur for two reasons. One is that the SESP grants are expected 

to serve an average of about 1,600 participants, which is a greater number than is the case for the 

other two types of grants.36 The second is that, because the SESP grants are for three years and not 

two, we have assumed that a higher percentage of grant participants can be included in a random 

assignment study. 

 35 The subgroups of grantees shown in Tables IV.2 and IV.3 differ slightly from the subgroups shown in Table 

IV.1 because we have focused our analysis on examples of subgroups that we think are likely to be the most interesting 

to DOL; our selections depend on the numbers of grantees for different grant types and the sizes of the MDIs. 

36 The SESP grants are about $5.5 million on average. As with the other grants, some activities other than direct 

service provision to participants are likely to be included as part of the grants. However, this might be the case especially 

for the SESP grants, which include the development of a steering committee to develop (or refine) and guide the state’s 

energy sector initiatives. 
Pre-Design Study of the ARRA Grants 

47 Table IV.2. Minimum Detectible Impacts on the Percentage Employed and Average Quarterly Earnings, Pat

Grants Random Assignment Approach # 2 (Randomly Subsampling Grantees) [Survey, Administrative Data] Random Assignm

(Purposefully Sel

[Survey, Admin

Percentage Employed Average Quarterly Earnings ($) Percentage Employed 

50–50 Treatment-Control Group Assignment Rate 10% of Grant Participants Included in Random Assignment 10 grantees (5 national and 5 local) .125, .114 325, 298 .112, .101 

All 8 national grantees n.a. n.a. .109, .098 5 national grantees .159, .145 413, 377 .147, .132 All 30 local grantees n.a. n.a. .079, .070 10 local grantees .156, .141 406, 368 .148, .133 

20% of Grant Participants Included in Random Assignment 10 grantees (5 national and 5 local) .097, .090 252, 235 .080, .071 

All 8 national grantees n.a. n.a. .077, .069 5 national grantees .121, .112 315, 291 .104, .093 All 30 local grantees n.a. n.a. .056, .050 10 local grantees .117, .107 304, 279 .105, .094 

75–25 Treatment-Control Group Assignment Rate 10% of Grant Participants Included in Random Assignment 10 grantees (5 national and 5 local) .140, .128 364, 333 .130, .116 

All 8 national grantees n.a. n.a. .126, .113 5 national grantees .180, .163 468, 425 .171, .152 All 30 local grantees n.a. n.a. .091, .081 10 local grantees .177, .160 461, 418 .171, .153 

20% of Grant Participants Included in Random Assignment 10 grantees (5 national and 5 local) .107, .099 278, 258 .092, .082 

All 8 national grantees n.a. n.a. .089, .080 5 national grantees .135, .124 351, 323 .121, .108 All 30 local grantees n.a. n.a. .064, .058 10 local grantees .131, .120 341, 312 .121, .108 
Pre-Design Study of the ARRA Grants 

Table IV.2 (continued) 

48 

Source: Author’s calculations based on publicly available information about grantees’ plans for serving parNote: Because of the likelihood that the cost of conducting an evaluation using Random Assignment A

prohibitive, we do not include in the table the MDIs using this approach. 

 Survey sample sizes assumed in calculations are the lesser of 2,500 or the total number of expe

in each group. The MDI formula used for the calculations in Approach 2 is as follows: 

2 2 2 11 (1 )(1 ) (1 )(1 ) # site ind T c FPC factor R c R sites r N r N σρ ρ ⎛ ⎞ × ⎛ ⎞ × − − +− − + ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ ⎝ ⎠ × ×

 where 

σ is the standard deviation of quarterly earnings ($1,250) or employment status (0.48) b

previous similar studies, FPC is the finite population correction, r is the response rate (0.80 for 

administrative records), 

R2 is 0.20 both at the site and individual levels, the intraclass correlation 

ρ i

of treatment and control groups within sites c is 0.70, 

NT and NC are pertinent sample sizes for t

groups, and #sites is the total number of sites selected in the approach being considered. The MDtwo-tailed tests. We assume 80 percent power and a 5 percent significance level. factor varies as a fuof sites being considered, but generally falls in the range of 2.8 to 3.19. Using variables that are deand with similar values, the MDI formula used for the calculations in Approach 3 is as follows: 

2 1 1 (1 ) ind

T c

factor R rN rN

σ

⎛ ⎞

×− + ⎜ ⎟ ⎝ ⎠ × ×

. 

aA grantee is designated as an auto restructuring grantee if it has any component that is targeted to a

community. Not all grantees that plan to serve a community affected by auto restructuring will devote all of th

these communities. MDI = minimum detectable impact. n.a. = not applicable. 
Pre-Design Study of the ARRA Grants 

49 Table IV.3. Minimum Detectible Impacts on the Percentage Employed and Average Quarterly Earnings, 

Partnership Grants Random Assignment Approach # 2 (Randomly Subsampling Grantees) [Survey, Administrative Data] Random Assignm

(Purposefully Sel

[Survey, Admin

Percentage Employed Average Quarterly Earnings ($) Percentage Employed 

50-50 Treatment-Control Group Assignment Rate 30% of Grant Participants Included in Random Assignment All 21 auto restructuring granteesa n.a. n.a. .040, .026 

10 auto restructuring grantees .069, .065 180, 170 .050, .045 15 grantees .057, .050 149, 131 .042, .031 10 grantees .070, .068 183, 176 .043, .039 8 grantees .083, .080 217, 209 .050, .045 5 grantees .125, .121 325, 314 .072, .065 

40% of Grant Participants Included in Random Assignment All 21 auto restructuring granteesa n.a. n.a. .040, .023 

10 auto restructuring grantees .064, .061 168, 160 .044, .039 15 grantees .057, .048 149, 125 .042, .027 10 grantees .070, .065 181, 169 .042, .034 8 grantees .080, .077 208, 201 .044, .039 5 grantees .119, .116 311, 303 .063, .056 

75-25 Treatment-Control Group Assignment Rate 30% of Grant Participants Included in Random Assignment All 21 auto restructuring granteesa n.a. n.a. .041, .030 

10 auto restructuring grantees .075, .070 196, 183 .059, .052 15 grantees .060, .053 157, 139 .046, .036 10 grantees .076, .071 198, 185 .053, .045 8 grantees .089, .084 231, 220 .059, .052 5 grantees .131, .126 342, 328 .084, .075 

40% of Grant Participants Included in Random Assignment 

All 21 auto restructuring granteesa n.a. n.a. .040, .026 
Pre-Design Study of the ARRA Grants 

 50 Table IV.3 (continued) Random Assignment Approach # 2 (Randomly Subsampling Grantees) [Survey, Administrative Data] Random Assignm

(Purposefully Sel

[Survey, Admin

Percentage Employed Average Quarterly Earnings ($) Percentage Employed 

10 auto restructuring grantees .071, .065 185, 170 .053, .045 15 grantees .058, .050 150, 131 .043, .031 10 grantees .073, .068 190, 176 .048, .039 8 grantees .085, .080 222, 209 .053, .045 5 grantees .125, .121 325, 314 .072, .065 

Source: Author’s calculations based on publicly available information about grantees’ plans for serving parNote: Because of the likelihood that the cost of conducting an evaluation using Random Assignment A

prohibitive, we do not include in the table the MDIs using this approach. Survey sample sizes assumed in calculations are the lesser of 2,500 or the total number of expein each group. The MDI formula used for the calculations in Approach 2 is as follows: 

2 2 2 11 (1 )(1 ) (1 )(1 ) # site ind T c FPC factor R c R sites r N r N σρ ρ ⎛ ⎞ × ⎛ ⎞ × − − +− − + ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ ⎝ ⎠ × ×

 where 

σ is the standard deviation of quarterly earnings ($1,250) or employment status (0.48) b

previous similar studies, FPC is the finite population correction, r is the response rate (0.80 for 

administrative records), 

R2 is 0.20 both at the site and individual levels, the intraclass correlation 

ρ i

of treatment and control groups within sites c is 0.70, 

NT and NC are pertinent sample sizes for t

groups, and #sites is the total number of sites selected in the approach being considered. The MDtwo-tailed tests. We assume 80 percent power and a 5 percent significance level. factor varies as a fuof sites being considered, but generally falls in the range of 2.8 to 3.19. Using variables that are deand with similar values, the MDI formula used for the calculations in Approach 3 is as follows: 

2 1 1 (1 ) ind

T c

factor R rN rN

σ

⎛ ⎞

×− + ⎜ ⎟ ⎝ ⎠ × ×

. 
Pre-Design Study of the ARRA Grants 

 51 Table IV.3 (continued) 

aA grantee is designated as an auto restructuring grantee if it has any component that is targeted to a

community. Not all grantees that plan to serve a community affected by auto restructuring will devote all of th

these communities. MDI = minimum detectable impact. n.a. = not applicable. 

Pre-Design Study of the ARRA Grants Mathematica Policy Research 

53 

V. OTHER TYPES OF METHODOLOGIES 

Although random assignment is generally considered to be the most rigorous approach for 

estimating impacts of an intervention, other types of methods can be used to estimate impacts when 

random assignment is infeasible or undesirable. The main limitation of these methods, such as 

those that rely on a comparison group, stems from the fact that all nonexperimental designs suffer 

to some degree from a risk that factors other than the intervention might explain the patterns in 

outcomes that are observed. In such a situation, one cannot have confidence that the intervention 

itself (such as ARRA-funded training) is responsible for any detected effects. 

In this chapter, we describe three nonexperimental designs: regression discontinuity (RD), 

comparison group, and interrupted time series designs. Although each of these designs is less 

rigorous than a random assignment design, other features of these designs may avoid or mitigate 

some of the implementation challenges associated with random assignment, as discussed in Chapter 

IV. 

Although DOL has expressed an interest in pursuing an evaluation of individual-level 

outcomes, we include a brief discussion of designs that could be used to examine community-level 

outcomes because these designs could complement an analysis of individual-level outcomes. In 

Section A, we describe the RD design in general and two variations on that design that would be 

appropriate for an evaluation of ARRA grants. Section B describes two comparison group designs, 

one based on examining individual-level outcomes, and the other based on examining grantee-

/community-level outcomes. Finally, Section C briefly describes an interrupted time series design. 

Variations of each of these three designs could be implemented to look at either individual- or 

community-level outcomes.37 Our discussions in these sections focus on the analytical differences in 

these methodologies, given the scope of this pre-design study. However, if policymakers would like 

to use one of these approaches, the decision about which methodology is preferable will likely need 

to take into account other considerations that are not discussed here, such as the costs of the 

different designs. 

A. Regression Discontinuity with Individuals in Winning Grantees or Grant 

Applicants 

As described in Chapter IV, the goal of a random assignment design is to ensure that the 

individuals in the treatment and control groups are similar on both observed measures, such as age 

and education level, and unobserved measures, such as underlying motivation. This equivalence of 

the two groups enables estimation of what would have happened to treatment group members in 

the absence of the intervention (in this case, the ARRA grant-funded training). 

Although random assignment designs use a random selection process to assign individuals to 

the treatment and control groups, another method, called RD, takes advantage of a systematic, 

nonrandom allocation into a treatment group that can receive the intervention and a group that 

 37As discussed in Chapter III, examples of outcomes for individuals who are offered training financed by an ARRA 

grant include employment status and average quarterly earnings. Examples of community-level outcomes, which look at 

the whole community rather than the individuals within it, could include the number of new jobs created in the target 

industries. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

54 

cannot. A recent study indicates that, under certain conditions, an RD design can provide impact 

estimates comparable to those of a random assignment study conducted on the same population 

(Cook & Wong, forthcoming). 

In this section, we first provide an overview of the RD design generally and the conditions 

under which it provides valid impact estimates. We then describe two variations of the general 

design that would be appropriate for an evaluation of ARRA grants. The section concludes with a 

discussion of the relative strengths and weaknesses of the RD designs considered. 

1. Overview of RD Designs 

RD designs can be applied when a rule based on a continuous numerical variable is used to 

determine the eligibility of a person or entity for an intervention. This rule can be used to generate 

two groups—eligible and ineligible for the intervention—that can be distinguished from each other 

on the basis of a cutoff score. For example, applicants for grant funding might be reviewed and 

given a score that is used to determine those applicants who receive funding and those who do not. 

The group of applicants that receives funding forms the treatment group; the other group, which 

does not receive funding, forms the comparison group. Of critical importance for this design to 

provide valid estimates of effects, the score must be assigned according to well-designed and 

replicable rules that were adhered to during the scoring process. 

When using an RD design, the effects of the intervention being studied—for example, the grant 

funding mentioned earlier—is estimated exactly at the eligibility cutoff score as the difference in outcomes 

between the group that receives the intervention and the group that does not. More specifically, a 

line or curve is fit to the data for the intervention group and another line or curve is fit to the data 

for the comparison group to model statistically the relationship between the outcome of interest and 

the eligibility score (see Figure V.1). The estimated impact of the intervention is the difference in 

outcomes for individuals or entities just below and just above the cutoff score. Stated differently, 

the intervention has a nonzero impact if there is a “discontinuity” in the two lines or curves—hence 

the “regression discontinuity” name for this study design. All types of evaluation designs need to have 

large enough samples of people or entities to ensure that effects of an intervention can be estimated 

precisely, but an evaluation that uses an RD design needs a much larger sample (on the order of 

three times as large for nonclustered designs [Goldberger 1972] and three to four times as large for 

clustered designs [Schochet 2009]) than an evaluation of the same intervention that uses random 

assignment. This is because of the substantial correlation between the treatment status and the 

score variables that are included in the impact models. 

For this design to be appropriate, it is critical that applicants just above and just below the 

cutoff score are virtually identical, and differ only because those just above (or just below) the cutoff 

qualify for the intervention. Ultimately, this means that the scoring variable is the only variable that 

determines one’s assignment to the treatment or comparison group. This condition can be checked 

through a statistical examination of whether the treatment and comparison groups are alike on key 

characteristics measured prior to the start of the intervention. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

55 

Figure V.1. Visual Interpretation of a Regression Discontinuity Design 

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

12,000 12,500 13,000 13,500 14,000 14,500

Outcome (such as Average 

Quarterly Earnings, in dollars)

Estimated Impact

Note: This figure uses hypothetical data and regression lines to demonstrate a regression discontinuity design 

with an outcome such as average quarterly earnings on the Y-axis and annual income used to determine 

eligibility for an intervention on the X-axis.

.  .    

.  . . .    .  .

.  .

.  . . .  .   .    .

.  . . . . .

.  . . . .    .   .  . .  .

.  .

.  .

Income Used to Determine Eligibility for an Intervention, in dollars

.   

.  .    ..

...      

.

.

.  . Cutoff Income

2. RD Designs for an Evaluation of ARRA Grants 

We consider two types of RD designs for an evaluation of ARRA grants. The first design, 

which we refer to as the individual-level RD, uses a cutoff value based on income at the individual 

level at winning grantees to examine individual-level outcomes of interest. The second design, 

which we refer to as the grantee-level RD, uses variation in the scores of grant applicants to divide 

grantees into treatment and comparison groups. With this latter approach, outcomes could be 

examined at either the individual or community level. 

a. Individual-Level RD with Winning Grantees 

An individual-level RD approach would compare the outcomes of individuals in the treatment 

group with those in a comparison group, where membership in the treatment group is based on an 

observable and precise cutoff. For instance, it is often the case that there are income restrictions for 

participation in training; individuals whose income is too high (as determined by WIA regulations) 

are not eligible to participate. This income cutoff can serve to identify members of the treatment 

and comparison groups whose outcomes can be compared to give a credible estimate of the 

treatment’s effectiveness. For instance, suppose the income cutoff were $20,000, so that individuals 

who apply for training but whose incomes are above that amount were not eligible for training. The 

RD design could compare the outcomes of individuals with income just above $20,000 (control 

group) with the outcomes of individuals with income just below $20,000 (treatment group). The 

intuition is that individuals close to the cutoff income limit are similar, on average, and thus the 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

56 

design can generate credible estimates of treatment effects. It should be noted that it is not 

necessary that every grantee use the same cutoff; in fact, that could be advantageous for estimating 

impacts across a range of cutoff scores. 

As applied to an evaluation of ARRA grants, the individual-level RD design would be 

appropriate for grantees in which the following three conditions are met. First, there must be some 

type of continuous program eligibility cutoff that can be used to establish the discontinuity. 

Although the grantees receiving the Pathways out of Poverty grants, which are specifically targeted 

to high-poverty communities, might use an income cutoff to determine an individual’s eligibility for 

grant-funded services, ETP grantees that do not specifically target high-poverty individuals might 

not have any eligibility criteria that would allow an RD design to be used. Second, the cutoff must 

not also be used to determine eligibility for other, non-ARRA services, as this would make it difficult 

to isolate the effect attributable to ARRA grants. For instance, Pathways out of Poverty grants may 

use the federal poverty line to determine eligibility, but since this also determines eligibility for a host 

of other services it would not be an appropriate cutoff for an RD design. Finally, the cutoff must 

not be subject to widespread manipulation on the part of participants or program staff at the 

grantee. 

b. Strengths and Weaknesses of an Individual-Level RD Approach 

The RD design in general has several advantages compared with other types of designs. 

Compared with a random assignment design, this approach has two advantages. First, it often can 

be applied even after programming has begun, as long as the scoring process has been documented. 

Second, the approach requires minimal interference with normal program activity; in short, grantees 

are likely to be less burdened under this design than a random assignment design. Compared with 

other nonexperimental methodologies, discussed later in this chapter, this approach is more 

rigorous; when the scoring rule is fully known and determined exogenously, selection bias is 

minimized. 

However, there are a few potential drawbacks to this design. First, as previously discussed, it 

would apply only to grant types in which there is some eligibility threshold for training, such as the 

income cutoff mentioned earlier. Second, in order for this design to produce credible estimates, it is 

necessary that there be minimal manipulation of the score used to determine eligibility by either 

individuals who desire grant services or program staff, who presumably will be eager to provide 

services. Third, this design requires sample sizes about three times larger than a random assignment 

design to estimate effects to the same level of precision. Fourth, compared with a random 

assignment design, external validity is more limited, because it applies only to individuals close to the 

cutoff score; this would be mitigated if a range of scores were used across grantees. Finally, unless 

extensive baseline information has been collected before an individual’s eligibility determination, it 

may be difficult to obtain outcome data (both administrative and survey) for those who were 

deemed ineligible.38 

 38 If outcome data are to come exclusively from administrative wage records, then the identifying information that 

is used to match with these records, such as Social Security numbers, would be necessary. If some of the outcome data 

are to come from follow-up surveys, then a more extensive set of identifying and locating information would be 

necessary to avoid differential rates of response to the follow-up survey. This information might include the individuals’ 

names, dates of birth, addresses, and telephone numbers, and other contact information. Some of this identifying 

information might normally be collected as part of the process to access an individual’s eligibility for grant services. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

57 

c. Grantee-Level RD with Grant Applicants 

A grantee-level RD approach could use the scores assigned during the grant award process to 

determine treatment status. As applied to an evaluation of ARRA grants, this design would be 

appropriate for those grant types wherein many more applications were made than grants available, 

and grants were awarded based on a well-defined and adhered-to scoring process. Although 

information on the number of grant applicants is not available, it seems unlikely that a grantee-level 

RD approach would be feasible for the SESP grants. The only types of eligible grantees were state 

workforce investment boards, in which the term “state” pertains to the 50 states, the District of 

Columbia, and U.S. territories. Because 34 SESP grants were awarded, it is likely that there were too 

few nonwinning applicants for the grantee-level RD approach to be appropriate in an evaluation of 

the SESP grants. However, it seems likely that the other three types of ARRA grants had a larger 

number of grant applicants, which might help to ensure that this methodological approach is 

suitable. 

If a grantee-level RD approach is used for an evaluation, the outcomes could be either at the 

individual level or at the community level. If individual-level outcomes are of interest, the outcomes 

of individuals served by the winning grantees would be compared with the outcomes of individuals 

in communities served by grant applicants who were not awarded a grant. But at the individuallevel, it is unclear who would comprise a suitable comparison group. Although the applicants for 

ARRA grant funding were instructed to specify their target groups for services, should they be 

awarded a grant, it is likely to be challenging for evaluators to define and collect data on such 

individuals or a subset of them. Although an applicant who did not receive an award might serve 

similar types of customers to those who would have been served with an ARRA-funded grant, one 

could never be sure that this is the case and that there was no selection bias of who forms the 

individual-level comparison group sample. A best-case scenario, from an evaluation perspective, 

would be to focus on applicants that did not receive an award but that had specified plans to serve 

participants who are very similar to adult or dislocated workers. Information on these individuals’ 

outcomes could be made available from a centralized data system maintained by the local workforce 

investment areas (LWIAs).39 Even in this case, however, it would not be possible to identify which 

members of the comparison group would have received training at the grantee, had the grantee won; 

thus, there would be concern about selection bias in the sample. 

In other cases, the target groups of nonwinning grant applicants might be difficult to find for 

inclusion in the study, such as when a grant applicant planned to serve workers who were 

detrimentally affected by national energy and environmental policy. These individuals might not 

request services or be traceable through any other method that uses a centralized data system. 

If community-level outcomes are of interest, the outcomes in communities served by winning 

grantees would be compared with the outcomes in communities served by grant applicants who 

were not awarded a grant. For instance, the Quarterly Workforce Indicators could be used to 

 

(continued) 

However, Mathematica’s experience is that follow-up surveys are most likely to be successful if additional information is 

collected. An implication of this is that, like a random assignment design, the RD design could benefit from having the 

evaluation design in place prior to the start of participants’ enrollment in grant activities. 

39 As discussed in Chapter III, there is a possibility that many of the individuals in communities that applied for but 

did not receive ARRA grants will access similar types of services through other funding sources. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

58 

provide quarterly measures of total employment; measures of change such as job flow, new hires, 

and separations; and average earnings by county, metro area, WIA, or by two-digit industry code. 

d. Strengths and Weaknesses of a Grantee-Level RD Design 

The grantee-level RD design shares the advantages noted for the individual-level RD design 

mentioned earlier: namely, it is a rigorous nonexperimental design, it can be applied after 

programming has begun, and it requires minimal interference with normal program activity. In 

addition, it would likely be less cumbersome to collect outcome data on communities served by 

grant applicants rather than individual-level data because community-level outcomes can more 

readily be accessed through existing data. 

However, the design has several drawbacks. First, the treatment assignment rule may not be 

perfectly known; although DOL used a committee to determine the ranked scores of grant 

applicants based on well-specified criteria, it could also take into account other factors in 

determining awards, such as geographic representativeness, which might compromise the rank 

ordering necessary to establish the discontinuity. Second, given the limited number of grant 

applicants for each grant type, even if this approach is technically feasible, this design would have 

limited statistical power. Third, external validity is more limited with this approach, as it applies only 

to communities served by grant applicants, or individuals within them, close to the cutoff score. 

Finally, as discussed earlier, an examination of individual-level outcomes using the grantee-level RD 

approach necessitates identifying an appropriate set of individuals in communities that were not 

awarded grants and gathering data on them. 

B. Comparison Group Designs 

In some cases in which random assignment is not feasible or appropriate, and in which the 

conditions for an RD design are not likely to be met, it might be possible to use a comparison group 

design to develop an estimate of the impact of an intervention. A comparison group design 

attempts to compare the outcomes of people or entities that are as similar as possible with the group 

that receives the intervention in order to reduce potential selection bias when estimating treatment 

effects. 

In this section, we first provide a general overview of comparison group designs, followed by 

two specific designs that might be used for an evaluation of ARRA grants. We also present the 

strengths and weaknesses associated with each of the designs. 

1. Overview of Comparison Group Designs 

In a comparison group design, observable characteristics are used to create a comparison group 

composed of individuals or entities that look very similar to treatment group members. The 

assumption is that, because the comparison group is similar to the treatment group on observed 

variables, it must also be fairly similar on unobserved variables that have an influence on the 

outcomes of interest. This is a strong assumption, and for this reason comparison group designs are 

generally thought to be less rigorous than either (1) a random assignment design, which uses a 

random allocation to assign people to groups; or (2) an RD design, which uses a systematic 

allocation process to assign people to groups. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

59 

However, when a comparison group is selected, the outcomes of treatment group participants could 

be compared with the outcomes of the individuals in the comparison group to obtain an estimate of 

the impact of an intervention (see Figure V.2). As the figure suggests, the comparison group 

approach has similarities to a random assignment design, but there is one very important difference: 

Although researchers could try to take into account the differences between the treatment and 

comparison groups that they can observe and measure in a comparison group design, they cannot 

guarantee that they have accounted for all important differences between the two groups. 

2. Comparison Group Designs for an Evaluation of ARRA Grants 

We consider two types of comparison group designs for an evaluation of ARRA grants. The 

first design, which we refer to as the individual-level propensity-score matching design, uses a 

statistical routine to match treatment group members to comparison individuals who are 

observationally similar; the comparison group members are selected as the best matches to the 

treatment group from among a large set of potential comparison group members. Then, the 

outcomes of the treatment and comparison group members are compared. The second design, 

which we refer to as the community-level difference-in-differences design, uses variation in 

community-level outcomes both across time and across communities to determine treatment 

effectiveness. 

Figure V.2. Impact Estimation from a Comparison Group Design 

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

5,000

Comparison Treatment

Outcome

(such as Average

Quarterly Earnings, in dollars)

Estimated

Impact

Note: This figure uses hypothetical data to demonstrate a comparison group design with an 

outcome such as average quarterly earnings on the Y-axis.
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

60 

a. Individual-Level Propensity–Score Matching Design 

A relatively rigorous individual-level comparison group design would first match treatment 

group members (that is, individuals in winning grantees) to similar individuals who did not receive 

the treatment via propensity-score matching. It would then assess treatment effects by comparing 

the outcomes of those in the treatment group with those in the matched comparison group. 

Statistical theory and some empirical evidence indicates that use of propensity-score matching is 

sufficient to remove bias between treatment and control group members due to the differences 

between the two groups on their observed characteristics (Rosenbaum and Rubin 1983). 

Propensity-score matching to select a comparison group from among a pool of potential 

comparison group members entails three steps: (1) identify through a regression how the 

characteristics of treatment and potential comparison group members are associated with the 

likelihood of being in the treatment group; (2) estimate a single number (a “propensity score”) for 

each individual to represent this likelihood; (3) identify which potential comparison group 

member(s) are close enough to a treatment group member on this score, and then select them to be 

in the matched comparison group. The resulting comparison group can then be assessed to see how 

similar its members are on observable characteristics to the treatment group’s observable 

characteristics, on average. In general, the most promising comparison groups can be drawn when 

the treatment and comparison group members are drawn from the same local market areas, a 

common data source contains the matching variables for both groups, and rich pre-intervention 

measures of the outcomes are available. 

Because ARRA grantees were given considerable flexibility in choosing the target groups that 

they could plan to serve, the propensity-score matching approach would need to be similarly 

flexible. First, the pool of potential comparison group members for each grantee that is included in 

an evaluation using this approach must be chosen carefully to correspond with individuals from 

whom participants could be drawn. For instance, some ETP and SESP grantees plan to serve 

workers who are affected by national energy and environmental policy. For these grantees, one 

could try to identify dislocated workers to use for a comparison group pool, even though there is 

unlikely to be a data item in LWIA records that pertains precisely to this characteristic. 

Second, the precise list of characteristics to include in each propensity-score matching model is 

likely to depend on the specific target group chosen by each grantee. For instance, the Pathways 

grantees were allowed to target adults who are unemployed, high school dropouts, have a criminal 

record, or are disadvantaged and living in areas of high poverty. Many of these attributes will be 

shared by formula-funded WIA adult customers who have been and will continue to be served in 

the grant communities; therefore, WIA adult customers could form a potential pool of individuals 

from which to draw comparison group matches to the Pathways-funded participants. Given the 

target population of Pathways grants, it is likely important for the propensity-score model in this 

case to include indicator variables for having a criminal record, noncompletion of high school, and 

living in a high poverty area in order to ensure the comparison group members are as similar as 

possible to grant participants. 

It might be more challenging in some other cases to define an appropriate pool of potential 

comparison group members and obtain sufficient data on observable characteristics to yield a highquality match of comparison group members to the treatment group. For example, if a Health Care 

grantee targets incumbent workers for the provision of training, then it might not be appropriate to 

use WIA adult or dislocated worker customers as the source of the comparison group pool. Other 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

61 

incumbent workers from similar employers might be another source for the pool, but it is likely that 

these incumbent workers will differ from the ARRA participants on unobservable characteristics 

because many of them are unlikely to pursue training and career development. 

Regardless of the expectations about the quality of the source for potential comparison group 

members, the evaluation will need to assess the quality of the match process after a comparison 

group is selected. In some cases, it might be necessary to refine the propensity-score model and 

select another group. 

Finally, we note that the propensity-score matching approach could also theoretically be 

conducted at the grantee level. This would involve matching (based on observable characteristics) 

communities served by winning grantees to similar communities that were not served by ARRA 

funds. However, given the potentially limited number of comparison pool members, it is probably 

not feasible to implement this approach. 

b. Strengths and Weaknesses of Propensity-Score Matching Relative to Other Designs 

The main strength of the propensity-score matching design is that it allows for an evaluation of 

program effectiveness even if all or most individuals received ARRA-funded training, as long as a 

suitable comparison group can be found. Like the RD design, the comparison group strategy using 

propensity-score matching at the individual level can be applied after programming has begun and 

with minimal interference with normal program activity. This design also takes into account a wide 

range of observable characteristics to minimize the selection bias associated with impact estimates. 

However, the approach has several potential drawbacks. First, it is not as rigorous as a wellimplemented randomized control trial or RD design. Second, because of the diversity of the target 

populations served by grantees as mentioned previously, the pool of comparison group members 

may have to be tailored to each grantee, which can be a labor-intensive process. Moreover, when a 

suitable comparison pool has been identified, the matching algorithm requires extensive baseline 

data on the comparison pool members, which may not be readily available. Ultimately, this 

approach presents a risk that one might have to conclude that the highest-quality comparison group 

available differs significantly from the treatment group—a finding that would cast doubt on 

estimates of the impacts of the ARRA grants on participants’ outcomes. 

c. Community-Level Difference-in-Differences Design 

A difference-in-differences design, which requires less intensive data collection than the method 

described previously, can be applied when using a comparison group design to examine outcomes at 

the community level. In this design, one needs to find a suitable comparison group that did not 

receive the treatment and compare the change in outcomes for the treatment group over time with 

the change in outcomes over time for a suitable comparison group that did not receive the 

intervention. The use of a comparison community helps to control for influences, such as 

macroeconomic trends and changes in other funding sources, occurring at the same time the 

treatment was administered and that might affect post-treatment levels of the outcomes in both 

comparison and treatment groups. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

62 

Figure V.3 shows how the difference-in-differences approach works. In the figure the change 

over time for the treatment group is the difference between points C and F, whereas the change for 

the comparison group is the difference between points D and E. However, the change over time 

for the treatment group can be thought of as consisting of two parts. The first part, represented by 

the difference between points C and G, is what would have happened to the treatment group if 

there were no intervention. (That is, the treatment group would have experienced the same change 

as was experienced by the comparison group, because point G is defined such that the slope of the 

lines between A and G is the same as the slope of the line between B and E). The second part, 

represented by the difference between F and G, is the change that is unique to the treatment group. 

Figure V.3. Visual Interpretation of a Difference-In-Differences Design 

Outcome (such as 

number of jobs in green 

industries)

Baseline Follow-Up Time

F

G

C

E

D

Change Attributable to 

Intervention

Hypothetical Change for 

Intervention Group

Change for Comparison 

Group

A

B

Note: This figure uses hypothetical regression lines to demonstrate a difference-indifferences design with an outcome such as number of jobs in green industries 

on the Y-axis and time on the X-axis.
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

63 

By taking the difference between pre- and post-period outcomes in the treatment group and 

comparing those with the difference in pre- and post-period outcomes in the comparison group, a 

“difference-in-differences” estimate of the program effects is obtained. For example, suppose that 

from 2010 to 2011 the number of green jobs created in treatment communities increased by 3,000 

(that is, the difference between C and F = 3,000, after controlling for other factors). Suppose that 

over that same period the number of jobs created in comparison communities increased by 1,000 

(that is, the difference between D and E = 1,000). Then, one could credit an increase of 2,000 jobs 

(= the difference between F and G) to the treatment. 

Similar to the propensity-score matching approach described earlier, the suitable comparison 

groups would likely vary depending on the type of grant being evaluated.40 For example, Pathways 

grantees need to serve high-poverty communities as defined by Public Use Microdata Areas 

(PUMA) data. One could use the PUMA data to identify other communities of similar size and in 

the same areas of the country that have similar poverty rates to form the comparison group. For the 

auto-related subsets of the Health Care and ETP grants, one could use the 312 auto restructuring 

communities that did not receive grants and were in similar geographic areas as the comparison 

group. 

The ETP and Pathways out of Poverty grantees may have credible comparison groups if it is 

the case that the national networks that were awarded grants had local affiliates to whom they did 

not distribute funds; the communities served by affiliates who did not receive funds could be 

compared with the communities served by affiliates who did receive funds. Finally, the SESP grants 

were awarded to some substate areas and not others; the areas that did not receive ARRA funding 

through the grants could serve as comparison communities for those that did receive funding. 

We note that a difference-in-differences approach could also be used to examine individuallevel outcomes. The difference-in-differences approach benefits from extensive historical data on 

outcomes in order to establish the pre-intervention trends experienced by both groups. This would 

likely be difficult and resource-intensive to obtain at the individual-level. 

d. Strengths and Weaknesses of the Community-Level Difference-in-Differences Design 

The main strength of the community-level difference-in-differences design is that it allows for 

an evaluation of program effectiveness, even if all or most grant applicants have received funding, as 

long as a suitable comparison group can be found. The design can be implemented after 

programming has begun with no program interference (because outcomes are collected at the 

community level). This approach also accounts for other secular trends that may be occurring in 

treatment and comparison communities even in the absence of the treatment, thereby providing a 

more confident assessment of program effectiveness. 

The main drawback to this design is that one may still have concerns that unobservable 

characteristics influence why some sites were part of the grant activities and others were not. For 

instance, in comparing the local affiliates of national networks that received funding to those that 

 40 In contrast to the propensity-score matching approach, there is no statistical routine to match comparison 

communities. Rather, they are selected on a more ad hoc basis. For this reason, this approach is rated as less rigorous 

than a propensity-score matching approach. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

64 

did not, one might worry that there were unobserved reasons the national network decided to 

distribute funds to those particular affiliates. It might be the case that they only distributed funds to 

the affiliates they thought were most likely to be successful; these affiliates may also have been the 

most successful even in the absence of the ARRA funding. 

In addition, the difference-in-differences approach requires extensive data on characteristics of 

grantee communities and potential comparison communities for the period preceding the 

intervention in order to establish a pre-intervention trend with adequate precision. Although 

extensive outcome data are collected at the county and metro area in sources such as the Quarterly 

Workforce Indicators, it may be difficult to obtain data on potential comparison communities within 

those broader areas. For instance, obtaining data on job creation in high-poverty communities 

within a county may be difficult.41 Finally, doubts will remain as to how comparable the treatment 

communities are with the comparison communities that were chosen. 

C. Interrupted Time Series Design 

In cases in which random assignment is not feasible, there is no selection variable to use as a 

cutoff score for an RD design, and finding a suitable comparison group is not feasible, it might be 

possible to use an interrupted time series design, also known as a pre-/post- or before/after design, 

to estimate the impact of an intervention. In contrast to the designs discussed previously, an 

interrupted time series design does not involve a control or comparison group. Because this design 

is far less rigorous than other designs, we provide only a brief description and discussion of the 

strengths and weaknesses of the approach. 

1. Overview of Interrupted Time Series Designs 

In an interrupted time series design, outcomes of the treatment group after the intervention is 

implemented are compared with the outcomes of the treatment group before the intervention was 

implemented. This approach is based on an assumption that any differences between the outcomes 

before and after the intervention can be fully explained by characteristics that can be quantified and 

used in a statistical analysis. After controlling for these characteristics, the remaining difference 

between the pre-intervention outcomes and the post-intervention outcomes is attributed to the 

intervention (see Figure V.4). When this type of study design is used, it is very important to have 

data on a long history of relevant outcomes before and after the intervention, so that the pre- and 

post-intervention trajectories shown in the figure can be precisely estimated. 

This design can be used to examine outcomes at either the individual or grantee level. For 

example, one could look at the outcomes of individuals before and after they participate in ARRAfunded training. Alternatively, for an analysis at the community level, one could look at communitylevel outcomes before and after grant receipt. 

 41 As described in Chapter II, DOL did not specify a target size for the communities to be served by each grant 

type. However, for the Pathways out of Poverty grants, DOL expected that the target communities would contain 

between 10,000 and 100,000 people. For urban areas, those communities might be neighborhoods within a city rather 

than the entire city; for rural areas, they might be collections of between one and three counties. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

65 

2. Strengths and Weaknesses of Interrupted Time Series Designs 

The interrupted time series design has the same advantages as the difference-in-differences 

design described earlier, with the additional advantage that finding suitable comparison individuals 

and communities and collecting data on them are not necessary. However, this design requires 

detailed baseline information data and benefits from extensive historical information, which may be 

difficult to collect for individuals (less so for communities). 

Moreover, any advantages to this design come at a significant cost, namely that changes in 

outcomes could have occurred over time because of other factors that are not taken into account 

statistically or because of unknown or unanticipated changes that coincide in time exactly with the 

intervention being studied; the study could incorrectly attribute to the intervention the effects of 

changes that have nothing to do with it. Concerns about other changes that coincide with the 

ARRA grants and that might explain estimated impacts would be especially severe given the 

magnitude and characteristics of the recent economic recession. 

Figure V.4. Visual Interpretation of an Interrupted Time Series Design 

3,000

4,000

5,000

6,000

7,000

8,000

9,000

2007 Q1 2008 Q1 2009 Q1 2010 Q1 2011 Q1

Outcome (such as Average 

Quarterly Earnings, in 

dollars)

Estimated Impact

Note: This figure uses hypothetical data and regression lines to demonstrate an interrupted time 

series design with an outcome such as average quarterly earnings on the Y-axis and time on 

the X-axis.

Time

. 

. 

. 

. 

. . 

. . 

. . . 

. . 

. . . 

. 

. . 

Pre-Design Study of the ARRA Grants Mathematica Policy Research 

67 

VI. SUMMARY OF CONSIDERATIONS WHEN PROCEEDING WITH AN 

EVALUATION OF THE ARRA GRANTS 

As explained in Chapter I, the purpose of this white paper is to provide guidance to DOL as it 

assesses potential methodologies for evaluating one or more of the four sets of ARRA grants: (1) 

Energy Training Partnership Grants; (2) Pathways Out of Poverty Grants; (3) State Energy Sector 

Partnership and Training Grants; and (4) Health Care Sector and Other High-Growth and Emerging 

Industries Grants. Although these sets of grants share many similarities, they differ by providing 

funding to different types of grantees and, in some cases, by targeting different industries. 

Furthermore, the grantees will have different target populations; some focus on workers who are 

unemployed and in need of basic skills development and others focus on workers dislocated from 

failing industries or those who are already in high-growth and emerging industries but in need of 

updated skills. 

Using the publicly available information about the grants, as well as our prior experience 

conducting evaluations for DOL and other clients, we have discussed possible outcomes of interest 

to DOL, which are summarized in Table VI.1. We have compared and contrasted a wide range of 

possible approaches for an evaluation of the grants. Inherently, we have had to make many 

assumptions about how a potential evaluation might unfold, and our discussion of each potential 

approach is not as comprehensive as would be expected during the design phase of an evaluation, 

when more information would be available and fewer options under consideration. To facilitate a 

comparison of the approaches, however, we have summarized their primary strengths and 

weaknesses in Table VI.2. 

To facilitate comparison of the relative rigors of the different methodologies, we also have 

provided a four-tier categorization of the rigor of each of the approaches: (1) very high, (2) high, (3) 

moderate, and (4) low. While this categorization inherently is somewhat subjective, it is based on 

our understanding of consensus views within the research community about the rigor of different 

methodological approaches. It also takes into account our best guesses at how the application of a 

methodology in the distinctive context of the ARRA grants will influence its rigor. Ultimately, we 

intend for this categorization to serve as a guide, rather than an absolute metric, in the assessment 

and comparison of the approaches. 

Importantly, we have not recommended a single methodological approach as the best way to 

evaluate these grants; rather, we discuss the advantages and disadvantages of different 

methodological approaches so that DOL can consider its options for proceeding with any evaluation 

of these grants. 

Although we do not recommend how to proceed with an evaluation, the analysis presented in 

this paper provides insights that are likely to be useful to DOL as it proceeds with planning an 

evaluation of the ARRA grants. We briefly summarize the main insights below. 

Three possible objectives for an evaluation could be of interest to DOL. The first, which 

we have focused on given DOL’s expressed interest in it, is learning about the microeconomic 

(individual-level) effects on workers of training funded by ARRA grants. However, other objectives 

could include learning about the macroeconomic effects of the grants on communities or industries 

and learning about implementation issues associated with the grants. Studies that examine 

macroeconomic or implementation issues could be done either in concert with an analysis at the 

individual level or instead of such an analysis. Regardless of the specific objectives of an evaluation 
Pre-Design Study of the ARRA Grants 

 68 Table VI.1. Overview of Research Questions and Considerations for the Evaluation Type of Outcome Examined Research Question Example Outcomes Considera

Individual-level What was the impact of access to ARRA grant funding on the outcomes of participants? An individual’s probability of being employed, quarterly earnings (across industries or within certain target industries) Community-level What was the impact of ARRA grant funding on the macroeconomic conditions in communities that received them? Number of jobs created in target industries in the community, unemployment rate in the community An underst

counterfacregardless design or agreat amoutraining frothrough ARany potentcomparisocommunitireceive funof successfcontrol/coindividuals receiving twhich granmay limit wfrom compgrant partinonparticip

Note: Although DOL might want to consider conducting an evaluation of implementation or operational i

the ARRA grants, we do not include that type of approach in this table, nor do we discuss the issue

type of evaluation in detail in this paper. DOL has expressed interest primarily in rigorous quantitatan evaluation of the grants. Even if implementation issues are not the primary focus of an evimplementation study could be an important component of an evaluation. 
Pre-Design Study of the ARRA Grants 

69 Table VI.2. Overview of Potential Methodologies to Evaluate the ARRA Grants Approach Summary of Approach Strengths Weaknesses 

Individual-Level Outcomes Random assignment of all individuals at all grantees At all winning grantees, random assignment uses a statistical process to randomly assign some proportion of individuals seeking services to a treatment group and the remainder to a control group, which may not receive grant services. It compares outcomes of those in the treatment group with those in the control group. Random assignment ensures that there will be no systematic differences between the average characteristics of the treatment and control groups, so any detected impacts can be ascribed to the treatment. This approach has external validity and the most power to detect impacts of any design considered. If control group assignment rates are high enough, it might support impacts for specific sets of grantees with a high degree of statistical precision. Such a wide-scale data collection effort is unlikely to be feasible due to cost considerations. Compared with other designs, it takes a long time to implement, and it may be difficult to implement after grantees have begun serving participants. These timing issues might influence either the feasibility of the approach or the size of the sample that can be included in the study. Grantees may not be comfortable withholding services for individuals assigned to the control group. 

Random selection of grantees, then random assignment of individuals at the selected grantees First, this approach uses a statistical process to randomly select a subset of all winning grantees for study participation; this is to ensure that study sites are representative of all grant recipients. Then, it uses a statistical process to randomly assign some proportion of individuals seeking services to a treatment group and the remainder to a control group, which may not receive grant services. It compares outcomes of those in the treatment group to those in the control group. Random assignment ensures that there will be no systematic differences between the average characteristics of the treatment and control groups, so any detected impacts can be ascribed to the treatment. Focusing on a subsample of grantees can reduce the heavy cost burden associated with random assignment of sample members at all grantees. If the number of grantees sampled is large enough, then stratification on dimensions of interest prior to grantee selection might allow for informative grantee-level subgroup analyses. It has external validity. Compared with other designs, it take

a long time to implement and it may be difficult to implement after grantees have begun serving participants. These timing issues might influence either the feasibility of the approach or the size of the sample that can be included in the study. Grantees may not be comfortable withholding services for individuals assigned to the control group. This approach leads to a loss of statistical power due to the sampling of grantees, relative to random assignment at all grantees. 
Pre-Design Study of the ARRA Grants 

Table VI.2 (continued) 70 Approach Summary of Approach Strengths Weaknesses Random assignment of individuals at a nonrandom subset of grantees This approach selects a nonrandom subset of grantees on which to focus, perhaps because of a policy interest in the target groups, service plans, geographic locations, or other characteristics of the grantees. The approach then uses a statistical process to randomly assign some proportion of individuals seeking services to a treatment group and the remainder to a control group, which may not receive grant services. It compares outcomes of those in the treatment group with those in the control group. Random assignment ensures that there will be no systematic differences between the average characteristics of the treatment and control groups, so any detected impacts can be ascribed to the treatment. It has more statistical power than if randomly subsampling sites first. Focusing on a subsample of grantees can reduce the heavy cost burden associated with random assignment of sample members at all grantees and enables the study to use rigorous methods to estimate program impacts, while still focusing on particular grantees of special interest to policymakers. Compared with other designs, it take

a long time to implement, and it may be difficult to implement after grantees have begun serving participants. As with the other random assignment designs, these timing issues might influence either the feasibility of the approach or the size of the sample that can be included in the study. However, compared with including all or a random subset of grantees, a purposeful selection of grantees might slightly mitigate these problems. Grantees may not be comfortable withholding services for individuals assigned to the control group. The approach results in a small loss in statistical power relative to random assignment at all granteesResults pertain only to those grantees in the subset under study; there is limited external validity. 

Regression discontinuity (RD) with individuals in grantees that applied for funding (RD at the individual level) This approach assumes that at least some grantees used an eligibility cutoff score (such as income) to determine which individuals were eligible to access services. Individuals with income on one side of a preset cutoff value are assigned to the comparison group; those with scores on the other side of the cutoff are assigned to the treatment group. To determine treatment effects, outcomes of comparison group members who are near the cutoff are compared with outcomes of treatment group members who are near the cutoff. There is minimal interference with normal program activity. This approach can be implemented after programming has begun if adequate information about the sample members and their scores is known. There is less selection bias than in other nonexperimental designs because the selection rule for receiving services is fully known. This approach requires larger sample sizes to achieve impact estimates wit

the same level of statistical power as in a random assignment study (approximately three to four times). There is limited external validity, with results applicable only to grant applicants near the cutoff score for which grant applicants receive funding. Cannot be used if eligibility cutoff is same as that used for other social programs. 
Pre-Design Study of the ARRA Grants 

Table VI.2 (continued) 71 Approach Summary of Approach Strengths Weaknesses RD with individuals in grantees that applied for funding (RD at the grantee level) This approach is based on the availability of grantee applicant scores based on a prespecified scoring process that was adhered to (scores must be continuous). Grantees with scores above a preset cutoff value are assigned to the treatment group; those with scores below the cutoff are assigned to the comparison group. Outcomes of individuals at grantees that just made the cutoff are compared with outcomes of individuals in grantee communities that just missed the cutoff to determine treatment effects. There is minimal interference with normal program activity. This approach can be implemented after programming has begun if adequate information about the grant applicants and their scores is known. There is less selection bias than in other nonexperimental designs because the selection rule for grant receipt is fully known. It might be challenging to access information on grant applicants that did not win grants, as well on individuals in their communities. It may be difficult to determine which individuals form the appropriate comparison at the “losing grantee sites”. This approach requires larger sample sizes to achieve impact estimates with the same level of statistical power as in a random assignment study (approximately three to four times). There is limited external validity. Comparison group using other individuals in grantee sites This approach uses a statistical procedure and baseline variables to match individuals who received the treatment (in this case, received training financed by the grant) to individuals who had a similar propensity to receive the treatment but did not do so. Average outcomes of those who received the treatment are compared with those of the matched comparison group. There is minimal interference with normal program activity. This approach takes into account a wide range of observable characteristics to minimize the bias associated with impact estimates. Because of the diversity of the target populations served by grantees, finding any credible comparison group would be challenging. Extensive background characteristics

including a baseline measure of the outcome variable, are needed in ordefor the match to be credible. 

Interrupted time series This approach compares the average outcomes of individuals at winning grantee sites before the treatment (that is, winning the grant) with their outcomes after the treatment. Because the outcomes of individuals are recorded over time, the pre- intervention period acts implicitly as a comparison group for the post- intervention outcomes. This approach requires detailed baseline information data. Although not required, it also benefits from extensive historical information. It does not rule out other secular trends that may be causing changes in outcomes over time. 
Pre-Design Study of the ARRA Grants 

Table VI.2 (continued) 72 Approach Summary of Approach Strengths Weaknesses 

Community-Level Outcomes RD with grantees that applied for funding (RD at the grantee level) This approach is based on the availability of grantee applicant scores based on a prespecified scoring process that was adhered to (scores must be continuous). Grantees with scores above a preset cutoff value are assigned to the treatment group; those with scores below the cutoff are assigned to the comparison group. Aggregate outcomes in communities at grantees that just made the cutoff are compared with those in communities at grant applicants that just missed the cutoff to determine treatment effects. There is minimal interference with normal program activity. This approach can be implemented after programming has begun if adequate information about the grant applicants and their scores is known. It may be easier to collect outcome data at the grantee or community level from established data sources than at the individual level. There is less selection bias than in other nonexperimental designs because the selection rule for grant receipt is fully known. It might be hard to access informatio

on applicants that did not win grants. Depending on the variation in the chosen outcomes, there might be considerable statistical imprecision due to the use of a small number of grant applicants (compared with individual-level data analyses). Outcome measures must be chosen carefully to ensure there is a good relationship between grantees’ expected participant-level outcomes and the community-level measures chosen for analysis. There is limited external validity. 

Comparison group using other communities This approach uses a difference-in- differences approach to compare the changes in aggregate outcomes in winning grantees with the changes in those outcomes in a comparison community that is similar. Statistical matching approaches can be used to identify similar comparison communities. There is minimal interference with normal program activity. This approach can be implemented after programming has begun. It accounts for some secular trends in outcomes that may be occurring at the same time as the intervention. It might be hard to define and collect data for a suitable comparison community for each grant type. This approach requires extensive data on baseline characteristics of grantees and potential comparison communities for the period preceding the intervention. Outcome measures must be chosen carefully to ensure there is a good relationship between grantees’ expected participant-level outcomes and the community-level measures chosen for analysis. Notes: The information in this table summarizes different types of methodological approaches that might be used for evaluating th

that are part of the pre-design study. The first column names the approach; the second column provides a brief description 

three and four list the main strengths and weaknesses of each approach. The final column presents our assessment of the l

each approach. These assessments are based on commonly accepted standards for the rigor of research designs, such as Department of Education’s Institute for Education Sciences’ What Works Clearinghouse. Individual-level outcomes are those measured and examined at the level of individuals receiving training financed by the 

outcomes are employment status and quarterly earnings. Community-level outcomes are those measured and examined community, rather than the individuals within it; an example is the number of new jobs created in target industries. RD = regression discontinuity. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

73 

and the research questions that it intends to answer, however, the evaluation will need to take into 

account the unique environments in which the ARRA grants will likely operate, including both high 

rates of unemployment and other sources of training in many communities. 

Random assignment is the most analytically rigorous of the possible methodologies, 

but it also is more likely than other approaches to interfere with normal program activity 

and to take longer to implement. A random assignment evaluation could provide the most 

convincing estimate of the individual-level impacts of access to services funded by the ARRA grants. 

However, if DOL would like to proceed with a random assignment approach, it will be important to 

ensure that grantees are willing to cooperate fully with the evaluation efforts. Furthermore, 

issuesrelated to the potential timing of when grantees provide services and when random assignment 

could begin will influence (1) the portion of grant participants that could be included in an 

evaluation and (2) whether or not random assignment is feasible for the two-year ETP and Pathways 

Out of Poverty grants. 

With a random assignment approach, either random sampling or purposive selection of 

grantees might be desirable if cost considerations make it infeasible or undesirable to 

include all grantees in an evaluation. The most rigorous random assignment approach would 

include all grantees, but doing so is likely to be very expensive because of the number and 

geographic diversity of the grantees. Under certain assumptions, approaches that include in the 

study only a subset of grantees might be able to provide minimum detectible impacts (MDIs) that 

are in the same range as MDIs in other studies conducted for DOL. However, because we made 

many simplifying assumptions in our MDI calculations so that we could consider a range of 

different approaches, we caution that further investigation would be warranted to conclude 

definitively that a specific approach could yield acceptable MDIs. 

If random assignment is not feasible or desirable, alternative methodologies could be 

considered. Excluding random assignment, a regression discontinuity (RD) approach provides the 

most analytical rigor. For this approach to be appropriate, there must be a scoring variable that is 

(or was) used to determine whether or not individuals or groups have access to grant-funded 

services. This approach could be conducted at the individual level, in which outcomes of individuals 

who are determined eligible for grant services based on a score measure are compared with 

outcomes of individuals who expressed interest in receiving grant services but who, based on the 

score, are determined ineligible for services. An RD design also could be conducted at the grantee 

level, in which outcomes for either individuals or communities affiliated with successful grant 

applicants are compared with individuals or communities affiliated with unsuccessful grant 

applicants. Regardless of the details of the RD approach, however, important challenges that would 

need to be addressed during a design phase of an evaluation using an RD approach would include 

the need to ensure that (1) the comparison group is defined sensibly, especially given the variability 

in grantees’ target groups; (2) certain technical requirements about how the scoring process is (or 

was) conducted are met; and (3) the necessary data can be made available for the evaluation. 

Other types of methodologies, such as those using individual- or community-level 

comparison groups or those that do not use a comparison group, could be used for an 

evaluation but are less likely to provide convincing impact estimates. An important limitation 

of comparison group approaches is that the evaluation would be unable to state definitively that 

differences in outcomes between the group that has access to grant-funded services and the group 

that does not are not caused by unobservable differences between the groups. For the design 

comparing pre- and post-treatment outcomes without a comparison group, the evaluation would be 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

74 

unable to state definitively that changes in outcomes over time were caused by the intervention, and 

not by other factors occurring at the same time. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

75 

REFERENCES 

Bellotti, Jeanne, Andrew Clarkwest, Ronald D’Amico, Kate Dunham, Kenneth N. Fortson, Sheena 

M. McConnell, Karen E. Needels, Linda C. Rosenberg, Peter Z. Schochet, and Andrew 

Wiegand. “The Workforce Investment Act Evaluation: Design Report.” Report submitted to 

the U.S. Department of Labor, Employment and Training Administration. Princeton, NJ: 

Mathematica Policy Research, January 16, 2009. 

Benus, Jacob, Sheena M. McConnell, Jeanne M. Bellotti, Theodore Shen, Kenneth N. Fortson, and 

Daver Kahvecioglu. “Growing America Through Entrepreneurship: Findings from the 

Evaluation of Project GATE.” Columbia, MD: IMPAQ International, LLC, May 2008. 

Cook, T. D., and V. C. Wong. “Empirical Tests of the Validity of the Regression Discontinuity 

Design.” Annales d’Economie et de Statistique, forthcoming. 

Department of Labor. “American Recovery and Reinvestment Act of 2009: Notice of Availability 

of Funds and Solicitation for Grant Applications for the Health Care Sector and Other High 

Growth and Emerging Industries.” Federal Register, vol. 74, no. 139, July 22, 2009a, pp. 

36255–36269. Retrieved from http://www.doleta.gov/grants/pdf/SGA-DFA-PY-09-01.pdf 

on October 9, 2009. 

Department of Labor. “American Recovery and Reinvestment Act of 2009: Notice of Availability 

of Funds and Solicitation for Grant Applications for Energy Training Partnership Grants.” 

Federal Register, vol. 74, no. 120, July 22, 2009b, pp. 30152–30165. Retrieved from 

http://www.doleta.gov/grants/pdf/SGA-DFA-PY-08-18.pdf on October 9, 2009. 

Department of Labor. “American Recovery and Reinvestment Act of 2009: Solicitation for Grant 

Applications (SGA) Amendment One—Energy Training Partnership Grants.” Federal 

Register, vol. 74, no. 140, July 23, 2009c, pp. 36531–36533. Retrieved from Amend-01-SGADFA-PY-08-18.pdf on December 30, 2009. 

Department of Labor. “American Recovery and Reinvestment Act of 2009: Notice of Availability 

of Funds and Solicitation for Grant Applications for Pathways Out of Poverty.” Federal 

Register, vol. 74, no. 120, June 24, 2009d, pp. 30138–30152. Retrieved from 

http://www.doleta.gov/grants/pdf/SGA-DFA-PY-08-19.pdf on October 9, 2009. 

Department of Labor. “American Recovery and Reinvestment Act of 2009: Solicitation for Grant 

Applications (SGA) Amendment One; Pathways Out of Poverty.” Federal Register, vol. 74, no. 

139, July 22, 2009e, pp. 36270–36271. Retrieved from 

http://www.doleta.gov/grants/pdf/SGA-DFA-PY-08-19-Amend01.pdf on October 14, 2009. 

Department of Labor. “American Recovery and Reinvestment Act of 2009: Solicitation for Grant 

Applications (SGA) Amendment Two; Pathways Out of Poverty.” Federal Register, vol. 74, 

no. 156, August 14, 2009f, p. 41171. Retrieved from http://www.doleta.gov/grants/pdf/ 

SGA-DFA-PY-08-19-Amend02.pdf on October 14, 2009. 
Pre-Design Study of the ARRA Grants Mathematica Policy Research 

76 

Department of Labor. “American Recovery and Reinvestment Act of 2009: Notice of Availability 

of Funds and Solicitation for Grant Applications for State Energy Sector Partnerships (SESP) 

and Training Grants.” Federal Register, vol. 74, no. 120, June 24, 2009g, pp. 30114–30128. 

Retrieved from http://www.doleta.gov/grants/pdf/SGA-DFA-PY-08-20.pdf on October 9, 

2009. 

Department of Labor. “American Recovery and Reinvestment Act of 2009: Solicitation for Grant 

Applications (SGA) Amendment One; State Energy Sector Partnerships (SESP) and Training 

Grants.” Federal Register, vol. 74, no. 140, July 23, 2009h, pp. 36528–36531. Retrieved from 

http://www.doleta.gov/grants/pdf/ Amend-01-SGA-DFA-PY-08-20.pdf on October 9, 2009. 

Goldberger, A. S.. “Selection Bias in Evaluating Treatment Effects: Some Formal Illustrations.” 

Madison, WI: Institute for Research on Poverty, 1972. 

McConnell, Sheena M., Irma L. Perez-Johnson, Jeanne M. Bellotti, Nuria Rodriguez-Planas, and 

Walter Corson. “Project GATE: The Evaluation Design.” Report submitted to the U.S. 

Department of Labor, Employment and Training Administration. Washington, DC: 

Mathematica Policy Research, March 3, 2004. 

Rosenbaum, P. R., and D. B. Rubin. “The Central Role of the Propensity Score in Observational 

Studies for Causal Effects.” Biometrika, vol. 70, no. 1, 1983, pp. 41–55. 

Schochet, Peter Z. “Evaluation of the Trade Adjustment Assistance Program: Design Report.” 

Report submitted to the U.S. Department of Labor, Employment and Training Administration. 

Oakland, CA, and Princeton NJ: Social Policy Research Associates and Mathematica Policy 

Research, January 2005. 

Schochet, Peter Z. “Statistical Power for Regression Discontinuity Designs in Education 

Evaluations.” Journal of Educational and Behavioral Statistics, vol. 34, no. 2, 2009, pp. 238–

266. 

U.S. Congress. “American Recovery and Reinvestment Act of 2009.” Pub. Law No. 111-5, 

February 17, 2009. Retrieved from U.S. Government Printing Office, http://frwebgate. 

access.gpo.gov/cgi-bin/getdoc.cgi?dbname=111_cong_bills&docid=f:h1enr.txt.pdf on January 

29, 2010. 

U.S. Department of Labor, Bureau of Labor Statistics. “The Employment Situation—December 

2009.” Washington, DC: DOL, January 2010. Retrieved from 

http://www.bls.gov/news.release/pdf/empsit.pdf on January 29, 2010. 
APPENDIX A 

KEY FEATURES OF AWARDED GRANTS 

Pre-Design Study of the ARRA Grants 

79 Table A.1. Energy Training Partnership Grant Recipients Grant Recipient Location(s) Served Award Amount Populations Served Serving an Auto Restructuring Area? National or Local Grant Type Primary Foc

1. Northwest Energy Efficiency Council 5 counties and 4 cities in Washington State $3,876,171 Older youth, dislocated workers, incumbent workers, veterans, women, and individuals with disabilities No Local 473 trained and place

occupations in industefficient building consand energy efficient a

2. United Auto Workers Labor Employment and Training Corporation (UAW-LETC) 2 counties in Missouri $3,200,000 Veterans, ex- offenders, individuals with disabilities, and women Yes Local Train and place appro

and incumbent workecommercial energy fieauto technicians, electechnicians, electric mand commercial energ

3. H-CAP Inc. 2 counties in California, the District of Columbia, 2 counties in Maryland, 5 counties in New York, and 1 county in Washington State $4,637,551 Immigrants and minorities Yes National Approximately 3,000 j

environmental service training and approximworkers will receive ce

4. Utility Workers Union of America (UWUA), AFL-CIO 3 counties in California, 3 counties in Massachusetts, and 6 counties in New Jersey $4,993,922 Women, minorities, older youth, and incumbent workers Yes National Approximately 672 pa

and placed in green ain the utility industry. apprenticeship and reprograms. 5. International Transportation Learning Center Statewide in New Jersey, New York, and Utah; Columbus, Ohio $5,000,000 Minorities No National Train 3,640 new and i

public transportation apprenticeships. 
Pre-Design Study of the ARRA Grants 

Table A.1 (continued) 

 80 Grant Recipient Location(s) Served Award Amount Populations Served Serving an Auto Restructuring Area? National or Local Grant Type Primary Foc

6. California State Labor Management Cooperation Committee for the International Brotherhood of Electrical Workers and the National Electrical Contractors Association (LMCC-IBEW- NECA) Statewide in California $5,000,000 Unemployed and underemployed electricians Yes Local Training to 2,292 jour7. Central Vermont Community Action Council Inc. All 14 counties and 256 minor civil divisions in Vermont $4,846,195 Unemployed, underemployed, veterans, high school dropouts, women, and individuals with disabilities No Local Provide intensive case 

to 398 individuals and others, focusing on enrenewable energy indrecycling, and waste r

8. E.C.I.A. Business Growth Inc. 27 counties in Iowa and portions of Minnesota and Wisconsin $2,060,250 Unemployed and dislocated workers, high school graduates, veterans, disconnected youth, women, and minorities Yes Local Will serve 392 particip

training, through 3 caelectrical power, energand energy efficient b

9. Institute for Career Development (ICD) Inc. 7 counties in Indiana, and 1 county each in New York, Ohio, and Pennsylvania $4,658,983 Dislocated steelworkers Yes National Will serve 2,000 partic

that 1,200 will compleas wind farm field tecinstallers, or geotherm1,200 will receive a crenergy generation sucand home retrofitting
Pre-Design Study of the ARRA Grants 

Table A.1 (continued) 

 81 Grant Recipient Location(s) Served Award Amount Populations Served Serving an Auto Restructuring Area? National or Local Grant Type Primary Foc

10. National Ironworkers and Employers Apprenticeship Training and Journeyman Upgrading Fund 20 counties in the 5 states affected by auto restructuring: areas within 400 miles of La Palma, Calif.; Joliet, Ill.; West Seneca, N.Y.; Arlington, Texas; and Salt Lake City, Utah $1,943,931 Dislocated workers Yes National At least 510 ironwork

increasing skills in the sector. 11. Blue Green Alliance 7-county metro area around the Twin Cities, 8 counties in northeastern Minnesota, and 38 counties in southern Minnesota $5,000,000 Dislocated workers, women, minorities, veterans, incumbent workers, and unemployed steelworkers Yes Local Will enroll approximattraining in green man

12. Oregon Manufacturing Extension Partnership 6 counties in Oregon and 3 counties in Washington $5,000,000 Unemployed and dislocated workers and employed workers at local renewable energy industries No Local It is expected that app

participants will compreceive a credential in power and biofuels fie

13. SER Metro-Detroit, Jobs for Progress Inc. Wayne County and Detroit, Michigan $4,298,673 Women, minorities, and veterans Yes Local Approximately 340 un

trained for green jobs opportunities in alternin building constructioweatherization, and e

14. The Providence Plan Providence, Rhode Island $3,720,000 Ex-offenders, minorities, and dislocated workers No Local Approximately 1,600 

residents will completapprenticeship, green journey-level skill upgbuilding construction generated electrical pbe placed in green en
Pre-Design Study of the ARRA Grants 

Table A.1 (continued) 

 82 Grant Recipient Location(s) Served Award Amount Populations Served Serving an Auto Restructuring Area? National or Local Grant Type Primary Foc

15. Montana Electrical Joint Apprenticeship & Training Council (MEJATC) Statewide in Montana $5,000,000 Unemployed workers and incumbent workers No Local Approximately 2,450 pa

training and receive credapprentices, or journeymefficiency and renewable building construction an

16. Communications Workers of America (CWA) National Education and Training Trust 8 counties in Ohio $3,969,056 Dislocated workers and veterans Yes National Will provide 1,000 disloc

training opportunities in manufacturing skills for wrelated, energy storage, environments. 17. Heritage Health Foundation Upper Monongahela Valley region of Southwestern Pennsylvania $1,408,601 Underemployed and unemployed workers, veterans, women, and minorities Yes Local Will enroll 120 trainees f

deconstruction, building management, weatherizrestoration. Aims to havunsubsidized employme

18. Thomas Shortman Training Scholarship and Safety Fund All 5 boroughs of New York City, New York $2,802,269 Incumbent workers and minorities No Local Will train 2,000 building 

make large apartment bwill train 30 commercial and 170 building analysperformance of large bu

19. Memphis Bioworks Foundation 21 counties in Tennessee and 5 counties in Arkansas $2,931,103 Dislocated workers Yes Local Will train 25 participants 

energy training program receive customized traintechniques. 20. International Training Institute for the Sheet Metal and Air Conditioning Industry 5 counties in Michigan; 4 counties in Ohio; St. Louis, Missouri; East St. Louis, Illinois; 5 cities in California; 3 counties in New Mexico; and 2 counties in Texas $4,995,188 Unemployed and underemployed individuals, veterans, minorities, and women Yes National Expects that 1,200 unem

metal workers will succetraining and transition inefficient occupations for construction, retrofitting, 
Pre-Design Study of the ARRA Grants 

Table A.1 (continued) 

 83 Grant Recipient Location(s) Served Award Amount Populations Served Serving an Auto Restructuring Area? National or Local Grant Type Primary Foc

21. Labor's Community Action Inc. Statewide in Colorado, but primarily serving Denver and Colorado Springs and, through online training, rural residents $3,604,162 Dislocated workers, incumbent workers, and newly trained workers No Local Expects to enroll 1,913 

unemployed constructiopercent of them into unsenergy efficient buildingand energy efficiency asTraining will be through registered apprenticeshapprenticeship or basic 

22. Austin Electrical Joint Apprenticeship Training Committee (JATC) Statewide for Arizona; Oklahoma; Kansas; New Mexico; and Texas $4,842,424 Unemployed workers and incumbent workers Yes Local Approximately 1,000 ind

solar-specific and smartExpects to place or retaemployment. 23. Community Housing Partners Corporation 15 counties and 5 cities in Virginia $3,865,480 Unemployed and dislocated workers Yes Local Will serve approximately 

that 280 participants wiltraining activities and obefficient building, constrefficiency assessment. 24. Broward County Minority Builders Coalition Broward County, Florida $3,280,656 Unemployed workers, veterans, women, ex- offenders, and minorities No Local Will recruit 1,000 particip

complete training requirein five occupations: solaand installer; solar PV (pinstaller (skilled); solar P

level); weatherization te(Leadership in Energy aGreen Associate. 25. Ohio Electrical Labor Management Cooperative Committee 44 counties in Ohio $4,826,073 Incumbent and dislocated workers, including veterans Yes Local Using a registered appre

training for 1,288 dislocawho will earn credentials on energy-efficient buildin combination with the sector. Source: DOL’s announcement of grant awards, retrieved from http://www.doleta.gov/ETA_News_Relea

http://www.doleta.gov/pdf/ETP_SGA_Award_Summaries_120409.pdf and accessed on January 27, 2010. AFL-CIO = American Federation of labor-Congress of Industrial Organizations; HVAC = heating, ventilation, and air conditioning.
Pre-Design Study of the ARRA Grants 

84 Table A.2. Pathways Out of Poverty Grant Recipients Grant Recipient Location Served Award Amount Participants Served Type of Grant Primary Focus 

1. Alternative Opportunities Inc. 4 PUMAs in St. Louis, MO $2,308,200 High school dropouts, unemployed individuals, ex- offenders, and veterans Local 200 participants enrolled to 

through an intensive five-mabout building performanceweatherization/retrofitting, 

installation, and solar photo

2. Better Family Life Inc. (BFL) 3 PUMAs in St. Louis, MO $3,305,493 Unemployed individuals, high school dropouts, and individuals with criminal records Local 900 participants will be trai

weatherization technicians/installers, LEED green assoccollection technicians. 3. Boley Centers Inc. 1 PUMA in St. Petersburg, FL $2,300,678 Disadvantaged and unemployed urban youth Local 150 participants will take p

and certification in electrici

4. Citrus Levy Marion Regional Workforce Development Board Inc. 1 PUMA in Ocala, FL $2,985,175 Unemployed workers, low- income adults, high school dropouts, and individuals with a criminal history Local 665 participants will be trai

employment in green jobs senergy auditing, weatherizaproducts installation, organxeriscape growing and land

5. City of Minneapolis 5 PUMAs in Minneapolis and St. Paul, MN $4,000,000 Individuals living in poverty, veterans, and unemployed young adults who do not have a high school diploma Local 500 participants served in i

efficient building, constructefficient building maintenanmaterials use, recycling, ansustainable manufacturing. 6. CNY Works Inc. 1 PUMA in Syracuse, NY $3,715,931 Low-income individuals, ex- offenders, disadvantaged young-adults, and displaced workers Local 750 participants will receive 

options in burgeoning greeenergy efficiency and bio-fu

7. Community College of Philadelphia 2 PUMAs in Philadelphia, PA $3,184,428 Unemployed workers, ex- offenders, and veterans Local 250 participants will be serv

will implement two entry-levprograms in green manufacconstruction/weatherization

8. Consortium for Worker Education 2 PUMAs in Bronx, NY $4,000,000 Individuals with limited English proficiency, veterans and eligible spouses, persons with criminal records, disconnected youth, and women Local 425 participants will take a 

sustainable mechanical and and employment fundamencan choose from more than energy efficiency and renew
Pre-Design Study of the ARRA Grants 

Table A.2 (continued) 

 85 Grant Recipient Location Served Award Amount Participants Served Type of Grant Primary Focus 

9. East Harlem Employment Services Inc. doing business as STRIVE 4 PUMAs in New York, NY; 3 PUMAs in Philadelphia, PA; 1 PUMA in Hartford, CT; 1 PUMA in Benton, MI; 1 PUMA in Flint, MI; and 2 PUMAs in Baltimore, MD $4,728,419 Unemployed individuals, high school dropouts, and individuals with a criminal record National Will serve 1,819 participant

green construction training 

10. Eastern Maine Development Corp. 1 PUMA in Piscataquis County and Penobscot County, ME $2,109,088 Disadvantaged adult job seekers, dislocated workers, returning offenders, public assistance recipients, high school dropouts, and veterans Local Will provide 105 workers w

construction opportunities on housing retrofits and en

11. Florida State College at Jacksonville 1 PUMA in Duval County, FL $2,229,642 Unemployed individuals, high school dropouts, and individuals with a criminal record Local Will serve 390 participants t

the energy efficient building renewable electric power, aassessment industries. 12. Goodwill Industries International (GII) 3 PUMAs in Grand Rapids, MI; 1 PUMA each in Atlanta, GA; Austin, TX; Charlotte, NC; Phoenix, AZ; and Washington, DC $7,303,634 People with disabilities, chronically unemployed individuals, ex-offenders, older workers, homeless individuals, and high school dropouts National Will serve 1,300 participant

careers in the energy efficieand retrofit and renewable 

13. Grand Rapids Community College 1 PUMA in Grand Rapids, MI $4,000,000 Unemployed workers, high school dropouts, and individuals with criminal records Local Will serve 1,080 participant

efficient building constructienergy, energy efficiency, gmanagement, and advanced industries. 14. It's My Community Initiative 1 PUMA in Oklahoma City, OK $4,000,000 Underemployed individuals and ex-offenders Local Will serve 236 participants, 

through industry-developed technician certification progtraining programs. 
Pre-Design Study of the ARRA Grants 

Table A.2 (continued) 

 86 Grant Recipient Location Served Award Amount Participants Served Type of Grant Primary Focus 

15. Jobs for the Future Inc. (JFF) 3 PUMAs in Chicago, IL; 3 PUMAs in Detroit, MI; 10 PUMAs in Los Angeles, CA; 1 PUMA in Milwaukee, WI; 2 PUMAs in Philadelphia, PA $7,997,936 Unemployed and disadvantaged individuals National Will serve 1,130 participant

efficient building, constructrenewable electric power in

16. Lehigh Valley Workforce Investment Board Inc. 1 PUMA in Allentown, PA $4,000,000 At-risk youth, veterans and eligible spouses, and underemployed and unemployed individuals Local Will serve 200 participants, 

energy efficient building, coindustries; renewable electrefficiency assessment indus

17. Los Angeles Community College District (LACCD) 3 PUMAs in the Los Angeles, CA communities of Watts, Willowbrook, and Florence- Graham $4,000,000 Dislocated, unemployed, underemployed, low-income workers, and veterans Local Will train 925 participants f

occupations: green plumbinconstruction manager/buildcontractor, solar PV or solaenergy efficiency auditor an

18. MDC Inc. 4 PUMAs in Charlotte, NC; 1 PUMA in North Charleston, SC; 1 PUMA in Orangeburg, Calhoun, and Bamberg Counties, SC; 1 PUMA in Wise and Dickenson Counties, VA; 1 PUMA in Scott County, VA $3,780,816 Low-wage workers and unemployed individuals National Will train 700 individuals w

weatherization and alternat

19. Mi Casa Resource Center for Women Inc. 1 PUMA in Denver, CO $3,633,195 Unemployed individuals, high school dropouts, individuals with a criminal record, women, and minorities Local Will serve 500 participants w

following industries: energy construction & retrofits, rendeconstruction & materials assessment (residential, com
Pre-Design Study of the ARRA Grants 

Table A.2 (continued) 

 87 Grant Recipient Location Served Award Amount Participants Served Type of Grant Primary Focus 

20. Mott Community College (MCC) 2 PUMAs in Flint, MI and adjoining suburbs $3,662,403 Low-income individuals Local Will allow 200 participants t

training course and a 12-co

covering the basic skills and enter employment in the grwell as the Energy Conserva(ECAR) program and the RoApprenticeship Readiness (R

21. Moultrie Technical College 1 PUMA in Tift County, GA $3,753,579 Individuals on probation, high school dropouts, residents with disadvantaged backgrounds, and displaced workers Local Will provide education and 

participants for employmenenergy industry: biofuels, eassessment, and renewable wind, biomass, and geother

22. National Association of Regional Councils (NARC) 1 PUMA each in Apache Junction, AZ; Bisbee, AZ; Odessa, TX; and Dayton, OH; 2 PUMAs in Midland, TX $7,994,999 Limited English proficiency individuals, Native Americans, and ex-offenders National Will train and place approxi

the building performance; econstruction, and retrofit; rand deconstruction and ma

23. National Council of La Raza 2 PUMAs each in San Jose, CA and Chicago, IL; 1 PUMA in San Diego, CA $3,063,839 Low-income and unemployed individuals, and individuals with limited English proficiency National Will train and place approxi

unsubsidized employment wand clean energy industries

24. Northern Rural Training and Employment Consortium (NoRTEC) 1 PUMA each in Butte, CA; Del Norte, Lassen, Modoc, and Siskiyou, CA; Shasta County, CA; and Tehama and Trinity, CA $4,000,000 High school dropouts, at-risk youth, welfare recipients, individuals with a criminal record, unemployed and dislocated workers, and veterans Local For 554 participants, will pr

including the California CleTraining Program and GreeApprenticeship Program. 25. Opportunities Industrializa- tion Centers of America Inc. 1 PUMA each in Asheville, NC and Phoenix, AZ; 2 PUMAs in Broward County, FL $4,900,000 Unemployed individuals, high school dropouts, and individuals with criminal records National Will train 1,350 participants 

including weatherization teadvanced and entry-level soLEED green associates; and rendering technicians. 
Pre-Design Study of the ARRA Grants 

Table A.2 (continued) 

 88 Grant Recipient Location Served Award Amount Participants Served Type of Grant Primary Focus 

26. PathStone Corp. 1 PUMA each in Rochester, NY; Scranton, PA; Juana Diaz, Santa Isabel and Villalba, PR; and Arroyo, Coamo, Guayama and Salinas, PR $8,000,000 Unemployed individuals, high school dropouts, and individuals with criminal records National Will serve 1,176 participants. Will 

retrofit, welding, and diesel mechaand hazardous materials (PR). 27. Private Industry Council of Westmoreland/F ayette Inc. 1 PUMA in Fayette County, PA $2,732,719 Unemployed individuals, high school dropouts, and individuals with criminal records Local Will serve 250 participants fefficient building industry, industry, and energy efficie

28. Providence Economic Development Partnership 1 PUMA in Providence, RI $2,489,111 Ex-offenders and low-literacy individuals Local Will serve 300 participants w

certifications and employmeenergy efficiency, and green 

29. Roca Inc. 1 PUMA in Chelsea and Revere, MA $2,398,778 High-risk youth Local Will serve 225 individuals. 

program will integrate acadoccupational training relatebuilding construction and refficiency assessment; and 

30. SER – Jobs for Progress of the Texas Gulf Coast Inc. 8 PUMAs in Houston, TX $3,122,554 High school dropouts, ex- offenders, unemployed individuals, and disadvantaged individuals Local 300 participants will receive 

one of three occupations: (1technician coupled with gre(2) solar panel installer; or (house diagnostics. 31. Southeast Community College Area 1 PUMA in Lincoln, NE $2,331,278 Unemployed individuals, veterans, high school dropouts, individuals with criminal records, refugees, and immigrants Local Will serve 400 eligible parti

positions in the energy effic

32. Southwest Housing Solutions Corp. (SWHS) 2 PUMAs in Southwest Detroit, MI $4,000,000 Unemployed individuals, high school dropouts, individuals with criminal records, and veterans Local 360 trainees will be placed 

in training programs in weatherization/deconstructand urban agriculture. 33. West Hills Community College District 3 PUMAs in communities in or near Fresno and Kings Counties, CA $3,000,000 Disadvantaged individuals Local Training efforts will result ifor 150 participants, in fieldwater management, sustainconstruction. 
Pre-Design Study of the ARRA Grants 

Table A.2 (continued) 

 89 Grant Recipient Location Served Award Amount Participants Served Type of Grant Primary Focus 

34. Western Iowa Tech Community College (WITCC) 1 PUMA in Woodbury County, IA $3,999,459 Dislocated workers, low- income adults, and disconnected youth Local Will train 300 workers into 

retrofitting, renewable enerproduction; and other energ

35. White Earth Band of Chippewa 1 PUMA in Mahnomen, Clearwater, and Becker counties, MN $3,086,817 Native American high school dropouts, unemployed individuals, and individuals with ,,a criminal records Local Will serve 240 participants f

industries: energy efficient and retrofit; energy efficiensustainable manufacturing. 36. Workforce Development of Seattle-King County 2 PUMAs in Southeast Seattle, WA $3,639,530 High school dropouts, unemployed adults, veterans, previously incarcerated youth and adults, and other disadvantaged individuals — with a specific focus on communities of color, individuals with limited English proficiency and individuals with disabilities Local Will connect 450 participan

introductory green courses for green construction, manweatherization industries oinitiatives related to construmanufacturing, and a green employment program. 37. The WorkPlace Inc. 1 PUMA in Bridgeport, CT $4,000,000 High school dropouts, individuals with criminal records, unemployed individuals, and people facing other significant disadvantages Local Will serve 700 participants a

overview of the core sectors then enable them to move iprograms. 38. Worksystems Inc. 2 PUMAs in East Multnomah County, OR $4,000,000 Native Americans, African Americans, Latinos, immigrants, veterans, individuals with criminal records, and homeless individuals Local Will serve 360 participants. 

certificates in green manufasystems, and weatherizatio

Source: DOL’s announcement of grant winners found at http://www.doleta.gov/ETA_News_Release

http://www.doleta.gov/pdf/Pathways_Poverty_grants.pdf and accessed on January 27, 2010. HVAC = heating, ventilating, and air conditioning; LEED = Leadership in Energy and Environmental Design; PUMA = public use microdata are
Pre-Design Study of the ARRA Grants 

90 Table A.3. State Energy Sector Partnership Grant Recipients Grant Recipient State Auto-Affected Counties Served Award Amount Participants Served Primary Fo

1. Alabama Department of Economic and Community Affairs (ADECA) Alabama Montgomery, Tuscaloosa, Lowndes, Butler, Madison, Limestone, and Marshall $6,000,000 Veterans, at-risk youth, ex-offenders, dislocated workers, incumbent workers, low-income workers, and disadvantaged individuals Will prepare individua

energy efficiency and industries and establRenewable Energy Seapproximately 1,442 

2. Alaska Department of Labor and Workforce Development Alaska None $3,600,000 Minorities, recipients of public assistance, veterans, people with disabilities, low-income individuals, unemployed and underemployed individuals, dislocated workers, out-of- school youth, and incumbent workers Will implement a sust

the supply of workers skills to support enertechnology and the gwind turbine, and bio700 workers will rece

3. Arizona Department of Economic Security Arizona None $5,000,000 Dislocated workers, underemployed workers, veterans, incumbent workers, at-risk youth, individuals with criminal records, and individuals with disabilities Approximately 1,502 

education and traininwill earn industry-recand energy-efficiency aided drafting sustaininstaller photo voltaic certificates, and greeconstruction certifica

4. Arkansas Workforce Investment Board/Department of Workforce Services Arkansas Cleburne, Desha, Greene, and Logan $4,866,479 High school dropouts, offenders, unemployed workers, and other disadvantaged, at-risk individuals Will create three ener

that will develop and programs to be used and 7 apprenticeship 2,800 participants foprograms. 5. State of California Employment Development Department California Alameda and Los Angeles $6,000,000 Dislocated workers, veterans, unemployed and underemployed workers, low-income youth and adults, new workforce entrants, individuals with disabilities, and individuals with criminal records Each of five regional courses that will coveprinciples, principles energy fundamentalswater heaters, and bu1, 000 participants wtraining, participants apprenticeship progr
Pre-Design Study of the ARRA Grants 

Table A.3 (continued) 

 91 Grant Recipient State Auto-Affected Counties Served Award Amount Participants Served Primary Fo

6. Colorado Department of Labor and Employment Colorado None $5,998,050 Older workers, dislocated workers, incumbent workers, veterans, women, individuals with disabilities, Temporary Assistance for Needy Families (TANF) recipients, dislocated workers, and ex- offenders Will prepare workers 

economy through sevsector entrepreneuriaapprenticeship partnegreen training curricutraining in energy effsafety, and solar enerparticipants; youth coyouth outreach campparticipants. 7. Connecticut Employment and Training Commission Connect- icut None $3,360, 000 Incumbent workers, unemployed workers, veterans, at-risk youth, ex- offenders, underrepresented ethnic minorities and recent immigrants Will serve 895 particienergy efficiency assemanufacturing, and e

8. Hawaii Department of Labor and Industrial Relations Hawaii None $6,000,000 Incumbent workers, low-income/low- skilled individuals, disadvantaged persons with limited English proficiency, people with disabilities, and veterans Approximately 1,391 

education and trainineither certificates or LEED, solar photovoltprocessing, and powe

9. Idaho Department of Labor Idaho None $5,991,184 Veterans, dislocated workers, low-skill adults and youth, and other targeted populations Will be used to create 

will upgrade existing secondary curricula; sarticulation; and provprograms to better pcareers in energy effienergy industries. Wparticipants. 10. Illinois Department of Commerce and Economic Opportunity Illinois Boone, Cook, and McLean $6,000,000 Low-skilled individuals, new labor market entrants, dislocated workers, incumbent workers, underemployed individuals, veterans, minority contractors, disadvantaged adults and youth, individuals with disabilities, and individuals with criminal records Approximately 1,310 and placed in energy Additionally, 1,694 pearn industry-recognbuilding analyst and 

11. Indiana Department of Workforce Development Indiana 45 countiesa $6,000,000 Incumbent and dislocated workers, unemployed workers, adults with barriers to employment, and out-of- school youth Will establish an Adva

Center to facilitate thgreen economy; will coordinate traditionatechnology programscompleters and 2,075 employment in green
Pre-Design Study of the ARRA Grants 

Table A.3 (continued) 

 92 Grant Recipient State Auto-Affected Counties Served Award Amount Participants Served Primary Fo

12. Iowa Workforce Development Iowa Cass, Chickasaw, Fremont, Howard, Iowa, Louisa, Plymouth, Poweshiek, Union, and Wright $5,997,000 Dislocated workers, unemployed individuals, and incumbent workers Will implement a stat

development plan witneeds, as identified btask force. 13. Kansas Department of Commerce Kansas Allen $5,999, 890 Dislocated workers, incumbent workers, veterans, older youth, former prisoners, and individuals seeking career pathways out of poverty Will recruit candidate

renewable energy operenewable energy machain, energy transmconstruction industrindividuals, with 1,311,053 completing tracertification. 14. Education and Workforce Development Cabinet Kentucky Cumberland, Fulton, Hopkins, Pulaski, and Trigg $4,740,457 Dislocated workers, unemployed individuals, out-of-school youth, and veterans Proposed trainings in

auditors/raters, Smarinstallations, and mapipefitters; and steamGreen Systems AwareApproximately 444 ptraining and be placeoccupations. 15. Maryland Department of Labor, Licensing and Regulation Maryland None $5,793,183 Incumbent and dislocated workers, veterans and reservists, low-wage workers, and individuals with criminal records Will improve manufacpractices, waste streato-green practices in develop and expand options by providing 

construction-related tand assist workers as environmental technoApproximately 1,585 and placed into energ

16. Commonwealth of Massachusetts, Executive Office of Labor and Workforce Development Massa- chusetts None $5,973,657 Unemployed, underemployed, and incumbent workers Will implement a train

following four key coindustry sectors for wpolicy is acting as a dtarget occupations wworkers who can be pthree years; (3) leverathrough state energy leverage the workforcserve 1,379 participa
Pre-Design Study of the ARRA Grants 

Table A.3 (continued) 

 93 Grant Recipient State Auto-Affected Counties Served Award Amount Participants Served Primary Fo

17. Michigan Department of Energy, Labor, & Economic Growth Michigan Midland, Saginaw, Genesee, Bay, Roscommon, Losco, Livingston, Monroe, Wayne, Clair, Macomb, and Oakland $5,819,999 Dislocated workers and incumbent workers Will implement the G

Initiative and provide energy storage, solar efficient construction 

18. Minnesota Department of Employment and Economic Development Minnesota Clearwater and Ramsey $6,000,000 Unemployed workers, incumbent workers, individuals with limited English proficiency, low-income individuals, individuals with disabilities, individuals with criminal records, high school dropouts, youth, dislocated workers, veterans, and minority populations Will serve 1,495 parti

industries: energy effconstruction, and retand biofuels. 19. Missouri Division of Workforce Development Missouri St. Louis, St. Charles, Lincoln, Clay, and Randolph $6,000,000 Veterans and active duty military personnel, incumbent workers, and unemployed workers Will provide 480 incuapprentices with enhveterans and/or qualcomplete a RenewablProgram; 150 participrenewable energy degcommunity colleges, minors or certificate degree-granting engi

20. Nebraska Department of Labor Nebraska Dawson $4,839,511 Workers affected by energy policy, incumbent workers, veterans, unemployed individuals, at-risk youth, and individuals with criminal records Will serve 950 partici

careers in wind, biofubuilding technologies 

21. Nevada Department of Employment, Training and Rehabilitation Nevada None $6,000,000 Dislocated workers, individuals with criminal records, individuals with disabilities, individuals who are homeless, veterans, former foster youth, young pregnant and single mothers, individuals receiving public assistance, and other low-income individuals Will serve 7,125 parti

careers in energy effiand other green indu
Pre-Design Study of the ARRA Grants 

Table A.3 (continued) 

 94 Grant Recipient State Auto-Affected Counties Served Award Amount Participants Served Primary Fo

22. New Jersey Department of Labor and Workforce Development New Jersey Middlesex and Union $6,000,000 Out-of school youth, adults with limited basic skills, individuals with criminal histories, and individuals with disabilities Has developed a dive

projects that will impmid-level career-path upgrades to incumbemaintain employmenpaths; open new posidisadvantaged populworkers and nontradoccupations; and ensfunded training progparticipants. 23. New Mexico Department of Workforce Solutions New Mexico None $5,999,989 Incumbent workers, unemployed workers, women, veterans, military spouses, and high school dropouts Establishes a training 

access to lifelong lealocal and regional labApproximately 400 pfor solar, wind, energconstruction and retremployment. 24. North Carolina Department of Commerce, Division of Workforce Development North Carolina Scotland $5,976,512 Unemployed workers, military spouses, veterans, at-risk youth, individuals with criminal records, farmers, and migrant and seasonal workers Will provide 1,137 paleads to industry-recoenergy efficient buildretrofitting industriesassessment. 25. State of Ohio Ohio 44 countiesa $6,000,000 Dislocated workers, veterans, women, minorities, and Appalachian residents Focusing on wind, so

industries, will serve and offer “stackable crearranged to offer bstudents the flexibilitrapidly changing eco

26. State of Oklahoma Oklahoma Oklahoma and Marshall $6,000,000 Unemployed workers and incumbent workers Will create a Center o

Innovation at Tulsa CNortheast Campus, to practical applications green landscaping. Wparticipants. 
Pre-Design Study of the ARRA Grants 

Table A.3 (continued) 

 95 Grant Recipient State Auto-Affected Counties Served Award Amount Participants Served Primary Fo

27. State of Oregon Oregon None $5,383,568 Unemployed workers and incumbent workers Will implement the G

Workforce project to will deliver training in Approximately 1,039 degree or certificate industry. 28. Commonwealth of Pennsylvania, Department of Labor & Industry Pennsyl- vania Allegheny and Tioga $6,000,000 Dislocated workers, youth and high school dropouts, individuals with disabilities, veterans, and individuals with limited English proficiency Will form the Pennsyl

Careers and will addrcompetencies needed demand for 81 targetsix regional projects and will serve 1,379 

29. South Dakota Department of Labor South Dakota None $2,500,000 Veterans, dislocated workers, unemployed individuals, and individuals with criminal records With a focus on energ

efficiency, energy maindustries, expecting trained and 285 to co

30. Utah Department of Workforce Services Utah Box Elder $4,600,000 Dislocated workers, disadvantaged youth, and veterans Will employ an enhan

Stop model, which incurriculum that integtechnician-level trainioccupational training participants for careeindustries. Will place provide 700 participaWorkKeys, yield 456 degrees or certificateparticipants in appren

31. Washington State Workforce Training and Education Coordinating Board Washing- ton None $5,973,635 Dislocated workers, incumbent workers, at-risk youth, low-income adults, individuals with disabilities, and veterans Approximately 4,731 

and placed in energy such as energy efficiecommercial and publ

32. WorkForce West Virginia West Virginia Putnam $6,000,000 Veterans, unemployed workers, and individuals with criminal records Will use grant funds t

training providers’ knindustries; train currebuilding constructioninstallation occupatioskills and entreprenesupport the start up ocollege programs relatechnology and waterWill serve 2,186 parti
Pre-Design Study of the ARRA Grants 

Table A.3 (continued) 

 96 Grant Recipient State Auto-Affected Counties Served Award Amount Participants Served Primary Fo

33. Wisconsin Department of Workforce Development Wisconsin Kenosha and Rock $6,000,000 Veterans, dislocated workers affected by auto industry restructuring, individuals who are currently employed in the targeted trades and in need of updated green building or green manufacturing skills training, recipients of public assistance, and other low-income individuals Will implement the W

the Green Economics green skills training imanufacturing, and soccupations within thgrantee will create neprograms, such as weenergy auditor, and woperator. The granteexisting apprenticeshabout 2,944 participa

34. Wyoming Workforce Development Council Wyoming None $4,495,704 Unemployed individuals, tribal members, individuals with criminal records, veterans, at-risk youth, and dislocated workers Will use eight local tr

train workers in energincluding the use of mtraining to individualWill serve about 1,04

Source: DOL’s announcement of grant winners found at http://www.doleta.gov/ETA_News_Releas

http://www.doleta.gov/pdf/SESP_Summaries.pdf and accessed on January 27, 2010. 

a The counties are listed at http://www.doleta.gov/grants/pdf/Amend-01-SGA-DFA-PY-08-20.pdf, available as of January 27, 2010. 

DOL = U.S. Department of Labor; HERS = Home Energy Rating System; LEED = Leadership in Energy and Environmental Design; SAGE = SeEconomics.

www.mathematica-mpr.com 

Improving public well-being by conducting high-quality, objective research and surveys 

Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ Washington, DC 

Mathematica® is a registered trademark of Mathematica Policy Research