GAO 

United States Government Accountability Office

Report to Congressional Committees

EMPLOYMENT AND 

TRAINING 

ADMINISTRATION 

Increased Authority 

and Accountability 

Could Improve 

Research Program 

January 2010 

GAO-10-243 
Page i GAO-10-243 

Contents 

Letter 1

Appendix I Briefing Slides 7 

Appendix II Dissemination Time Frames for ETA’s Research 

Products Published in 2008 57 

Appendix III Comments from the Department of Labor 59 

Appendix IV GAO Contact and Staff Acknowledgments 63 

Related GAO Products 64 

Table 

Table 1: ETA’s 2008 Research Products by Dissemination Time 

Frames 57 

 Employment and Training Administration 
Abbreviations 

EPA Environmental Protection Agency 

ERS Economic Research Service 

ETA Employment and Training Administration 

HUD Department of Housing and Urban Development 

IES Institute of Education Sciences 

NSF National Science Foundation 

OMB Office of Management and Budget 

OPDR Office of Policy Development and Research 

USDA U. S. Department of Agriculture 

WIA Workforce Investment Act of 1998 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately. 

Page ii GAO-10-243 Employment and Training Administration
Page 1 GAO-10-243 

United States Government Accountability Office

Washington, DC 20548 

 

January 29, 2010 

The Honorable Tom Harkin 

Chairman 

The Honorable Michael B. Enzi 

Ranking Member 

Committee on Health, Education, Labor, and Pensions 

United States Senate 

The Honorable Patty Murray 

Chairman 

The Honorable Johnny Isakson 

Ranking Member 

Subcommittee on Employment and Workplace Safety 

Committee on Health, Education, Labor, and Pensions 

United States Senate 

The Honorable Tom Harkin 

Chairman 

Subcommittee on Labor, Health and Human Services, 

 Education and Related Agencies 

Committee on Appropriations 

United States Senate 

With current rising unemployment rates and the need for a more skilled 

workforce, it is important for the Department of Labor’s (Labor) 

Employment and Training Administration (ETA) to invest in sound 

research that identifies the most effective and efficient ways to train and 

employ workers for 21st century jobs. While ETA traditionally has played 

an important role in providing job training, employment assistance, and 

labor market information for the nation’s workers, the current 

unemployment crisis has made this role more critical. The Bureau of 

Labor Statistics estimates that jobless rates have increased over the past 

year in all 50 states and the District of Columbia. In particular, several 

states have reported unemployment rates well over 10 percent. 

As ETA’s new leadership works to help the nation meet these economic 

challenges, it must have solid information that is supported by sound 

research to guide decision-making. Since 2002, GAO and others have 

criticized ETA for not focusing sufficient attention on its research 

program, particularly with regard to complying with congressional 

mandates, conducting policy-relevant research, and disseminating key 

 Employment and Training Administration 
 

research findings in a timely way. In this context, we have examined the 

structure and processes of ETA’s research and evaluation center in terms 

of the elements that leading national research organizations cite as 

essential to a sound program; that is, research independence, transparency 

and accountability, and policy relevance. Based on these elements, we 

addressed the following questions: (1) How does ETA’s organizational 

structure provide for research independence? (2) What steps has ETA 

taken to promote transparency and accountability in its research program? 

(3) How does ETA ensure that its research is relevant to workforce 

development policy and practice? 

On December 3, 2009, we briefed your staff on the results of our analysis. 

This report formally conveys the information provided during this briefing 

(see app. I for the briefing slides). In summary, we found that ETA’s 

research center lacks independent authority for research, has limitations 

with regard to transparency and accountability processes, has not 

routinely involved stakeholders in developing its research agenda, and has 

been slow to evaluate the programs and activities carried out under the 

Workforce Investment Act of 1998 (WIA). 

• ETA is currently in the process of revising its organizational structure. 

Previously, its research and evaluation center, within the Office of Policy 

Development and Research (OPDR), was several levels removed from the 

Office of the Assistant Secretary and lacked independent authority for 

conducting research. Under ETA’s revised structure, it is unclear whether 

OPDR will report directly to the Assistant Secretary or the Deputy 

Assistant Secretary. Furthermore, unlike the heads of some other research 

and evaluation centers, the head of ETA’s research and evaluation center 

does not have the authority to set the research agenda, to approve 

requests for funded projects, or to disseminate research and evaluation 

reports. 

• While ETA has recently made improvements to its research program, some 

limitations remain with regard to its accountability processes and timely 

dissemination of research products. In 2007, ETA documented all stages of 

its research process from project selection to dissemination. ETA also 

began coordinating its research process with the Office of Management 

and Budget (OMB) and renewed its practice of sponsoring biennial 

research conferences to address a range of policy and program issues. 

Despite these recent changes, ETA’s research processes lack specificity, 

including specific time frames for key milestones and an established 

criteria for peer review. In addition, ETA’s research program lacks an 

information system to track its research from completion to dissemination 

and does not have a mechanism to ensure that research findings are 

Page 2 GAO-10-243 Employment and Training Administration
 

disseminated in a timely manner. In 2008, ETA disseminated 34 research 

products to the public. However, 20 of these products, which had a 

combined cost of about $28 million, were delayed for between 2 and 5 

years. 

• Labor, consistent with recent efforts of OMB, has taken steps to 

emphasize the value of research and evaluation, but ETA does not have 

sufficient mechanisms to ensure that its research can inform policy 

decisions. OMB currently has efforts underway to help agencies to 

enhance their research and evaluation programs. Labor also announced 

plans to create a position of chief evaluation officer and a new evaluation 

center within the Office of the Assistant Secretary for Policy to foster 

research relevant to policy. However, ETA has been slow to comply with a 

congressional mandate to evaluate WIA1

 and it also lacks a standard 

process and advisory bodies to consistently involve stakeholders in the 

development of its research agenda. Similarly, ETA has no formal process 

in place to ensure that research findings are used to inform strategic 

planning and policy. 

GAO is making several recommendations concerning the structure and 

processes of ETA’s research and evaluation center. Specifically, GAO 

recommends that the Secretary of Labor: 

• take steps to ensure that ETA clarifies its organizational structure and 

OPDR reports directly to ETA’s Assistant Secretary; 

• provide sufficient decision-making authority to ETA’s research and 

evaluation center regarding its research; 

• direct ETA’s research and evaluation center to establish more specific 

processes, including time frames for dissemination of research; 

• create an information system to track research projects; and 

• instruct ETA’s research and evaluation center to develop processes to 

involve outside experts in setting its research agenda. 

We used several methodologies to develop our findings. To identify 

elements of sound government-sponsored research, we reviewed relevant 

guidelines established by the American Evaluation Association and the 

 

1

29 U.S.C. § 2917(c). 

Page 3 GAO-10-243 Employment and Training Administration
 

National Research Council of the National Academy of Sciences. To apply 

these guidelines to ETA’s research and evaluation center, we examined 

ETA’s organizational structure and processes in terms of three basic 

elements of sound research—independence, transparency and 

accountability, and policy relevance. As part of this examination, we 

analyzed information on ETA’s disseminated research to determine 

whether research products were released in a timely manner. (App. II 

contains additional information regarding the time frames of these 

research products.) In addition, we reviewed the research structures and 

processes of five other federal agencies’ research and evaluation centers 

that had some of the characteristics that were identified by the American 

Evaluation Association and the National Research Council of the National 

Academy of Sciences to support sound research.2

 We did not evaluate the 

operation or quality of the research produced by the centers but rather the 

structures and policies supporting their research. We also reviewed 

relevant federal laws. 

We conducted this performance audit from October 2009 through 

December 2009 in accordance with generally accepted government 

auditing standards. Those standards require that we plan and perform the 

audit to obtain sufficient, appropriate evidence to provide a reasonable 

basis for our findings and conclusions based on our audit objectives. We 

believe that the evidence obtained provides a reasonable basis for our 

findings and conclusions based on our audit objectives. 

We provided a draft of this report to ETA for its review and comment. ETA 

provided written comments, which are reproduced in appendix III. In 

addition, ETA provided technical comments, which we incorporated 

where appropriate. 

In its response, ETA concurred with two of our recommendations and 

cited ongoing activities addressing the areas covered in the other three. 

Regarding our recommendation to clarify its organizational chart to ensure 

that OPDR reports directly to ETA’s Assistant Secretary, ETA concurred 

 

2

In addition to ETA, we present information for comparative purposes on the following 

research centers: the Department of Agriculture’s (USDA) Economic Research Service 

(ERS) and National Institute of Food and Agriculture; the Department of Education’s 

(Education) Institute of Education Sciences (IES); the Department of Housing and Urban 

Development’s (HUD) Office of Policy Development and Research; the Environmental 

Protection Agency’s (EPA) Office of Research and Development; and the National Science 

Foundation (NSF). 

Page 4 GAO-10-243 Employment and Training Administration
 

and has since clarified this point in an updated organizational chart that it 

has posted on its Web site. ETA also concurred with our recommendation 

to create an information-tracking system for its research products, noting 

that it has plans to implement such a system to track product milestones. 

However, officials did not provide specific time frames for 

implementation. Because of the importance of developing such a system, 

we urge ETA to give this effort high priority. 

On the remaining three recommendations, ETA did not agree or disagree, 

noting instead its relevant ongoing activities. 

• Regarding our recommendation that ETA provide its research and 

evaluation center with sufficient decision-making authority, officials cited 

ongoing efforts to provide recommendations to the Assistant Secretary 

and to collaborate with relevant program offices on ETA’s research plans. 

We considered these activities in our review and discussed them in our 

report, but these efforts alone fall short of providing OPDR with the 

necessary authority to make key decisions, and they do not serve to 

insulate OPDR from undue political influence. 

• Regarding our recommendation that ETA establish more specific 

processes to guide research, officials cited their work with OMB to 

establish time frames for disseminating research. However, specifying 

time frames is but one component of ETA’s research process that needs to 

be addressed. Our report cites other steps that ETA should take including 

specifying the criteria to be used in deciding whether a report should be 

peer reviewed. In ETA’s technical comments, officials noted that they will 

be updating their process steps in early 2010. As ETA reviews and updates 

its processes, we urge the agency to make the steps more specific in order 

to help ensure transparency and accountability. 

• Regarding our recommendation to develop processes to routinely involve 

outside experts in setting its research agenda, ETA officials cited the 

various informal methods they currently use to gather input from outside 

experts. These efforts may serve to increase outside involvement in setting 

the agenda, but they are informal and ad hoc and, therefore, do not 

provide ETA with a formal mechanism to ensure outside experts— 

including researchers, policy makers, and practitioners—consistently have 

a role in setting its research agenda. 

Page 5 GAO-10-243 Employment and Training Administration
 

We are sending copies of this report to relevant congressional 

committees. In addition, this report will be available at no charge on 

GAO’s Web site at www.gao.gov. 

If you or your staff have any questions about this report, please contact me 

at (202) 512-7215 or scottg@gao.gov. Contact points for our Offices of 

Congressional Relations and Public Affairs may be found on the last page 

of this report. GAO staff members who made key contributions to this 

report are listed in appendix IV. 

George A. Scott 

Director, Education, Workforce, 

 and Income Security Issues 

Page 6 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

Page 7 GAO-10-243 

Appendix I: Briefing Slides 

1

Employment and Training Administration: 

Increased Authority and Accountability 

Could Improve Research Program 

Briefing to Congressional Committees

December 3, 2009

Appendix I: Briefing Slides

 Employment and Training Administration 
Appendix I: Briefing Slides 

2

Overview

• Introduction

• Research Objectives

• Scope and Methodology

• Summary of Findings

• Background

• Findings

• Conclusions and Recommendations

Page 8 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

3

Introduction

A sound program of research and evaluation is essential to 

finding effective ways for training and employing workers

for 21st century jobs.

Since 2002, we and others have criticized the Employment

and Training Administration (ETA) for not focusing sufficient 

attention on its research program, particularly with regard to:

• complying with statutory research mandates; 

• conducting research to help policy makers understand 

what works and what does not; and

• disseminating key research findings in a timely way.

Page 9 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

4

Introduction

• Most recently, we have been asked to review the 

extent to which ETA’s structure and processes 

promote sound research. 

• At the time of our review, ETA was transitioning

from one administration to another, and

• ETA’s new leadership is formulating new plans

on how best to carry out its research and evaluations.

Page 10 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

5

Introduction

Leading national research organizations, including the 

American Evaluation Association and the National 

Research Council of the National Academy of 

Sciences—have identified a number of elements of a 

sound research and evaluation program.

Three of the elements of sound research are:

Page 11 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

6

Objectives

Based on these elements, we examined ETA’s research 

program to answer the following questions:

1. How does ETA’s organizational structure provide

for research independence?

2. What steps has ETA taken to promote transparency

and accountability in its research program? 

3. How does ETA ensure that its research is relevant

to workforce development policy and practice? 

Page 12 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

7

Scope and Methodology

To identify basic principles of sound government-sponsored 

research, we reviewed relevant guidelines as developed by

the American Evaluation Association and the National 

Research Council of the National Academy of Sciences.

These guidelines are suitably applicable to a full range

of government research and evaluation centers. 

Page 13 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

8

Scope and Methodology

To examine ETA’s organizational structure and processes, we:

• Interviewed agency officials, stakeholders, and experts. 

• Examined agency documents, policies, and procedures.

To examine the structure and processes of other government 

research and evaluation centers, we:

• Interviewed research experts. 

• Examined agencies’ documents and external reports.

• Obtained information from GAO officials with knowledge

of these centers. 

Page 14 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

9

In addition to ETA, we present information for comparative purposes

on the following research centers:

• The Department of Agriculture’s (USDA) Economic Research Service

(ERS) and National Institute of Food and Agriculture;

• The Department of Education’s (Education) Institute of Education

Sciences (IES);

• The Department of Housing and Urban Development’s (HUD) Office of 

Policy Development and Research;

• The Environmental Protection Agency’s (EPA) Office of Research and 

Development; and

• The National Science Foundation (NSF).

Scope and Methodology

Page 15 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

10

Scope and Methodology

We did not evaluate the operation or quality of the research 

produced by the centers but rather the structures and policies 

supporting their research. 

We selected these centers as examples because they had 

some of the characteristics that support sound research

as identified by the American Evaluation Association

and the National Research Council of the National Academy 

of Sciences guidelines.

Page 16 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

11

Scope and Methodology

We conducted this performance audit from October 2009 to 

December 2009 in accordance with generally accepted 

government auditing standards. Those standards require that 

we plan and perform the audit to obtain sufficient, appropriate 

evidence to provide a reasonable basis for our findings and 

conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our 

findings and conclusions based on our audit objectives.

Page 17 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

12

Key Terminology

• Governing or advisory board: This term generally refers to an 

independent body that provides advice to senior leadership on a 

range of issues, including recommending research priorities, 

helping to safeguard the independence of the agency, and 

providing critical links to practice and policy communities.

• Peer review: A quality assurance process that involves selecting 

independent and knowledgeable individuals to review and provide 

objective feedback on an agency’s research. 

• Research and evaluation center: The entity within an agency 

responsible for planning and conducting research and evaluations.

Page 18 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

13

• ETA is currently in the process of revising its organizational structure. 

Previously, its research and evaluation center, within the Office of Policy 

Development and Research (OPDR), was several levels removed from the 

Office of the Assistant Secretary and lacked independent authority for 

conducting research. 

• While ETA has recently taken steps to improve its research program, some 

limitations remain with regard to its accountability processes and timely 

dissemination of research products.

• The Office of Management and Budget (OMB) and Labor have both 

emphasized the value of research and evaluation. However, ETA has not 

routinely involved stakeholders in developing its research agenda and has 

also been slow to evaluate the Workforce Investment Act of 1998 (WIA).

Summary of Findings

Page 19 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

14

Some Essential Elements of a Sound 

Government Research and Evaluation Program

Background

Some elements of sound research practices that can help 

agencies’ strengthen their research and evaluation function, 

regardless of key differences related to the agency’s overall 

mission and the size of its research and evaluation center. 

Page 20 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

15

According to leading experts, to achieve research 

independence, research and evaluation centers should be:

• Located at a level of influence within the agency or 

department. That is, research centers should be located at 

a level that allows its leaders to report directly to the senior

executive of the organization. 

• Responsible for key aspects of its research program, 

including the design, conduct, and dissemination of 

research studies.

Background

Some Essential Elements

of Sound Practices

Page 21 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

16

Background

To achieve transparency and accountability, research and 

evaluation centers should:

• Release their research agendas to the public;

• Document research policies and procedures; and

• Publicly disseminate research findings in a timely

and accessible manner.

Some Essential Elements

of Sound Practices

Page 22 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

17

To achieve policy relevance, research and evaluation centers 

should:

• Use research results to both inform and respond to agency 

strategic planning;

• Focus research agenda on issues of importance to policy

and practice; and

• Involve key stakeholders (researchers, policy makers, and 

practitioners) in planning, designing, and conducting research.

Background

Some Essential Elements

of Sound Practices

Page 23 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

18

Previously, ETA’s OPDR Did Not Report 

Directly to the Assistant Secretary

Objective 1

Research and evaluation center

was several levels below the 

Assistant Secretary.

Source: Department of Labor’s Employment and Training Administration and GAO analysis.

Page 24 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

19

ETA’s Emerging Organizational Structure 

Does Not Ensure OPDR is Located at a 

Level of Influence

• At the time of our review, ETA was revising its organizational structure.

• Under the revised structure, OPDR is placed at the same level as several 

program offices such as the Offices of Apprenticeship and Trade 

Adjustment Assistance.

• It is unclear from ETA’s revised organizational chart whether OPDR reports 

directly to the Assistant Secretary or the Deputy Assistant Secretary.

• It is also unclear whether, as part of OPDR, ETA’s research and evaluation 

center is located at a level of sufficient influence within ETA.

Objective 1

Page 25 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

20

Objective 1

Alternative Example: Education’s 

Research Center (IES) Reports Directly to 

the Secretary

Source: Department of Education and GAO analysis.

Page 26 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

21

The head of ETA’s research and evaluation center lacks sufficient 

authority to approve:

• its research agenda;

• requests for funded proposals, grant announcements, and 

statements of work; or 

• dissemination of research and evaluation publications.

According to ETA officials, these authorities reside with ETA’s Office 

of the Assistant Secretary.

Neither ETA’s research and evaluation center, nor the Office of Policy 

Development and Research of which it is a part, has a specific 

mechanism that could provide insulation from undue influence.

Objective 1

ETA’s Research and Evaluation Center 

Lacks Sufficient Independence

Page 27 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

22

For example, the directors of IES and NSF take the following actions:

• Implement their research agenda; 

• Design funded projects; 

• Submit reports for peer review and publish research, statistics,

and evaluations with or without the approval of any other office

within their organizations; and

• Work with governing boards on research agenda and policies.

Objective 1

Other Agencies Have Mechanisms Designed 

to Promote Research Independence 

Page 28 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

23

ETA Has Recently Documented Its 

Research Process

In 2007, ETA began documenting its research process.

• This documentation details the eight steps ETA uses to 

conduct its research, from project selection to dissemination, 

and the expected time frames for each step.

• ETA developed this documentation at the request

of OMB.

• Prior to this, ETA did not have a documented process.

However, ETA has not made this documentation available to the 

public and includes little information regarding its processes

on its Web site.

Objective 2

Page 29 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

24

Objective 2

ETA’s Research Process Involves 

Coordination with OMB

In September 2007, ETA agreed to inform OMB of sizable research 

projects and evaluations. 

• OMB reviews the statement of work for all projects that exceed 

$250,000.

• OMB informs ETA if it wants to discuss the statement of work.

• OMB has requested that ETA alert it of research reports 

submitted to ETA for dissemination and not approved by the 

Office of the Assistant Secretary within 9 months.

Page 30 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

25

Since January 2008, ETA has included a provision in its 

research contracts that allows the contractor to release 

products submitted to ETA but not approved by the 

Assistant Secretary within 9 months.

However, ETA does not broadly advertise the availability

of these products on its Web site to the public.

Objective 2

ETA Now Allows Contractors to Release 

Products Delayed for More Than 9 Months

Page 31 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

26

In the past, ETA has sponsored biennial research conferences to address

a range of policy and program issues and document promising program 

practices. This practice ended in 2003.

In September 2009, ETA renewed this practice by sponsoring a national 

research conference on “Recovery and Reemployment” that highlighted 

ETA sponsored research.

• The conference also assembled leading researchers from a variety

of government and private entities and fields of research and practice. 

• ETA has made the PowerPoint presentations of conference sessions

available on the Internet and is in the process of making the audio and 

visual transcripts available to the public.

ETA Has Also Renewed Its Practice of 

Sponsoring National Conferences to 

Highlight Research Findings

Objective 2

Page 32 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

27

Time frames for ETA’s key milestones are stated loosely. 

For example: 

• During the final review and approval process for 

dissemination, project managers are instructed

to inform management when products have not been 

approved by the Office of the Assistant Secretary,

and as a result, have stalled in the process for an 

“excessive” amount of time. 

• However, ETA has not explained what constitutes

an "excessive” amount of time.

Objective 2

Despite Recent Changes, ETA’s 

Research Processes Lack Specificity

Page 33 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

28

ETA’s research process provides for peer review but does not 

specify the criteria that will be used to determine whether a 

report should be peer reviewed. 

ETA officials told us that the scope of a project determines 

whether it will be peer reviewed, but did not provide specific 

criteria.

ETA does not provide information on or guidelines for the peer 

review process, including how comments will be used and 

who will be selected to serve as peer reviewers.

Objective 2

ETA Does Not Have Specific Guidance 

for Peer Review

Page 34 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

29

According to USDA information:

• ERS conducts peer reviews of its reports and has 

established review guidelines, including criteria for 

selecting qualified peer reviewers.

• Each report is subjected to both internal and formal, 

independent, external peer review.

• ERS requires thorough review of data products by 

knowledgeable staff prior to dissemination.

Objective 2

ERS Reported That It Has Specific 

Policies and Procedures for Peer Review

Page 35 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

30

ETA has not established an information system to track its 

research products from completion to dissemination. 

• As a result, ETA is unable to track the status of products 

and determine which products are delayed and where in 

the process they are delayed. 

• Without a tracking system, it is difficult for ETA to 

determine if it is following its processes or complying 

with established time frames.

Objective 2

ETA Lacks a System to Track Research 

Progress

Page 36 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

31

ETA disseminated 34 research products to the public in 2008.

• Twenty of these products had been waiting 2 to 5 years for approval. 

(See app. II for information on report dissemination time frames.)

• ETA officials attributed the delays to inadequate methodologies and 

poorly written products. However, ETA had previously agreed to 

these methodologies at design and reviewed and approved earlier 

drafts of these reports.

• Presently ETA is reviewing its procedures to identify

areas where it can obtain organizational buy-in earlier

in the process. 

Objective 2

ETA Has Not Consistently

Disseminated Research Findings

in a Timely Manner

Page 37 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

32

• Most of ETA’s sponsored research whose publication was delayed 

until 2008 were evaluations of workforce development strategies.

• The combined cost of the 20 delayed research products was about 

$28 million.

• Due to the delays, findings from these studies could have a limited 

impact in influencing policy and practice. 

• Delayed products covered a range of topics including 

evaluations of labor exchanges in the one-stop delivery system, 

the youth offender demonstration project, and workforce 

development in rural areas.

Objective 2

Dissemination Delays Could Have 

Cost, Policy, and Practice Implications

Page 38 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

33

ETA’s dissemination of research findings is largely limited

to postings on its Web site and consists of research

and evaluation reports that it has sponsored. 

ETA also issues e-mail advisories to notify its subscribers of 

newly released research publications.

ETA has not used dissemination strategies, such as the use of 

clearinghouses, to more broadly disseminate its research.

Objective 2

ETA’s Dissemination of Research 

Findings Is Limited

Page 39 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

34

According to department information: 

• HUD uses a dedicated Web site to disseminate information that it and 

others have collected on housing issues, including housing market 

trends, economic and demographic data, housing finance, and energy 

and resource-efficient housing designs and construction.

• It has two clearinghouses that provide (1) information on state and 

local regulatory reform strategies to support affordable housing and 

(2) access to HUD-sponsored research and publications.

• HUD also makes data sets collected for both research and 

administrative purposes routinely available to the public.

Objective 2

HUD Disseminates Its Research, including

Data Sets, through Clearinghouses

and a Dedicated Web Site

Page 40 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

35

According to department information:

• EPA partners with other federal agencies and with states to disseminate 

and publicize its research.

• EPA’s Web site contains a searchable database of EPA science activities 

and scientific and technical products produced by EPA and through EPAfunded assistance agreements. 

• It also provides public access to its data sets.

• Users can receive e-mail notifications of new products based on key 

words, topics, or general research.

• EPA issues press releases and holds research and press conferences.

Objective 2

EPA Disseminates Its Research Products 

through Partnerships and a Range of Media

Page 41 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

36

Current efforts are underway at OMB to encourage agencies to 

increase their emphasis on the importance of research and 

evaluations and strengthen evaluation methods.

OMB plans to work with agencies to:

• Expand information about program evaluations that are made 

public.

• Establish an interagency working group to build evaluation 

capacity and create effective evaluations that work.

• Provide additional funding to agencies for high-priority 

evaluation activities.

Objective 3

OMB Has Taken Steps to Promote and 

Strengthen Evaluation Efforts

Page 42 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

37

In September 2009, Labor announced plans to create a position of chief 

evaluation officer and a new evaluation center within the Office of the 

Assistant Secretary for Policy. 

• This position is designed to link evaluation with policy development 

and strategic planning. 

• However, the linkage between the new position and ETA’s research

and evaluation center is unclear. 

Recently, ETA’s Assistant Secretary has acknowledged the importance of 

research-driven policy and the need to conduct rigorous evaluations.

ETA has taken steps to increase its evaluation activities and has applied to 

OMB for additional funds to conduct rigorous evaluations. 

Objective 3

Labor Has Taken Recent Steps to 

Foster Research Relevant to Policy

Page 43 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

38

Congress required an impact evaluation of WIA activities by 

2005.1

• However, ETA did not award a contract to conduct the 

evaluation until 2008.

• As of mid-November 2009, the evaluation’s research 

design has not yet been finalized.

• Officials told us that they plan to implement the 

evaluation in January 2010.

1 29 U.S.C. § 2917(c).

Objective 3

However, ETA Has Been Slow to 

Evaluate WIA Program Activities

Page 44 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

39

ETA has had no standard process for selecting stakeholders to review

its research agenda or incorporating their comments.

ETA’s documents indicate that, in recent years, ETA’s Assistant Secretary 

had set and implemented the agenda with limited involvement from

external stakeholders. 

• For example, ETA did not solicit public comment on its finalized 2007 

research plan. 

• Moreover, ETA never finalized its 2003 and 2005 revisions to its

5-year research plan or made the plans available to the public. 

• However, in 1999, ETA advertised the availability of a draft of its plan 

through a Federal Register notice and solicited public comments for 

consideration before the plan was finalized.

Objective 3

Key Stakeholders Have Not Been 

Consistently Involved to Ensure ETA’s 

Research Is Relevant

Page 45 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

40

For example:

• EPA’s Board of Scientific Counselors provides advice, information, 

and recommendations about its research program.

• IES’ National Board for Education Sciences approves or 

disapproves the priorities for the Institute proposed by the Director 

and ensures that its priorities are consistent with its mission.

• NSF’s National Science Board establishes its research policies

and approves its strategic budget direction.

Objective 3

Unlike Other Agencies, ETA Does Not Have 

Advisory Bodies to Involve Stakeholders in 

the Development of Its Research Agenda

Page 46 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

41

Currently, 14 advisory boards assist Labor and serve a variety of 

purposes. For example: 

• The Advisory Committee on Apprenticeship provides assistance 

and advice to the Assistant Secretary of ETA on policies and 

programs regarding apprenticeship.

• The Bureau of Labor Statistics Data Users Committee 

provides advice on the collection and analysis of the 

Bureau's statistics, its published reports, and its overall 

mission. 

Objective 3

Labor Is Currently Assisted by 14 Advisory 

Boards That Perform a Variety of Functions

Page 47 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

42

According to USDA, ERS’ procedures call for consulting with 

stakeholders—including policy makers and key institutions 

that influence public policy.

• ERS involves stakeholders in discussions of past 

research accomplishments, and program effectiveness 

and impact. 

• ERS also consults with and works across department 

offices and programs in devising cross-cutting research.

Objective 3

ERS Routinely Consults with 

Stakeholders to Ensure Its Research

Is Relevant and Useful

Page 48 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

43

• Involving advisory committees requires additional time and effort, as well 

as some minimal expense. 

• In creating advisory committees, federal agencies are generally required to 

follow the provisions of the Federal Advisory Committee Act, which include 

provisions limiting when such committees can be established. 

• Most of the agencies we reviewed had advisory committees.

For example, during FY 2008, the General Services Administration 

reported a total of 917 active committees with a total of nearly 64,000 

members who provided advice and recommendations to 50 federal 

agencies. 

Objective 3

While Advisory Committees Can Promote

Research Relevance, They Involve Some 

Trade-Offs

Page 49 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

44

Objective 3

ETA Does Not Have a Formal Process

in Place to Ensure Strategic Planning

Is Informed by Research

Although ETA coordinates with its strategic planning office, 

there is no formal process in place to ensure that research 

findings are used to inform strategic planning and policy.

For example:

• Officials told us that they respond to requests from other 

departments to provide information on research and 

evaluation findings.

• However, there is no formal process in place to ensure 

that all research and evaluation findings are shared

on a consistent basis.

Page 50 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

45

Objective 3

In Contrast, EPA’s Structure Has the 

Potential to Encourage Coordination 

Between Research and Strategic Planning

According to agency documents, EPA has established two 

offices within its research office to help ensure its policy 

and planning are informed by its research.

• Office of Science Advisor – establishes specific 

mechanisms for ensuring that scientific results and 

hypotheses, with technical evaluation and peer-review, 

play a prominent role in all regulatory decisions.

• Office of Science Policy – ensures that the scientific 

information generated through EPA and other research 

is used to inform EPA decisions.

Page 51 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

46

According to department information, USDA’s advisory board plays a 

key role in coordinating strategic planning for the National Institute 

of Food and Agriculture.

The board is to help this institute to: 

• establish policies and priorities on a quarterly basis, 

• evaluate the effectiveness of those policies and priorities, and

• develop its 5-year strategic plan.

In addition, according to the National Institute of Food and 

Agriculture, it revises its strategic plan on an ongoing basis so that 

its research, education, and extension strategies align with 

Agriculture’s goals and objectives.

Objective 3

USDA’s Advisory Board Reported That It 

Serves as a Link to Strategic Planning

Page 52 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

47

The National Board for Education Services is required by law to:

• Solicit advice and information from those in the field of 

education to recommend topics that require research, and

• Recommend ways to enhance partnerships and 

collaboration among other federal and state research 

agencies.

IES also involves stakeholders by establishing: 

• Fellowships for education research at universities, and 

• Standing review panels of scientists from university

and industry settings who review grant applications.

Objective 3

Additionally, IES’ Board Facilitates 

Stakeholder Involvement

Page 53 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

48

Conclusions

• A variety of elements—including research independence, 

transparency and accountability, and policy relevance—may 

foster conditions that can help achieve sound and relevant 

research. 

• ETA’s OPDR and its component research center have limited 

decision-making authority and lack formal access to 

executive leadership leaving it potentially vulnerable to undue 

political influence.

Page 54 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

49

Conclusions

• Moreover, lacking specific processes and a tracking system to 

monitor progress through review inhibits transparency and may 

lead to continued delays in approving and disseminating research

products. 

• The involvement of stakeholders and advisory bodies can help 

ensure that decisions to conduct research are grounded in policy

implications. Such involvement also serves to foster public 

accountability. While deciding to involve advisory bodies comes at 

some cost in time and resources, such a move may lend credibility 

to the research program.

Page 55 GAO-10-243 Employment and Training Administration
Appendix I: Briefing Slides 

50

Recommendations

To improve ETA’s research program, we recommend that the Secretary of Labor:

• Take steps to clarify ETA’s revised organizational structure and ensure that 

OPDR reports directly to ETA’s Assistant Secretary;

• Provide sufficient authority to ETA’s research and evaluation center to plan, 

conduct, and disseminate research;

• Direct ETA’s research and evaluation center to establish more specific 

processes, including time frames for dissemination of research to promote 

transparency and accountability; 

• Create an information system to track research projects at all phases to ensure 

timely completion and dissemination; and

• Instruct ETA’s research and evaluation center to develop processes to routinely 

involve outside experts in setting its research agenda and to the extent 

required, do so consistent with the Federal Advisory Committee Act.

Page 56 GAO-10-243 Employment and Training Administration
Appendix II: Dissemination Time Frames for 

ETA’s Research Products Published in 2008 

Table 1: ETA’s 2008 Research Products by Dissemination Time Frames 

Publication title ETAOP# 

Date of final 

submission to 

ETA/OPDR 

Date posted on 

ETA publication 

database/Web site 

Time frames 

between submission 

and disseminationa

1 Current Strategies to Employ and Retain Older 

Workers 

2008-01 1/1/2008 3/3/2008 2.2 months 

2 Reemployment and Eligibility Assessment 

(REA) Study 

2008-02 3/1/2008 3/12/2008 11 days 

3 Assessment of Strategies to Retain 

Experienced Technical and Professional 

Healthcare Personnel After Retirement Age: 

Mature Healthcare Workers Focus on Group 

Research - Final Report 

2008-04 1/22/2008 5/8/2008 4 months 

4 Early Implementation of Generation I of the 

Workforce Innovation in Regional Economic 

Development (WIRED) Initiative: 2007 Interim 

Evaluation Report 

2008-03 5/16/2008 5/21/2008 5 days 

5 Evaluation of State Implementation of Section 

303 (K), Social Security Act, “SUTA Dumping” - 

Final Report 

2008-05 5/7/2008 6/12/2008 1 month 

6 Evaluation of Prisoner Re-Entry InitiativeInterim Report 2008-06 9/28/2007 6/16/2008 8.5 months 

7 Unemployment Insurance: Assessment of the 

Impact of the 2002 Reed Act Distribution 

2004-11 12/1/2004 7/30/2008 3.8 years 

8 Responses to Personal Remployment 

Accounts (PRAs): Findings from the 

Demonstration States — Final Evaluation 

Report 

2008-07 6/6/2008 8/11/2008 2 months 

9 Navigating the U.S. Labor Market: Trends and 

Prospects for Workers 

2003-09 9/22/2003 8/14/2008 4.10 years 

10 Review of Recent Pilot, Demonstration, 

Research, and Evaluation Initiatives to Assist in 

the Implementation of Programs under the 

Workforce Investment Act 

2003-10 9/22/2003 8/14/2008 4.11 years 

11 Net Impact Estimates for Services Provided 

through the Workforce Investment Act 

2005-06 10/1/2005 8/14/2008 2 .10 years 

12 Review of Alternative Methodologies for 

Employment and Training Research 

2003-11 9/22/2003 8/15/2008 4.9 years 

13 Anatomy of a One-Stop: Baltimore City East 

Side Career Center 

2006-07 8/20/2008 8/29/2008 9 days 

14 Anatomy of Two One-Stops: Camdenton and 

Columbia, Missouri 

2006-08 8/20/2008 8/29/2008 9 days 

15 Youth Offender Demonstration Project 

Evaluation Final Report – Volume One 

2006-06 6/1/2006 9/4/2008 2.3 years 

Appendix II: Dissemination Time Frames for 

ETA’s Research Products Published in 2008 

Page 57 GAO-10-243 Employment and Training Administration
Appendix II: Dissemination Time Frames for 

ETA’s Research Products Published in 2008 

 Publication title ETAOP# 

Date of final 

submission to 

ETA/OPDR 

Date posted on 

ETA publication 

database/Web site 

Time frames 

between submission 

and disseminationa

16 Workforce Development in Rural Areas - 

Changes in Access, Service Delivery and 

Partnerships 

2005-07 6/30/2005 9/4/2008 3.2 years 

17 Unemployment Insurance and Reemployment 

among Older Workers 

2006-09 7/1/2006 9/4/2008 2.2 years 

18 Youth Offender Demonstration Project Process 

Evaluation Final Report - Round Two 

2004-10 1/30/2004 9/8/2008 4.2 years 

19 Evaluation of Labor Exchange in the One-Stop 

Delivery System Environment 

2004-09 2/1/2004 9/11/2008 4.7 years 

20 Use of Experimental Methods in Workforce 

Evaluations 

2005-08 4/18/2005 9/16/2008 3.5 years 

21 On the Use of Administrative Data for 

Workforce Development Program Evaluations 

2005-09 4/18/2005 9/16/2008 3.5 years 

22 Immigration and the Effects on the US Labor 

Market 

2005-10 4/18/2005 9/16/2008 3.5 years 

23 The Labor Market Effects of Globalization and 

TAA 

2005-11 4/18/2005 9/16/2008 3.5 years 

24 Systemic Disincentive Effects of the UI 

Program 

2005-12 4/18/2005 9/16/2008 3.5 years 

25 Older workers and the Labor Market/Labor 

Market Policies for Older Workers 

2005-13 4/18/2005 9/16/2008 3.5 years 

26 Programs to Support Out-of-School Youth 2005-14 4/18/2005 9/16/2008 3.5 years 

27 Community College Training and the Workforce 

Investment System 

2005-15 4/18/2005 9/16/2008 3.5 years 

28 Project GATE 2008-08 5/1/2008 9/17/2008 4.5 months 

29 Flexible Learning Options for Adult Students 2008-09 4/1/2008 10/16/2008 6.5 months 

30 Implementation Analysis of High Growth Job 

Training Initiative (HGJTI) 

2008-10 6/1/2008 10/16/2008 4.5 months 

31 Literature Review: Intermediaries and FBCOs 

Working Together 

2008-11 10/2/2008 10/29/2008 27 days 

32 Youth Opportunity Grants Initiative (YO) 2008-12 12/1/2005 12/23/2008 3.0 years 

33 What’s Known About the Effects of PubliclyFunded Employment and Training Programs 2006-10 4/21/2006 12/15/2008 2.8 years 

34 Competency Models - A Review of the 

Literature and the Role of the Employment and 

Training Administration (ETA) 

2008-13 1/29/2008 12/23/2008 11 months 

Source: Department of Labor’s Employment and Training Administration and GAO analysis of data. 

Note: The information presented in this table was provided by ETA officials. 

a

Time frames were calculated using the date each research product was submitted to ETA and the 

date it was posted on ETA’s Web site. 

Page 58 GAO-10-243 Employment and Training Administration
Appendix III: Comments from the Department Appendix III: Comments from the of Labor 

Department of Labor 

Page 59 GAO-10-243 Employment and Training Administration 
Appendix III: Comments from the Department 

of Labor 

Page 60 GAO-10-243 Employment and Training Administration
Appendix III: Comments from the Department 

of Labor 

Page 61 GAO-10-243 Employment and Training Administration
Appendix III: Comments from the Department 

of Labor 

Page 62 GAO-10-243 Employment and Training Administration
Appendix IV: 

A

GAO Contact and Staff 

cknowledgments 

Page 63 GAO-10-243 

Appendix IV: GAO Contact and Staff 

Acknowledgments 

George A. Scott, Director (202) 512-7215 or scottg@gao.gov GAO Contact 

In addition to the contact listed above, Dianne Blank (Assistant Director) 

and Kathleen White (Analyst-in-Charge) supervised the development of 

this product. Linda Stokes, Amy Sweet, and Ashanta Williams made 

significant contributions to all aspects of this report. In addition, 

Stephanie Shipman assisted with the conceptual development of our 

study; Amanda Miller assisted with design and analyses; James Bennett 

provided graphic assistance; Susan Bernstein provided writing assistance; 

Alex Galuten provided legal support; and Karen Brown and Melissa Jaynes 

verified our findings. 

Acknowledgments 

 Employment and Training Administration 
Related GAO Products Related GAO Products 

Workforce Investment Act: Labor Has Made Progress in Addressing 

Areas of Concern, but More Focus Needed on Understanding What Works 

and What Doesn’t. GAO-09-396T. Washington, D.C.: February 26, 2009. 

Federal Research: Policies Guiding the Dissemination of Scientific 

Research from Selected Agencies Should Be Clarified and Better 

Communicated. GAO-07-653. Washington, D.C.: May 17, 2007. 

Data Quality: Expanded Use of Key Dissemination Practices Would 

Further Safeguard the Integrity of Federal Statistical Data. GAO-06-607. 

Washington, D.C.: May 31, 2006. 

Workforce Investment Act: Substantial Funds Are Used for Training, but 

Little Is Known Nationally about Training Outcomes. GAO-05-650. 

Washington, D.C.: June 29, 2005. 

Program Evaluation: An Evaluation Culture and Collaborative 

Partnerships Help Build Agency Capacity. GAO-03-454. Washington, 

D.C.: May 2, 2003. 

Workforce Investment Act: Improvements Needed in Performance 

Measures to Provide a More Accurate Picture of WIA Effectiveness.

GAO-02-275. Washington, D.C.: February 1, 2002. 

(130973) Page 64 GAO-10-243 Employment and Training Administration
GAO’s Mission The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help 

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost 

is through GAO’s Web site (www.gao.gov). Each weekday afternoon, GAO 

posts on its Web site newly released reports, testimony, and 

correspondence. To have GAO e-mail you a list of newly posted products, 

go to www.gao.gov and select “E-mail Updates.” 

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Phone The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s Web site, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information. 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Ralph Dawn, Managing Director, dawnr@gao.gov, (202) 512-4400 

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, DC 20548 

To Report Fraud, 

Waste, and Abuse in 

Federal Programs 

Congressional 

Relations 

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548 

Public Affairs 

Please Print on Recycled Paper