Employment and Training Administration: More Actions Needed to Improve Transparency and Accountability of Its Research Program
Published: 2011-03-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: To help guide the nation's workforce development system, the Department of Labor's (Labor) Employment and Training Administration (ETA) conducts research in areas related to job training and employment. Building upon our earlier work, GAO examined the following: (1) To what extent do ETA's research priorities reflect key national employment and training issues and how useful were the studies funded under them? (2) What steps has ETA taken to improve its research program? (3) How has ETA improved the availability of its research since our last review in January 2010? To answer these questions, GAO reviewed ETA's research reports disseminated between January 2008 and March 2010 costing $1 million or more, as well as ongoing studies costing $2 million or more. GAO also convened a virtual expert panel, interviewed Labor and ETA officials, and reviewed relevant documents.
section_title: What GAO Found
paragraphs: ETA's 2007 to 2012 research plan generally addressed key employment and training issues, but some studies were limited in their usefulness. Most experts on our panel reported that the areas in ETA's plan reflected key national employment and training issues at least to a moderate extent. ETA invested most of its research and evaluation resources in the areas of Unemployment Insurance and increasing labor market participation of underutilized groups. Of the $96 million that ETA invested in the 58 research reports we reviewed, more than half--or about $56 million--funded studies in these two areas. The methodological approaches and statistical procedures researchers used in the studies we reviewed were generally consistent with the questions posed, but the studies were not always useful for informing policy and practice. For example, in one study, shortcomings in the data collection phase limited the strength of the findings. Experts suggested that ETA include more varied and rigorous methodologies in its future research projects. They also reported that future research should address additional areas, including a focus on employment and training approaches that work and for whom. Labor and ETA have taken steps to improve the way research is conducted, but additional actions are needed. In acknowledging the need for more rigorous evaluations to inform its policies, Labor recently established the Chief Evaluation Office to oversee departmentwide research and evaluation efforts. In addition, ETA made changes to some of its research practices. For example, ETA has begun involving outside experts in developing its research plan. Despite these improvements, ETA's process lacks critical elements needed to ensure that current improvements become part of its routine practices. For example, ETA's process lacks a formal provision to consult with the newly established Chief Evaluation Officer at important points in the research process. Moreover, ETA's current research practices fall short of ensuring research transparency and accountability--essential elements of a sound research program. For example, its research and evaluation center lacks safeguards to protect it from undue outside influence. ETA has recently begun efforts to increase the rigor of its research designs, but has faced design and implementation challenges. For example, some of ETA's ongoing research studies face challenges in recruiting large enough sample sizes to meet the studies' objectives. ETA has improved the availability of its research findings, but it lacks a plan for assessing the usability of its Web-based search page--the primary tool for making ETA's research publicly available. ETA recently improved the timeliness with which it disseminates its research reports, decreasing the average number of days to release its reports to the public from 804 days in 2008 to 76 days in 2009. ETA has taken steps to update its online, Web-based search page. However, the agency has not developed a formal plan for assessing the overall effectiveness of its Web-based search page, including user satisfaction. In addition to its research database, ETA uses a variety of other methods to disseminate its research, including providing its research reports at conferences and internal briefings. Experts suggested that ETA consider other effective dissemination methods, such as publishing a one-page summary of research findings.
section_title: What GAO Recommends
paragraphs: GAO recommends that ETA formally include the Chief Evaluation Officer in its research process, create a mechanism to enhance the transparency and accountability of its research program, and develop a plan to ensure that research reports are accessible through its Web-based search page. Labor agreed with our recommendations and noted its ongoing efforts to improve its research program. While these efforts are important, GAO stresses the need for additional actions to fully address the recommendations.

section_title: Letter
paragraphs: Across the country, the workforce development systems’ one-stop centers  serve as the key access point for services that are crucial in today’s  economy—services that include Unemployment Insurance (UI) benefits,  job training, and employment assistance. The Department of Labor’s  (Labor) Employment and Training Administration (ETA) is responsible for  guiding the nearly $13 billion public workforce development system. Its  mission is to help make the U.S. labor market function more efficiently by  developing policies that lead to high-quality job training, employment,  labor market information, and income maintenance services. To help  shape its policies, ETA conducts evaluations and research studies on a  range of employment-related topics. Congress appropriated about $103  million to ETA’s research and evaluation budget line items for 2010.
Over the last decade, however, ETA’s research and evaluation program has  fallen short in its efforts to conduct research that can help answer urgent  workforce policy questions. In 2008 and again in 2009, we faulted ETA for  failing to conduct research and evaluations that would lead to an  understanding of what works and what doesn’t. For example, in January  2010, we reported on shortcomings in ETA’s research structure and  processes. We found that ETA’s research and evaluation center, the  Office of Policy Development and Research (OPDR),  lacked independent authority to make key decisions about its research;    maintained processes that were unclear and that lacked transparency  lacked a standard process for ensuring stakeholder involvement or  other strategies to ensure that research addressed key national  priorities; and    had been slow to distribute its research findings and slow to respond to  its statutory mandate to evaluate the Workforce Investment Act of 1998  (WIA).
Based on these findings, we made several recommendations to Labor to  improve ETA’s research program. (For information on the status of those  recommendations, see app. I.)
As ETA’s leadership moves forward to help the nation meet its current  employment challenges, questions remain about how well ETA’s research  has prepared the workforce development system for the challenges of  today. Against this backdrop, you asked us to build upon our January 2010  review and further examine ETA’s research program. Specifically, we  answered the following questions:  1.  To what extent do ETA’s research priorities reflect key national  employment and training issues and how useful were the studies  funded under them?  2.  What steps has ETA taken to improve its research program?  3.  How has ETA improved, if at all, the availability of its research since  our last review in January 2010 and what other steps could ETA take to  further ensure its research findings are readily available?
To address our objectives, we reviewed the 58 research and evaluation  reports that ETA disseminated between January 2008 and March 2010 and  assessed the methodological soundness of the 11 completed studies that  cost $1 million or more. In addition, we reviewed the 10 ongoing studies  costing $2 million or more to determine if research practices or the  soundness of research designs had changed over time. In addition, we  convened a virtual (Delphi) expert panel of academics, researchers, and  consultants to obtain their opinions on ETA’s research priorities and  dissemination methods. To learn how ETA determines what research to  conduct, we interviewed Labor officials and reviewed relevant agency and  budget documents. In addition, we conducted site visits to two local  workforce agencies in Pennsylvania and Virginia that are implementing  ETA’s ongoing research studies to obtain information about challenges  and issues associated with participating in studies. To evaluate the  availability of ETA’s research, we analyzed dissemination time frames for  all publications released between January 2008 and March 2010 and we  tested the ability of ETA’s research database to support searches generally  available to users of research databases. (See app. II for more details on  our objectives, scope, and methodology.)
We conducted this performance audit from March 2009 through March  2011 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: WIA sets forth various requirements for the Secretary of Labor relating to  research and evaluation of federally funded employment-related programs  and activities. The law calls upon the Secretary of Labor to publish in the  Federal Register every 2 years a plan that describes its pilot,  demonstration, and research priorities for the next 5 years regarding  employment and training. Specifically, WIA requires the Secretary to    develop the research plan after consulting with states, localities, and    send the plan to the appropriate committees of Congress; and  take into account such factors as the likelihood that the results of the  projects will be useful to policymakers and stakeholders in addressing  employment and training problems.
Within ETA, OPDR’s Division of Research and Evaluation plans, conducts,  and disseminates employment and training-related research and  evaluations. Nearly all of the agency’s research and evaluation studies are  conducted under contract; these contractors represent a range of research  organizations and academic institutions. Furthermore, OPDR plans and  conducts its research and evaluation activities in consultation with ETA’s  program offices, such as the Office of Workforce Investment and the  Office of Trade Adjustment Assistance.
ETA’s research and evaluation funding is divided into two separate budget  line items:  Pilots, demonstrations, and research. Efforts in this category are focused  on developing and testing new ways to approach problems and to deliver  services. Under WIA, pilots and demonstrations shall be carried out “for  the purpose of developing and implementing techniques and approaches,  and demonstrating the effectiveness of specialized methods, in addressing  employment and training needs.” WIA also states that the Secretary shall  “carry out research projects that will contribute to the solution of  employment and training problems in the United States.”
Evaluations. Efforts in this category are focused on continuing  evaluations of certain programs and activities carried out under WIA.  These evaluations must address  the effectiveness of these programs and activities carried out under  WIA in relation to their cost;  the effectiveness of the performance measures relating to these  programs and activities;  the effectiveness of the structure and mechanisms for delivery of  services through these programs and activities;  the impact of the programs and activities on the community and  participants involved, and on related programs and activities;  the extent to which such programs and activities meet the needs of  various demographic groups; and    such other factors as may be appropriate.
In program year 2010, ETA’s combined budget appropriation for  conducting evaluations and pilots, demonstrations, and research was  about $103 million—or nearly $34 million above what the agency  requested. (See fig. 1.) About $84 million of the 2010 funds were  designated by the Congress for specific projects, including $30 million for  Transitional Jobs activities for ex-offenders, and another $5.5 million for  competitive grants addressing the employment and training needs of  young parents. According to agency documents, in 2008 and 2009, the  Congress similarly increased ETA’s requested budget for pilots,  demonstrations, and research, at the same time specifically designating  how the majority of those funds would be used, including $4.9 million in  2008 and $5 million in 2009 for the young parents’ demonstration.
subsections: section_title: Key Elements of Sound Research and Evaluation Programs
paragraphs: While there is no single or ideal way for government agencies to conduct  research, several leading national organizations have developed guidelines  that identify key elements that promote a sound research program. These  guidelines identify five elements as key: agency resources, professional  competence, independence, evaluation policies and procedures, and  evaluation plans.
Resources. Research should be supported through stable, continuous  funding sources and through special one-time funds for evaluation  projects of interest to executive branch and congressional policymakers.
Professional competence. Research should be performed by professionals  with appropriate training and experience for the evaluation activity (such  as performing a study, planning an evaluation agenda, reviewing  evaluation results, or performing a statistical analysis).
Independence. Although the heads of federal agencies and their  component organizations should participate in establishing evaluation  agendas, budgets, schedules, and priorities, the independence of  evaluators must be maintained with respect to the design, conduct, and  results of their evaluation studies.
Evaluation policy and procedures. Each federal agency and its evaluation  centers should publish policies and procedures and adopt quality  standards to guide evaluations within its purview. Such policies and  procedures should identify the kinds of evaluations to be performed and  the criteria and administrative steps for developing evaluation plans and  setting priorities, including selecting evaluation approaches to use,  consulting experts, ensuring evaluation product quality, and publishing  reports.
Evaluation plans. Each federal agency should require its major program  components to prepare annual and multiyear evaluation plans and to  update these plans annually. The planning should take into account the  need for evaluation results to inform program budgeting, reauthorization,  agency strategic plans, program management, and responses to critical  issues concerning program effectiveness. These plans should include an  appropriate mix of short- and long-term studies to produce results for  short- or long-term policy or management decisions. To the extent  practical, the plans should be developed in consultation with program  stakeholders.
Furthermore, leading organizations, including the American Evaluation  Association and the National Academy of Sciences, emphasize the need  for research programs to establish specific policies and procedures to  guide research activities. Based on several key elements identified by  these organizations, we developed a framework comprised of five  phases—agenda setting, selecting research, designing research,  conducting research, and disseminating research results. (See fig. 2.)
Agenda setting. Agencies should establish a structured process for  developing their research priorities. The process should identify how  agencies set research priority areas and provide for updating the areas on  a regular basis. The process should also allow for the consideration of  critical issues and state how internal and external stakeholders will be  included in developing the plan.
Selecting research. At this phase, the process should identify how the  research program’s staff identifies and selects studies to fund, including  the criteria it uses to make those decisions. Steps might describe how the  staff assembles a list of potential studies, works with internal program  offices, and makes final decisions.
Designing research. During the design phase, the process should identify  steps taken to select appropriate research approaches and methods and  the safeguards in place to ensure appropriate tradeoffs are made between  what is desirable and what is practical and between the relative strengths  and weaknesses of different methods.
Conducting research. At this stage, the process should include policies  and procedures to guide the conduct of research. The process should  ensure that key events, activities, and time frames are specified and that  knowledgeable staff in the sponsoring agency monitor the implementation  of the research.
Disseminating research. This process should describe how research  findings are made available to the public and disseminated to all potential  users. These dissemination methods should include safeguards to ensure  research findings are disseminated in a timely manner and are accessible  through the Internet with user-friendly search and retrieval technologies.
section_title: Research Terminology in This Report
paragraphs: In this report, we use several technical terms in describing ETA’s research  designs and study characteristics. (See table 1.)
section_title: ETA’s Research Areas Generally Reflect Key Issues, but Some Studies Are of Limited Usefulness
subsections: section_title: Experts Thought ETA’s 2007 to 2012 Research Plan Reflected Key Areas, but They Also Suggested New Ones for Future Research
paragraphs: Our expert panel generally considered ETA’s research areas to be the right  ones for the period the research plan covered. About three-fourths of the  panel members reported that ETA’s 2007 to 2012 research agenda reflected  key employment and training issues to at least a moderate extent.  However, a few experts commented that some of ETA’s research areas  may be too broad and lack specificity.
The areas in ETA’s 2007 to 2012 research plan covered a range of issues,  from job training to postsecondary education. Table 2 illustrates the scope  of ETA’s research areas.
With regard to the specific studies within these research areas, ETA  invested most of its research and evaluation resources in work that  focused on increasing the labor market participation of underutilized  workers and on UI. Of the estimated $96 million that supported the 58  research reports published between January 2008 and March 2010, more  than half—about $56 million—funded research that addressed these two  research areas. Other areas received far less funding. For example,  funding for studies addressing the methods of expanding U.S. workforce  skills and using state-level administrative data to measure progress and  outcomes accounted for about $6.5 million, or about 6.7 percent of the  cost of studies published during the period we examined. (See table 3.)  Overall, the individual studies that ETA funded addressed a wide variety of  issues and ranged in cost from about $15,000 to a high of about $22  million.
In addition to the research areas covered in ETA’s 2007 to 2012 research  plan, experts from our virtual panel suggested that ETA incorporate  additional research areas in its future research agenda. Of the research  areas identified, over half of our experts (28 of 39) ranked the  identification of employment and training approaches that work, and for  whom, as one of the top areas that ETA’s future research should address.  (See fig. 3.) Without such focus, experts commented that it will be difficult  to know how to improve the nation’s workforce system. Other issues  ranked at the top by experts included research on job creation strategies  and the impact of long-term and short-term training. (See app. III for more  information on issue-area rankings.)
In addition to identifying overall employment and training areas, including  issues related to UI, experts also identified specific aspects of the UI  system that could be examined in ETA’s future research. In particular,  most experts (34 of 39 respondents) reported that it would be at least  moderately important, in the future, for ETA to research the linkage  between UI and employment and safety net programs, such as Temporary  Assistance for Needy Families or the Supplemental Nutrition Assistance  Program. (See fig. 4.) This area of research may be particularly important  given the role that these programs play in supporting individuals during  economic downturns. In addition, many experts (24 of 39 respondents)  mentioned that ETA should make the examination of the incentives and  disincentives in the UI system a research priority, given the challenge of  supporting unemployed workers during difficult economic times, while  promoting self-sufficiency through employment.
Experts also reported that it is important to fund research on what works  for selected population groups. Of the population groups identified, the  experts on our virtual panel most often ranked the long-term unemployed,  economically disadvantaged workers, and adults with low basic skills as  the top populations on which to focus future research. Specifically, several  experts commented that research could help to identify the challenges  some of these groups face, as well as identify effective strategies that may  help these population groups obtain employment. (See app. III for a  complete list of responses to these items.)
In addition to population groups, experts also identified several  employment and training programs that they believe warrant research  attention. In particular, experts most often ranked three components of  the WIA program—WIA Adult, WIA Dislocated Workers, and WIA  Youth—as key to evaluate in ETA’s future research. Among those three,  WIA Adult was ranked the highest. (See app. III for a complete list of  experts’ responses on employment and training programs to evaluate.)
section_title: ETA’s Research Studies Generally Answered the Questions Posed, but Their Usefulness Was Limited
paragraphs: Research organizations and academic institutions with responsibility for  implementing ETA-funded research generally used methodologies  appropriate for the questions posed, but the studies were not always  useful for informing policy and practice. From January 2008 through  March 2010, ETA published 17 large research and evaluation reports—14  evaluations and 3 research reports—that each cost $1 million or more.  Four of these reports were designed to demonstrate what works and for  whom. Each of these four reports compared the employment-related  outcomes of individuals or regions who participated in training or  employment programs with the employment outcomes of similar  individuals who did not participate in the programs. The remaining 13  reports were descriptive and were not designed to assess program  outcomes.
In several studies we examined that cost $1 million or more, we found  that, for a number of reasons, ETA’s research studies were limited in their  usefulness and in their ability to inform policy and practices. For example,  in a study of the Prisoner Re-entry Initiative, shortcomings in the data  collection phase limited the strength of the findings and, as a result,  limited the study’s opportunity to influence policy directions. Among other  things, while the study provided information on employment-centered  opportunities for ex-offenders, the study relied on self-reported baseline  data, did not account for differences across sites where services were  received, lacked the capacity to record differences in the intensity of those  services, and researchers failed to ensure that data collectors were  properly trained.
In another study, researchers did not control for bias in selecting  participants, compromising their ability to draw conclusions about the  cause and effect of program outcomes. Authors of this study on the  Workforce Innovations in Regional Economic Development (WIRED)  initiative acknowledged that the study would be unable to attribute  outcomes to program services because it did not use random assignment  in selecting participating regions. We have previously criticized ETA for  failing to adequately provide for evaluating the effectiveness of its WIRED  initiative.
Moreover, some studies were limited due to observation periods that did  not match the needs of the studies’ objectives. For example, an evaluation  of an entrepreneurship training project was unable to assess the  effectiveness of the project in meeting its long-term goals of increasing  business ownership and self-sufficiency because the time frames for the  study were too short. In this study, data collection was limited to 18  months after participants were randomly assigned, a period far shorter  than the 60-month period recommended by experts. (See app. IV for  additional information on the methodological characteristics of these  studies.)
Experts generally agreed that ETA’s research had limited usefulness in  informing policy and practice. Over one-third of the 39 experts reported  that over the past 5 years, ETA’s research informed employment and  training policy and state and local practices to a little extent or not at all.  (See fig. 5.) Some experts commented that the design of these studies and  the length of time to complete them and disseminate results reduced their  usefulness. For example, many of the reports that we reviewed costing $1  million or more were multiyear projects that took, in most cases, about 3  to 5 years to complete. Some experts commented that the inclusion of  shorter-length studies may be useful in times of rapidly changing  economic conditions. At least one expert noted that some mixed-methods  studies would be useful—studies that would allow for short-term interim  findings that could facilitate changes in practice during the course of the  research study.
Members of our expert panel stressed the importance of ETA  incorporating varied methodological approaches into its future research  proposals to best position the agency to address key employment and  training issues. Twenty-seven of the 39 experts reported it was very  important that ETA evaluate its pilots and demonstrations. Twenty-three  reported that it was very important that more randomized experimental  research designs be integrated into ETA’s future research. (See fig. 6.)  While several experts noted that these randomized experiments will allow  ETA to identify the effectiveness of particular interventions or strategies,  at least one expert suggested that ETA should be strategic in choosing the  interventions it tests more rigorously, basing those decisions on what  appears most promising in preliminary studies.
Furthermore, 16 of the 39 experts also reported that it is very important  for ETA to consider including more quasi-experimental studies in the  future. As previously discussed, such studies would include designs that  compare outcomes between groups with similar characteristics, but do not  use random assignment. By including more quasi-experimental designs,  ETA may be able to better understand the link between services and  outcomes in those settings where random assignment is not possible,  ethical, or practical.
section_title: ETA Has Taken Steps to Improve Its Research Program, but Additional Actions Are Needed
subsections: section_title: Labor Has Taken Steps to Reform Its Research Program
paragraphs: Labor has taken several steps designed to improve the way it conducts  research, both at the department level and within ETA.
Department-level efforts. Labor has changed the organizational structure  of research within the department. In 2010, acknowledging the need for  better and more rigorous evaluations to inform its policy, Labor  established the Chief Evaluation Office to oversee the department’s  research and evaluation efforts. The office, which resides within the Office  of the Assistant Secretary for Policy, has no authority to direct research  within Labor’s agencies, according to officials. It does, however, manage  evaluations supported by funds from a departmentwide account, oversee  departmentwide evaluations, and provide consultation to Labor agencies,  including ETA. Specifically, the office is responsible for creating and  maintaining a comprehensive inventory of past, ongoing, and planned  evaluation activities within Labor and for ensuring that Labor’s evaluation  program and findings are transparent, credible, and accessible to the  public. In fiscal year 2010, the Chief Evaluation Office had an estimated  budget of $8.5 million, and two of its four staff were on board by the  beginning of fiscal year 2011.
ETA efforts. ETA has recently made changes to some of its research  practices—chief among them is the involvement of stakeholders and  outside experts in the research process. We previously criticized ETA for  failing to consistently involve a broad range of stakeholders, outside  experts, or the general public in deciding what areas of research it should  undertake. We recommended that ETA take steps to routinely involve  outside experts in the research agenda- setting process. For the  upcoming 2010 to 2015 research plan, ETA has awarded a grant to the  Heldrich Center at Rutgers University to convene an expert panel to help  inform the research plan. The center is expected to issue a report in May  2011 that outlines the panel’s recommendations for research areas to  include in the plan. In addition, ETA will work with other Labor agencies,  as well as the Departments of Education and Health and Human Services,  before finalizing its research agenda. Officials told us that they will also  solicit public comments before the research plan is finalized.
In addition to engaging stakeholders, ETA has also established a formal  research process. As we previously reported, ETA developed and  documented its research process in 2007. The agency’s actions were in  response to a request by the Office of Management and Budget (OMB) to  establish more formal policies and procedures to guide its research—a  request that came out of OMB’s concerns about the manner in which  ETA’s research was being carried out. Prior to 2007, ETA lacked a  documented research process, and its research was often conducted in an  ad hoc manner. ETA’s current research process identifies the steps,  activities, and time frames it uses to carry out its research. Figure 7  illustrates critical components of ETA’s 8-step research process.
ETA’s process contains several of the key elements identified by leading  organizations as important for guiding research activities. For example,  the process includes specific steps the agency should take to identify the  types of evaluations it will perform, as well as the administrative steps it  should take to develop evaluation plans and select the research projects to  fund. In addition, the process also specifies key events and time frames,  and provides for monitoring the implementation of the research. For  example, the process stipulates that ETA should alert OMB of research  reports that have not been approved for dissemination within 9 months of  being submitted and allows contractors to publicly release their research  reports within those same time frames.
section_title: Some Areas of ETA’s Research Program Merit Further Attention
paragraphs: Despite ETA’s efforts, more action is needed to improve its research  program. While ETA has taken steps to document its research process, its  process lacks specific details in some areas, creating ambiguities that  could undermine efforts to adhere to a formal process. For example, as we  previously reported, its process lacks clear criteria, such as a dollar  threshold or a particular methodological design feature, for determining  which projects require peer review. And while the process specifies the  actions project officers should take if reports are not released in a timely  manner, it does not specify the consequences for failing to do so. We  previously recommended that ETA establish more specific processes,  including time frames for disseminating research reports. ETA has taken  some action, such as revising the performance standards for project  officers to hold them accountable for meeting time frames, but these steps  do not fully satisfy the recommendation because the changes are not yet  reflected in the formal research process.
Moreover, ETA’s process is missing some critical elements that are needed  to ensure that the current improvements become routine practices.
Consulting with the Chief Evaluation Officer. ETA’s process lacks a  formal provision requiring consultation with the newly established  Chief Evaluation Officer at important points in the research process.  For example, it contains no provision for consulting with the Chief  Evaluation Officer when developing its annual list of research projects  or when determining how ETA will invest its research and evaluation  resources. Such consultation could help Labor better coordinate its  research and evaluation efforts and better leverage its research  funding. Moreover, the process contains no provision for involving the  Chief Evaluation Officer in the early stages of developing its research  projects. In the recent past, Labor officials told us that ETA has had  difficulty developing requests for research and evaluation proposals  that can pass OMB technical reviews. In particular, OMB has been  critical of ETA’s research designs because they failed to provide for  adequate sample size and appropriate methodologies that are needed  to obtain useful results. In addition, OMB has also expressed concerns  with ETA’s reliance on process evaluations rather than focusing on  outcomes. These difficulties have resulted in delays in the research  process. ETA has begun to consult with the Chief Evaluation Officer;  however, these consultations are not a routine component in the  formal process.
Setting the research agenda. ETA’s current process, as documented,  begins with phase two—selecting specific research studies—and  misses the important first step of setting the overall research agenda.  This first phase of the process should include the steps that ETA will  take to establish its research priorities and to update them on a regular  basis. It should also include provisions for ensuring critical issues are  considered and internal and external stakeholders are included in  developing the plan. Officials noted that they plan to incorporate the  agenda-setting phase into its formal process, but have not yet done so.  Setting the research agenda is key to ensuring that an appropriate mix  of studies is included in future research. Failing to make this phase part  of the formal process, including the specific steps to involve outside  stakeholders that are currently under way, may leave ETA with little  assurance that these efforts will continue in the future.
Beyond ETA’s process for conducting research, current research practices  fall short of ensuring research transparency and accountability—essential  elements of a sound research and evaluation program. The research  program has few, if any, safeguards to protect it from undue influence.  According to officials, at times in the past decade, many key research  decisions have been made outside of the office that is responsible for  research. For example, decisions about which research studies would and  would not be publicly released were made at the highest levels within  ETA, and the criteria used to make those decisions were unclear. Of the 34  reports that ETA released to the public in 2008, 20 had waited between 2  and 5 years to be approved for public release. Several reports that had  experienced long delays had relatively positive and potentially useful  findings for the workforce system, according to our analysis. Among the  studies delayed by almost 5 years was an evaluation of labor exchange  services in the one-stop system that found certain employment services to  be highly cost-effective in some situations. Another study, delayed for  about 3.5 years, was a compendium of past and ongoing experimental  studies of the workforce system, including early findings and  recommendations for future research.
In our previous report, we noted that ETA’s research and evaluation  center lacked a specific mechanism to insulate it from undue influence.  We reported that other federal agencies, such as the Department of  Education’s Institute of Education Sciences and the National Science  Foundation, engage advisory bodies in the research process. While not  without tradeoffs in terms of additional time and effort, such an approach  may serve to protect the research program from undue influence and  improve accountability. ETA is currently involving outside experts in  setting the research agenda for 2010 to 2015, but is not involving experts  more broadly on research policy and practices.
section_title: ETA Has Recently Included More Random Assignment Studies in Its Research Program
paragraphs: ETA has recently begun to include more rigorous studies in its ongoing  research. Of the 10 large, ongoing studies costing $2 million or more that  began during the period of our review, three—the WIA Gold Standard  Evaluation of the Adult and Dislocated Worker Programs, the Impact  Evaluation of the Young Parents Demonstration, and the Evaluation of  Project Growing America Through Entrepreneurship II (Project GATE  II)—use experimental design with random assignment, as recommended  by our experts. These ongoing studies—which range in cost from $2  million to nearly $23 million—have the potential to determine the  effectiveness of some of the program services. Table 4 outlines some key  characteristics of these three studies.
Experimental designs with random assignment are an important means to  understand whether various program components or services are  effective, but they are also often difficult to design and implement in real- world settings. For example, in doing evaluations of employment and  training programs, researchers often have difficulty in recruiting sample  sizes large enough to detect meaningful outcomes. Because employment  and training services may vary by location, and participants and their  socio-economic environments are diverse, researchers must find ways to  standardize procedures and treatment or service options. This often means  recruiting relatively large samples. However, studies can be intrusive,  often requiring program sites to change how they operate or to increase  the resources available to participants. As a result, recruiting sites and  sufficient numbers of participants may be difficult.
Some of ETA’s ongoing research studies face challenges in recruiting  sample sizes large enough to meet the studies’ objectives. For example,  based on an OMB review, it was determined that the sample size for the  Impact Evaluation of the Young Parents Demonstration had to be much  larger in order to be able to assess the effectiveness of the program. At  that time, ETA had already awarded two phases of grants. After consulting  with the new Chief Evaluation Officer, ETA changed the number of  participants required for the third phase from 100 to 400 to obtain a  sample large enough to address OMB’s concerns and provide reliable  estimates. However, grantees found it difficult to recruit even the 100  participants in the smaller sample, and it remains unclear whether they  will be able to recruit all of the needed participants for the expanded  design.
subsections: section_title: The WIA Gold Standard Evaluation of the Adult and Dislocated Worker Programs
paragraphs: The WIA Gold Standard Evaluation illustrates ETA’s difficulties in  planning and executing large-scale, rigorous random assignment studies.  WIA required that the Secretary of Labor conduct at least one multi-site  control-group evaluation of the services and programs under WIA by the  end of fiscal year 2005. ETA, however, delayed executing such a study,  finally soliciting proposals in November 2007 and awarding the contract in  June 2008. The contractor submitted the initial design report in January  2009 and provided ETA with design revisions in May 2010. Officials tell us  researchers will soon begin randomly assigning participants. ETA expects  to receive the first report (on implementation) during the winter of 2012- 2013 and the final report in 2015—10 years later than the WIA-mandated  time frame.
An OMB-selected panel of government experts—a technical working  group composed of experts chosen by ETA, the evaluation contractor, and  OMB staff—reviewed the original design for this study. Reviewers agreed  the design contained many strengths, including  the selection of an experimental design and a net impact approach;  the addition of a process or implementation study to evaluate  differences among sites and other implementation and data collection  issues;  the use of administrative and survey data;  the collection of information on services received by participants in the  control group; and  the collection of a wide range of outcome data for participants.
However, reviewers raised several concerns regarding the design. For  example, they were skeptical that the researchers would be able to obtain  a sufficiently large and representative sample to draw meaningful  conclusions about the effectiveness of the national workforce system. In  order to maximize participation, officials told us that the Assistant  Secretary of ETA made personal phone calls to all selected sites to  emphasize the importance of the study, offered an open door policy to site  officials to discuss issues, and followed up with an appreciation letter.  Furthermore, ETA required the evaluation contractor to provide  reimbursement payments to each site to offset implementation costs.
Reviewers also had several other concerns regarding which groups would  be included in the study and which groups would not. For example, some  experts raised concerns about getting accurate information on the youth  program because of the large, one-time infusion of funds the program  received from the American Recovery and Reinvestment Act of 2009.  Reviewers were further concerned about the appropriateness of the  evaluation objectives, the adequacy of steps taken to account for the effect  of variation in services across sites on evaluation outcomes, and the  external validity or generalizability of the study. In order to address these  concerns, ETA made substantial adjustments to the original design.  Specifically, ETA officials told us that based on an agreement with OMB,  they instructed the contractor to drop the youth component from the  evaluation and to focus only on the Adult and Dislocated Worker  programs. While we received information on the new design and time  frames for the WIA Gold Standard Evaluation, a finalized design plan is not  yet available. According to officials, a finalized design is being prepared  and will be available in June 2011.
section_title: ETA Has Improved the Availability of Its Research but There Are Opportunities to Improve Its Search Page and Dissemination Methods
subsections: section_title: ETA Has Improved the Timeliness of Its Disseminated Research
paragraphs: ETA has recently improved the timeliness with which it disseminates its  research reports. In our last review in January 2010, we found that 20 of  the 34 reports that ETA disseminated in 2008 had been waiting 2 to 5 years  to be publicly released. The 34 research reports published by ETA in 2008  took, on average, 804 days from the time the report was submitted to ETA  until the time it was posted to ETA’s research database. By contrast, from  2009 through the first quarter of 2010, the average time between  submission and public release was 76 days, which represents a more than  90 percent improvement in dissemination time compared with 2008.  Additionally, there were no research reports in 2009 that were delayed for  more than 6 months. Further, the average time to dissemination improved  significantly even when we excluded such outliers as the 20 research  reports that were delayed for 2 years or more. Without these outliers,  average time to dissemination for reports in 2008 was 100 days, indicating  that time to dissemination in 2009 through the first quarter of 2010 still  improved by 24 percent.
section_title: ETA Has Improved Its Research Database but Lacks Plans for Assessing the Usability of Its Search Page
paragraphs: In 2010, ETA updated its online, Web-based search page in order to  improve the usability of its research database—the primary tool for  making ETA research available to policymakers and the general public.  Officials told us that ETA’s old Web-based search page was so error-prone  and difficult to use that they opted to substitute it with one that had not  yet completed internal testing. Our review of the old Web-based search  page confirmed that it had serious limitations and did not consistently  return the same results. For example, when we searched the database by  title for a known ETA research report titled Registered Apprenticeship,  we successfully retrieved that report once. One month later, when we  entered the exact same search terms, we were unable to retrieve the  report. (For a more complete description of our analysis of ETA’s search  capability, see app. II.)
In our review of the updated Web-based search page, we found that the  updates make the research database more useable. Labor officials told us  they have taken other steps, as well, in efforts to improve its Web-based  search page. For example, they have developed a project plan that  articulates the steps Labor will take to update ETA’s Web-based search  page. In addition, they have assigned a database administrator whose  responsibilities include performing daily quality control spot checks in  order to monitor performance and address technical problems.
Although these changes have the potential to improve the usability of  ETA’s database, Labor has not developed a formal plan for assessing the  overall effectiveness of its Web-based search page, including user  satisfaction. Labor has made a number of changes to the way the page  operates, but it has not provided users with tips on how to use the search  functions, even though it is an industry standard to do so. Even skilled  users who were familiar with the old Web-based search page may need  guidance on the exact meaning of new terms and functions now available  on the new page. For example, the old Web-based search page gave users  the option of searching by “key word,” which is no longer an option in the  new page. Instead, “key word” searches have been replaced with a variety  of other options, including the ability to search the full text or abstract of a  research report. However, there is no guidance on the Web site on how to  use these new search options. Industry best practices suggest that a Web  site evaluation plan that incorporates data from routine reviews of Web  site performance and that assesses user satisfaction can help agencies  ensure the usability of their Web sites. ETA currently has no plans to do  such assessments.
section_title: ETA Uses Various Methods to Disseminate Research, but Experts Suggest Additional Methods
paragraphs: At present, ETA’s research database is the primary method that ETA uses  to make its research reports publicly available, according to officials. In  order to call attention to new reports available in that database, ETA sends  a Training and Employment Notice, also commonly known as a TEN, to an  e-mail list of the more than 40,000 subscribers who have signed up to  receive them. ETA’s research process specifies that for each new research  report that is approved for dissemination, ETA must draft a TEN and an  abstract before it is posted to ETA’s Web site. Beyond posting reports to  its database, ETA also distributes hard copies of some of its research  reports.
In addition to electronic distribution, ETA also organizes various  presentations to disseminate its research findings. These presentations,  however, are done on an ad hoc basis. As mentioned in our prior report,  ETA hosted a research conference in 2009 to present some of its research  findings, renewing a practice that had been discontinued in 2003. As ETA  looks to the future, officials tell us they will plan and organize similar  research conferences as resources permit. In addition to these research  conferences, ETA’s regional offices occasionally hold smaller, regional  conferences as well. Beyond these formal conferences, ETA also hosts an  internal briefing series at Labor headquarters where research contractors  present their findings to various officials. For each of these briefings, ETA  has a list of stakeholders that it invites, including various Labor officials,  outside agency officials, congressional staff, and other outside  stakeholders.
Experts who participated in our virtual panel provided their views on the  effectiveness of different methods for disseminating research reports, and  several of those rated more highly are methods currently employed by  ETA. (See fig. 8.)
Most of the experts (30 of the 39 respondents) in our panel reported that  using e-mail notifications, a searchable database of ETA papers, and  briefings at ETA for external audiences (including stakeholders and  policymakers) would be very effective or extremely effective approaches  for disseminating research. In addition, a majority of the experts (26 of the  39 respondents) in our panel reported that publishing one-page summaries  of research findings, not currently done by ETA, would be very or  extremely effective.
section_title: Conclusions
paragraphs: ETA plays an important role in developing workforce policies and helping  to identify the most effective and efficient ways to train and employ  workers for jobs in the twenty-first century. With the current economic  crisis and high unemployment rates, ETA’s role has become even more  critical. The agency has made some improvements in its research program,  even since our last review a year ago. But officials can do more to ensure  that the progress continues in the years to come.
ETA has taken a major step forward in establishing a formal research  process—one that documents most actions that must be taken in the life  cycle of a research or evaluation project. But, it is missing some key  elements that could help ensure the continuation of current practices.  While ETA is currently using outside advisory bodies to help it establish its  research agenda, the formal process does not include the agenda-setting  phase. Officials tell us they have plans to incorporate this phase in the  future, and we urge them to do so. Without a formalized agenda- setting  phase, ETA may miss opportunities to ensure that its research agenda  addresses the most critical employment and training issues and that  outside stakeholders are routinely involved. Moreover, ETA’s process has  not formalized the now ad hoc advisory role of the Chief Evaluation  Officer. Absent the routine involvement of the Chief Evaluation Officer at  key steps in the process, ETA may find it difficult to ensure that research  proposals are asking the right questions, are methodologically sound, and  that they can quickly pass critical OMB reviews.
ETA’s research findings are now available to the public on its Web site in  far less time than it took in 2008. Despite this clear improvement, ETA has  not taken the necessary steps to ensure that research products remain  readily available to the public. The decision regarding what and when to  make research publicly available is left in the hands of too few, and the  process lacks needed safeguards to ensure transparency and  accountability. Absent safeguards, key research decisions may again be  made in ways that harm the credibility of the program and prevent  important research findings from being used to inform policy and practice.
ETA’s Web-based search page is the primary means ETA uses to make the  research studies it funds readily available to the public. And, while ETA  has improved the functionality of its Web site, no effort has been made to  ensure that the problems that plagued the system in the past do not recur.  Absent such efforts, ETA will have little assurance that its research  findings are actually available to users.
section_title: Recommendations for Executive Action
paragraphs: To improve ETA’s research program, we recommend that the Secretary of  Labor require ETA to take the following three actions:    Formally incorporate into its research process the routine involvement  of the Chief Evaluation Officer at key milestones, including at the  development of ETA’s annual research agenda and spending priorities,  as well as at the early stages of developing specific research projects.
Develop a mechanism to enhance the transparency and accountability  of ETA’s research program. For example, such a mechanism might  include involving advisory bodies or other entities outside ETA, in  efforts to develop ETA’s research policies and processes.
Develop a formal plan for ensuring that ETA’s research products are  easily accessible to stakeholders and to the general public through its  searchable database. Such a plan could involve requiring Labor to  assess the overall effectiveness of its Web-based search page, including  user satisfaction with search features.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Department of Labor for review  and comment. Labor provided written comments, which are reproduced in  appendix VII. In addition, ETA provided technical comments, which we  incorporated where appropriate.
In its response, Labor generally agreed with our findings and all of our  recommendations, noting its ongoing efforts in support of the  recommendations.
Regarding our recommendation to formally incorporate into its  research process the routine involvement of the Chief Evaluation  Officer at key research milestones, Labor noted that it is currently  taking steps to do so. Officials reported that they have worked closely  with this office in various aspects of its research, including discussing  research, demonstration projects, and evaluations in the early stages of  development and plans to continue this collaboration in the future.  However, ETA’s comments did not discuss plans to update its  documentation on the formal research process.  We found in our  review that involving the Chief Evaluation Officer was not an official  component of ETA’s documented research process, and it occurred on  an ad hoc basis. As ETA moves forward, we urge the agency to modify  its current research process and document the involvement of the  Chief Evaluation Officer at critical research milestones.
Regarding our recommendation for ETA to develop a mechanism to  enhance the transparency and accountability of its research program,  officials cited several steps they are taking to improve the program,  including involving outside experts in the development of their 5-year  research plan and establishing advisory and peer review groups to  review major evaluations. While officials note they plan to engage  outside experts in broader research policies and processes, we  encourage ETA to formalize this involvement. Moreover, we encourage  ETA to continue to move forward in its efforts to further clarify  components of its research process that are not well defined, including,  for example, the criteria to be used when deciding when a peer review  should be performed.
Regarding our recommendation to develop a formal plan to ensure that  disseminated research is easily accessible to stakeholders and the  general public, officials cited specific steps the agency has taken to  improve its Web-based research database. While these actions are a  step in the right direction, we believe that it is still important for Labor  to develop a formal and comprehensive plan to ensure that  disseminated research continues to be accessible to the public.
Furthermore, Labor expressed concerns about how we characterized the  agency’s budget for pilots, demonstrations, and research. Recognizing  these concerns, we made changes to the report to better capture the  amount of funding ETA has available for research.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the appropriate  congressional committees, the Secretary of Labor, and other interested  parties. The report will also be available at no charge on GAO’s Web site at  http://www.gao.gov.
If you or your staffs have any questions about this report, please contact  me at (202) 512-7215 or scottg@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made key contributions to this report are  listed in appendix VIII.
section_title: Appendix I: Status of Prior GAO Recommendations to the Department of Labor, as of January 2011
paragraphs: Appendix I: Status of Prior GAO  Recommendations to the Department of  Labor, as of January 2011 Department of Labor’s response   The Department of Labor (Labor) does not agree with this  recommendation as written. According to Labor officials, the  Administrator of OPDR currently reports to the Deputy  Assistant Secretary, not directly to ETA’s Assistant  Secretary. However, Labor officials acknowledge that  important functions such as research and evaluation should  not have too many intermediary reporting layers. To  facilitate communication, officials further noted that the  OPDR Administrator, the Deputy Assistant Secretary, and  the Chief Evaluation Officer meet on a monthly basis with  the Assistant Secretary to discuss evaluations.
Labor agrees with this recommendation, but authority to  make key decisions still resides with the Office of the  Assistant Secretary for ETA. OPDR currently provides  recommendations to this office regarding plans for  conducting and disseminating research. In an effort to  improve evaluations departmentwide, the Secretary of Labor  recently established the Chief Evaluation Office to monitor  evaluation efforts across the department. OPDR has begun  to work informally with the Chief Evaluation Officer and the  Chief Economist to design and implement research and  evaluation projects.
Labor agrees with this recommendation. ETA reports that it  has taken some steps to establish more specific processes  regarding dissemination of research, citing changes in  performance standards for project officers. However, our  recommendation would make broader changes to their  research process and no such changes are reflected in the  documents the agency provided.
Labor’s actions do not  completely satisfy  recommendation  … create an information system to track  research projects at all phases to ensure  timely completion and dissemination.
Labor agrees with this recommendation. Officials report that  they have begun working on a centralized, electronic  tracking system for its research projects. However, the work  is still under way and no time frames have been provided for  its completion. Currently, OPDR uses an Excel document to  keep inventory of all research, demonstration, and  evaluation projects.
Labor’s actions do not  completely satisfy  recommendation  … instruct ETA’s research and  evaluation center to develop processes  to routinely involve outside experts in  setting its research agenda and to the  extent required, do so consistent with the  Federal Advisory Committee Act.
Labor agrees with this recommendation. OPDR has taken  steps to engage outside experts in setting its 5-year  research plan for 2011 and collaborate with the research  and evaluation centers of other federal agencies, such as  the Departments of Education and Health and Human  Services. OPDR also plans to convene an expert panel,  solicit public comments, and incorporate feedback from its  2009 Reemployment Research Conference and its 2010  ETA Reemployment Summit. However, despite these  current efforts, OPDR has not formally incorporated them in  its standard research process.
section_title: Appendix II: Scope and Methodology
paragraphs: We were asked to review the Employment and Training Administration’s  (ETA) research program to better understand its approach to conducting  and disseminating research. Specifically, we answered the following  research questions: (1) To what extent do ETA’s research priorities reflect  key national employment and training issues and how useful were the  studies funded under them? (2) What steps has ETA taken to improve its  research program? (3) How has ETA improved, if at all, the availability of  its research since our last review in January 2010 and what other steps  could ETA take to further ensure its research findings are readily  available?
To answer our research questions, we convened a virtual panel using a  modified Delphi technique to obtain selected employment and training  experts’ opinions on ETA’s research priorities and dissemination methods.  We also visited two workforce agencies in Pennsylvania and Virginia that  are implementing two of ETA’s ongoing research studies to learn about  implementation issues and how research is being conducted. In addition,  we reviewed 58 ETA-funded research and evaluation reports disseminated  between January 2008 and March 2010 and assessed the methodological  soundness of completed studies that cost $1 million or more. We also  reviewed ETA’s ongoing studies that cost $2 million or more. To determine  the availability of ETA’s research, we measured the time between when  the final version of a research report was submitted to ETA’s Office of  Policy Development and Research (ODPR) and when it was posted on  ETA’s Web site. We also conducted a series of systematic searches to test  the reliability of ETA’s research database. Furthermore, we interviewed  Department of Labor (Labor) and ETA officials to better understand ETA’s  research capacity, processes, and the use of research findings to inform  policy and practice. Lastly, we reviewed relevant agency documents and  policies, as well as relevant federal laws.
subsections: section_title: Web-Based Expert Panel
paragraphs: We convened a nongeneralizable Web-based virtual panel of 41  employment and training experts to obtain their opinions on ETA’s  research priorities and dissemination methods. We employed a modified  version of the Delphi method to organize and gather these experts’  opinions. To encourage participation by our experts, we promised that  responses would not be individually identifiable and that results would  generally be provided in summary form. To select the panel, we asked  several employment and training experts, on the basis of their experience  and expertise, to identify other experts who were knowledgeable of ETA  and the research it conducts and disseminates. After receiving  nominations from experts, we reviewed the list to ensure that it reflected a  range of perspectives and backgrounds, including academics, researchers,  and consultants.
Our Delphi process entailed two survey phases. (See app. V for a copy of  our phase I and phase II questionnaires.) In phase I, which ran from June  22, 2010, to August 9, 2010, we asked the panel to respond to five open- ended questions about ETA’s research priorities and dissemination  methods. We developed these questions based on our study objectives and  pretested them with four experts by phone to ensure the questionnaire  was clear, unbiased, and did not place an undue burden on respondents.  All relevant changes were made before we deployed the first Web-based  questionnaire to experts.
After the experts completed the open-ended questions in the first  questionnaire, we performed a content analysis of the responses in order  to identify the most important issues raised by our experts. Two members  of our team categorized experts’ responses to each of the questions. Any  disagreements were discussed until consensus was reached. Thirty-six of  the 41 panelists selected completed phase I of the survey (about an 88  percent response rate). Those that did not complete phase I were allowed  to participate in phase II. (For a list of experts who participated in phase I  and phase II, see app. VI.)
The experts’ responses to phase I were used to create the questions for  phase II. In phase II, we gathered more specific information on ETA’s  research and dissemination practices. Phase II, which ran from October  29, 2010, to December 14, 2010, consisted of 16 follow-up questions where  panelists were asked to either rank or rate the responses from phase I. We  pretested the questionnaire for the second phase with three experts to  ensure the clarity of the instrument. We conducted two of our expert  pretests in-person and one by phone. Thirty-nine of the 41 experts  completed phase II (about a 95 percent response rate).
section_title: Site Visits to Workforce Agencies Implementing ETA-Funded Research Studies
paragraphs: To further enhance our understanding of how ETA conducts its research,  we visited two workforce agencies that are implementing ETA’s ongoing  research studies. First, we visited the Lancaster County Workforce  Investment Board in Lancaster, Pa., which received funding from ETA to  implement the Young Parents Demonstration project. This project  provides educational and occupational skills training to promote  employment and economic self-sufficiency for mothers, fathers, and  expectant mothers ages 16 to 24. Second, we visited the Northern Virginia  Workforce Investment Board in Falls Church, Va., which received funding  from ETA to implement the second round of the Project Growing America  Through Entrepreneurship, also referred to as Project GATE II. This grant  helps dislocated workers aged 50 and over obtain information, classroom  training, one-to-one technical assistance, counseling, and financial  assistance to establish new businesses in order to help them start and  sustain successful self-employment.
We selected these workforce agencies because they were identified by  ETA as having active research projects in the implementation stage. These  sites also required minimum travel expenditure. During our site visits, we  toured each workforce agencies’ facilities and used a semistructured  interview protocol to interview the project director and staff about their  role and responsibilities, the extent to which they communicate with ETA,  and whether or not they face challenges with regards to implementation.  At the Lancaster County site, we participated in an informal on-site lunch  forum where local community programs that the agency partners with  talked with us about their collaboration with the program. At the Northern  Virginia GATE II site, we observed a focus group operated by the program  to facilitate information-sharing among participants.
After our site visits, we conducted phone interviews with the contractors  that received funding from ETA to evaluate the outcomes of two research  projects. Specifically, we interviewed the Urban Institute, which evaluates  the Young Parents Demonstration project, and IMPAQ International,  which evaluates Project GATE II. Both projects include an experimental  component with control and comparison groups to determine the effects  of program interventions on participants. During our interviews we used a  semistructured questionnaire and asked questions to better understand  their roles and responsibilities for the project, the extent to which they  communicate with ETA, and whether or not they experience  methodological and implementation challenges.
section_title: Analysis of Methodological Characteristics of ETA
paragraphs: We reviewed the 58 research and evaluation reports that ETA  disseminated between January 2008 and March 2010 and assessed the  methodological soundness of 11 completed studies that cost $1 million or  more. In addition, we reviewed 10 ongoing studies costing $2 million or  more to determine if research practices or the soundness of research  designs had changed over time. We categorized the 58 studies  disseminated between January 2008 and March 2010 by study type, cost,  and research area. For the larger studies costing $1 million or more, we  analyzed key characteristics including design features, scope,  generalizability, and the appropriateness of analytical approaches and  statistical procedures. These studies were analyzed independently by two  analysts and the agreement between their ratings was 100 percent. (For  results of this analysis, see app. VI.)
section_title: Analysis of the Timeliness and Effectiveness of ETA’s Dissemination Activities
paragraphs: To evaluate the availability of ETA’s research, we measured the time  between when the final version of a research report was submitted to  ODPR and when it was posted on ETA’s Web site. Specifically, we  measured the dissemination time frames for reports posted in 2008 and  compared that with the dissemination time frames for reports issued  between January 2009 through March 2010. In addition, we conducted a  series of systematic searches to test the reliability of ETA’s Web-based  research database. To perform our searches, we selected a random sample  of 30 reports from the 312 reports available on ETA’s research database at  the time of our review. Specifically, we tested a variety of search functions  available at the time of our review to determine the extent to which  research reports could be easily retrieved on ETA’s research database.  These functions included searches by title, keywords, author, and/or dates.  We classified a report as retrievable if it appeared anywhere in our search  results. We conducted our initial searches between June 30, 2010, and July  6, 2010. A second round of searches was conducted between August 6,  2010, and August 10, 2010. Further, we interviewed Labor and ETA  officials to learn more about the search capabilities of ETA’s research  database and the processes used to address errors and implement  changes. Finally, we interviewed officials to gather information about  ETA’s dissemination methods, including its current techniques and future  plans for disseminating research reports.
section_title: Interviews with Labor and ETA Officials
paragraphs: To better understand the agency’s research capacity, we interviewed ETA  officials and reviewed relevant agency and budget documentation.  Similarly, to obtain information on ETA’s research process and how  research findings are used to inform employment and training policy and  practice, we interviewed officials and reviewed agency documentation,  including relevant policies and procedures that guide ETA’s research. We  also reviewed relevant federal laws.
We conducted this performance audit from March 2009 through March  2011 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix III: The Panel’s Ratings of Key Employment and Training Issues, Populations, and Programs That ETA Should Address in Its Future Research
paragraphs: In our Delphi phase II Web-based questionnaire, we asked the panel of  experts to rate and rank the key employment and training issues,  populations, and programs that ETA should address in its future research.  These issues were identified by the panel during phase I. For our analysis,  we calculated basic descriptive statistics on these issues, which are  presented in tables 5 through 7.
section_title: Appendix IV: Characteristics of Research Studies Disseminated between January 2009 and March 2010 That Cost $1 Million or More
paragraphs: Other (comparison  v. treatment group  using propensity  score matching)
Increasing the  labor market  participation of  underutilized  populations  characteristics  collected before  the intervention,  but these were not  used to make  comparisons)
Integration of the  workforce and  regional  economic  development  network analysis)
Integration of the  workforce and  regional  economic  development  network analysis )
Experimental  (randomized  control trials)
Increasing the  labor market  participation of  underutilized  populations  collected on  characteristics  upon entry and  outcome  characteristics  collected after  completion)
section_title: Appendix V: Delphi Phase I and Phase II Questionnaires
section_title: Appendix VI: Experts Who Agreed to Participate in GAO’s Delphi Panel
paragraphs: Trachtenberg School of Public Policy and Public  Administration, George Washington University  Abt Associates Inc.
Mathematica Policy Research, Inc.
W.E. Upjohn Institute for Employment Research   National Bureau of Economic Research  Robert M. LaFollette School of Public Affairs, University  of Wisconsin-Madison  W.E. Upjohn Institute for Employment Research  Abt Associates Inc.
John J. Heldrich Center for Workforce Development,  Rutgers, The State University of New Jersey  Department of Economics, College of Arts and Sciences,  American University  Mathematica Policy Research, Inc.
Department of Economics, University of Missouri- Columbia  Humphrey School of Public Affairs, University of  Minnesota  Minnesota Department of Employment and Economic  Development  W.E. Upjohn Institute for Employment Research  Institute for Policy Studies, Johns Hopkins University  Center for Law and Social Policy  Peterson Institute for International Economics  Mathematica Policy Research, Inc.
section_title: Appendix VII: Comments from the Department of Labor
section_title: Appendix VIII: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the contact listed above, Dianne Blank, Assistant Director,  and Kathleen White, analyst-in-charge, managed all phases of the  engagement. Ashanta Williams assisted in managing many aspects of the  work and was responsible for final report preparation. Lucas Alvarez and  Benjamin Collins made significant contributions to all aspects of this  report. In addition, Amanda Miller assisted with study and questionnaire  design; Joanna Chan performed the data analysis; Stephanie Shipman  advised on evaluation approaches; James Bennett provided graphics  assistance; David Chrisinger provided writing assistance; Alex Galuten and  Sheila McCoy provided legal support; and Sheranda Campbell and Ryan  Siegel verified our findings.
section_title: GAO Related Products
paragraphs: Program Evaluation: Experienced Agencies Follow a Similar Model For  Prioritizing Research. GAO-11-176. Washington, D.C.: January 14, 2011.
Employment and Training Administration: Increased Authority and  Accountability Could Improve Research Program. GAO-10-243.  Washington, D.C.: January 29, 2010.
Workforce Investment Act: Labor Has Made Progress in Addressing  Areas of Concern, but More Focus Needed on Understanding What Works  and What Doesn’t. GAO-09-396T. Washington, D.C.: February 26, 2009.
Employment and Training Program Grants: Evaluating Impacts and  Enhanced Monitoring Would Improve Accountability. GAO-08-486.  Washington, D.C.: May 7, 2008.
Federal Research: Policies Guiding the Dissemination of Scientific  Research from Selected Agencies Should Be Clarified and Better  Communicated. GAO-07-653. Washington, D.C.: May 17, 2007.
Data Quality: Expanded Use of Key Dissemination Practices Would  Further Safeguard the Integrity of Federal Statistical Data. GAO-06-607.  Washington, D.C.: May 31, 2006.
Workforce Investment Act: Substantial Funds Are Used for Training, but  Little Is Known Nationally about Training Outcomes. GAO-05-650.  Washington, D.C.: June 29, 2005.
Program Evaluation: An Evaluation Culture and Collaborative  Partnerships Help Build Agency Capacity. GAO-03-454. Washington,  D.C.: May 2, 2003.
Workforce Investment Act: Improvements Needed in Performance  Measures to Provide a More Accurate Picture of WIA’s Effectiveness.  GAO-02-275. Washington, D.C.: February 1, 2002.