Employment and Training Administration: More Actions Needed to Improve Transparency and Accountability of Its Research Program
Released: 2011-04-14T00:00:00

Letter

Background

WIA sets forth various requirements for the Secretary of Labor relating to research and evaluation of federally funded employment-related programs and activities. The law calls upon the Secretary of Labor to publish in the Federal Register every 2 years a plan that describes its pilot, demonstration, and research priorities for the next 5 years regarding employment and training. Specifically, WIA requires the Secretary to develop the research plan after consulting with states, localities, and send the plan to the appropriate committees of Congress; and take into account such factors as the likelihood that the results of the projects will be useful to policymakers and stakeholders in addressing employment and training problems.

Within ETA, OPDR’s Division of Research and Evaluation plans, conducts, and disseminates employment and training-related research and evaluations. Nearly all of the agency’s research and evaluation studies are conducted under contract; these contractors represent a range of research organizations and academic institutions. Furthermore, OPDR plans and conducts its research and evaluation activities in consultation with ETA’s program offices, such as the Office of Workforce Investment and the Office of Trade Adjustment Assistance.

ETA’s research and evaluation funding is divided into two separate budget line items: Pilots, demonstrations, and research. Efforts in this category are focused on developing and testing new ways to approach problems and to deliver services. Under WIA, pilots and demonstrations shall be carried out “for the purpose of developing and implementing techniques and approaches, and demonstrating the effectiveness of specialized methods, in addressing employment and training needs.” WIA also states that the Secretary shall “carry out research projects that will contribute to the solution of employment and training problems in the United States.”

Evaluations. Efforts in this category are focused on continuing evaluations of certain programs and activities carried out under WIA. These evaluations must address the effectiveness of these programs and activities carried out under WIA in relation to their cost; the effectiveness of the performance measures relating to these programs and activities; the effectiveness of the structure and mechanisms for delivery of services through these programs and activities; the impact of the programs and activities on the community and participants involved, and on related programs and activities; the extent to which such programs and activities meet the needs of various demographic groups; and such other factors as may be appropriate.

In program year 2010, ETA’s combined budget appropriation for conducting evaluations and pilots, demonstrations, and research was about $103 million—or nearly $34 million above what the agency requested. (See fig. 1.) About $84 million of the 2010 funds were designated by the Congress for specific projects, including $30 million for Transitional Jobs activities for ex-offenders, and another $5.5 million for competitive grants addressing the employment and training needs of young parents. According to agency documents, in 2008 and 2009, the Congress similarly increased ETA’s requested budget for pilots, demonstrations, and research, at the same time specifically designating how the majority of those funds would be used, including $4.9 million in 2008 and $5 million in 2009 for the young parents’ demonstration.

Key Elements of Sound Research and Evaluation Programs

While there is no single or ideal way for government agencies to conduct research, several leading national organizations have developed guidelines that identify key elements that promote a sound research program. These guidelines identify five elements as key: agency resources, professional competence, independence, evaluation policies and procedures, and evaluation plans.

Resources. Research should be supported through stable, continuous funding sources and through special one-time funds for evaluation projects of interest to executive branch and congressional policymakers.

Professional competence. Research should be performed by professionals with appropriate training and experience for the evaluation activity (such as performing a study, planning an evaluation agenda, reviewing evaluation results, or performing a statistical analysis).

Independence. Although the heads of federal agencies and their component organizations should participate in establishing evaluation agendas, budgets, schedules, and priorities, the independence of evaluators must be maintained with respect to the design, conduct, and results of their evaluation studies.

Evaluation policy and procedures. Each federal agency and its evaluation centers should publish policies and procedures and adopt quality standards to guide evaluations within its purview. Such policies and procedures should identify the kinds of evaluations to be performed and the criteria and administrative steps for developing evaluation plans and setting priorities, including selecting evaluation approaches to use, consulting experts, ensuring evaluation product quality, and publishing reports.

Evaluation plans. Each federal agency should require its major program components to prepare annual and multiyear evaluation plans and to update these plans annually. The planning should take into account the need for evaluation results to inform program budgeting, reauthorization, agency strategic plans, program management, and responses to critical issues concerning program effectiveness. These plans should include an appropriate mix of short- and long-term studies to produce results for short- or long-term policy or management decisions. To the extent practical, the plans should be developed in consultation with program stakeholders.

Furthermore, leading organizations, including the American Evaluation Association and the National Academy of Sciences, emphasize the need for research programs to establish specific policies and procedures to guide research activities. Based on several key elements identified by these organizations, we developed a framework comprised of five phases—agenda setting, selecting research, designing research, conducting research, and disseminating research results. (See fig. 2.)

Agenda setting. Agencies should establish a structured process for developing their research priorities. The process should identify how agencies set research priority areas and provide for updating the areas on a regular basis. The process should also allow for the consideration of critical issues and state how internal and external stakeholders will be included in developing the plan.

Selecting research. At this phase, the process should identify how the research program’s staff identifies and selects studies to fund, including the criteria it uses to make those decisions. Steps might describe how the staff assembles a list of potential studies, works with internal program offices, and makes final decisions.

Designing research. During the design phase, the process should identify steps taken to select appropriate research approaches and methods and the safeguards in place to ensure appropriate tradeoffs are made between what is desirable and what is practical and between the relative strengths and weaknesses of different methods.

Conducting research. At this stage, the process should include policies and procedures to guide the conduct of research. The process should ensure that key events, activities, and time frames are specified and that knowledgeable staff in the sponsoring agency monitor the implementation of the research.

Disseminating research. This process should describe how research findings are made available to the public and disseminated to all potential users. These dissemination methods should include safeguards to ensure research findings are disseminated in a timely manner and are accessible through the Internet with user-friendly search and retrieval technologies.

Research Terminology in This Report

In this report, we use several technical terms in describing ETA’s research designs and study characteristics. (See table 1.)

ETA’s Research Areas Generally Reflect Key Issues, but Some Studies Are of Limited Usefulness

Experts Thought ETA’s 2007 to 2012 Research Plan Reflected Key Areas, but They Also Suggested New Ones for Future Research

Our expert panel generally considered ETA’s research areas to be the right ones for the period the research plan covered. About three-fourths of the panel members reported that ETA’s 2007 to 2012 research agenda reflected key employment and training issues to at least a moderate extent. However, a few experts commented that some of ETA’s research areas may be too broad and lack specificity.

The areas in ETA’s 2007 to 2012 research plan covered a range of issues, from job training to postsecondary education. Table 2 illustrates the scope of ETA’s research areas.

With regard to the specific studies within these research areas, ETA invested most of its research and evaluation resources in work that focused on increasing the labor market participation of underutilized workers and on UI. Of the estimated $96 million that supported the 58 research reports published between January 2008 and March 2010, more than half—about $56 million—funded research that addressed these two research areas. Other areas received far less funding. For example, funding for studies addressing the methods of expanding U.S. workforce skills and using state-level administrative data to measure progress and outcomes accounted for about $6.5 million, or about 6.7 percent of the cost of studies published during the period we examined. (See table 3.) Overall, the individual studies that ETA funded addressed a wide variety of issues and ranged in cost from about $15,000 to a high of about $22 million.

In addition to the research areas covered in ETA’s 2007 to 2012 research plan, experts from our virtual panel suggested that ETA incorporate additional research areas in its future research agenda. Of the research areas identified, over half of our experts (28 of 39) ranked the identification of employment and training approaches that work, and for whom, as one of the top areas that ETA’s future research should address. (See fig. 3.) Without such focus, experts commented that it will be difficult to know how to improve the nation’s workforce system. Other issues ranked at the top by experts included research on job creation strategies and the impact of long-term and short-term training. (See app. III for more information on issue-area rankings.)

In addition to identifying overall employment and training areas, including issues related to UI, experts also identified specific aspects of the UI system that could be examined in ETA’s future research. In particular, most experts (34 of 39 respondents) reported that it would be at least moderately important, in the future, for ETA to research the linkage between UI and employment and safety net programs, such as Temporary Assistance for Needy Families or the Supplemental Nutrition Assistance Program. (See fig. 4.) This area of research may be particularly important given the role that these programs play in supporting individuals during economic downturns. In addition, many experts (24 of 39 respondents) mentioned that ETA should make the examination of the incentives and disincentives in the UI system a research priority, given the challenge of supporting unemployed workers during difficult economic times, while promoting self-sufficiency through employment.

Experts also reported that it is important to fund research on what works for selected population groups. Of the population groups identified, the experts on our virtual panel most often ranked the long-term unemployed, economically disadvantaged workers, and adults with low basic skills as the top populations on which to focus future research. Specifically, several experts commented that research could help to identify the challenges some of these groups face, as well as identify effective strategies that may help these population groups obtain employment. (See app. III for a complete list of responses to these items.)

In addition to population groups, experts also identified several employment and training programs that they believe warrant research attention. In particular, experts most often ranked three components of the WIA program—WIA Adult, WIA Dislocated Workers, and WIA Youth—as key to evaluate in ETA’s future research. Among those three, WIA Adult was ranked the highest. (See app. III for a complete list of experts’ responses on employment and training programs to evaluate.)

ETA’s Research Studies Generally Answered the Questions Posed, but Their Usefulness Was Limited

Research organizations and academic institutions with responsibility for implementing ETA-funded research generally used methodologies appropriate for the questions posed, but the studies were not always useful for informing policy and practice. From January 2008 through March 2010, ETA published 17 large research and evaluation reports—14 evaluations and 3 research reports—that each cost $1 million or more. Four of these reports were designed to demonstrate what works and for whom. Each of these four reports compared the employment-related outcomes of individuals or regions who participated in training or employment programs with the employment outcomes of similar individuals who did not participate in the programs. The remaining 13 reports were descriptive and were not designed to assess program outcomes.

In several studies we examined that cost $1 million or more, we found that, for a number of reasons, ETA’s research studies were limited in their usefulness and in their ability to inform policy and practices. For example, in a study of the Prisoner Re-entry Initiative, shortcomings in the data collection phase limited the strength of the findings and, as a result, limited the study’s opportunity to influence policy directions. Among other things, while the study provided information on employment-centered opportunities for ex-offenders, the study relied on self-reported baseline data, did not account for differences across sites where services were received, lacked the capacity to record differences in the intensity of those services, and researchers failed to ensure that data collectors were properly trained.

In another study, researchers did not control for bias in selecting participants, compromising their ability to draw conclusions about the cause and effect of program outcomes. Authors of this study on the Workforce Innovations in Regional Economic Development (WIRED) initiative acknowledged that the study would be unable to attribute outcomes to program services because it did not use random assignment in selecting participating regions. We have previously criticized ETA for failing to adequately provide for evaluating the effectiveness of its WIRED initiative.

Moreover, some studies were limited due to observation periods that did not match the needs of the studies’ objectives. For example, an evaluation of an entrepreneurship training project was unable to assess the effectiveness of the project in meeting its long-term goals of increasing business ownership and self-sufficiency because the time frames for the study were too short. In this study, data collection was limited to 18 months after participants were randomly assigned, a period far shorter than the 60-month period recommended by experts. (See app. IV for additional information on the methodological characteristics of these studies.)

Experts generally agreed that ETA’s research had limited usefulness in informing policy and practice. Over one-third of the 39 experts reported that over the past 5 years, ETA’s research informed employment and training policy and state and local practices to a little extent or not at all. (See fig. 5.) Some experts commented that the design of these studies and the length of time to complete them and disseminate results reduced their usefulness. For example, many of the reports that we reviewed costing $1 million or more were multiyear projects that took, in most cases, about 3 to 5 years to complete. Some experts commented that the inclusion of shorter-length studies may be useful in times of rapidly changing economic conditions. At least one expert noted that some mixed-methods studies would be useful—studies that would allow for short-term interim findings that could facilitate changes in practice during the course of the research study.

Members of our expert panel stressed the importance of ETA incorporating varied methodological approaches into its future research proposals to best position the agency to address key employment and training issues. Twenty-seven of the 39 experts reported it was very important that ETA evaluate its pilots and demonstrations. Twenty-three reported that it was very important that more randomized experimental research designs be integrated into ETA’s future research. (See fig. 6.) While several experts noted that these randomized experiments will allow ETA to identify the effectiveness of particular interventions or strategies, at least one expert suggested that ETA should be strategic in choosing the interventions it tests more rigorously, basing those decisions on what appears most promising in preliminary studies.

Furthermore, 16 of the 39 experts also reported that it is very important for ETA to consider including more quasi-experimental studies in the future. As previously discussed, such studies would include designs that compare outcomes between groups with similar characteristics, but do not use random assignment. By including more quasi-experimental designs, ETA may be able to better understand the link between services and outcomes in those settings where random assignment is not possible, ethical, or practical.

ETA Has Taken Steps to Improve Its Research Program, but Additional Actions Are Needed

Labor Has Taken Steps to Reform Its Research Program

Labor has taken several steps designed to improve the way it conducts research, both at the department level and within ETA.

Department-level efforts. Labor has changed the organizational structure of research within the department. In 2010, acknowledging the need for better and more rigorous evaluations to inform its policy, Labor established the Chief Evaluation Office to oversee the department’s research and evaluation efforts. The office, which resides within the Office of the Assistant Secretary for Policy, has no authority to direct research within Labor’s agencies, according to officials. It does, however, manage evaluations supported by funds from a departmentwide account, oversee departmentwide evaluations, and provide consultation to Labor agencies, including ETA. Specifically, the office is responsible for creating and maintaining a comprehensive inventory of past, ongoing, and planned evaluation activities within Labor and for ensuring that Labor’s evaluation program and findings are transparent, credible, and accessible to the public. In fiscal year 2010, the Chief Evaluation Office had an estimated budget of $8.5 million, and two of its four staff were on board by the beginning of fiscal year 2011.

ETA efforts. ETA has recently made changes to some of its research practices—chief among them is the involvement of stakeholders and outside experts in the research process. We previously criticized ETA for failing to consistently involve a broad range of stakeholders, outside experts, or the general public in deciding what areas of research it should undertake. We recommended that ETA take steps to routinely involve outside experts in the research agenda- setting process. For the upcoming 2010 to 2015 research plan, ETA has awarded a grant to the Heldrich Center at Rutgers University to convene an expert panel to help inform the research plan. The center is expected to issue a report in May 2011 that outlines the panel’s recommendations for research areas to include in the plan. In addition, ETA will work with other Labor agencies, as well as the Departments of Education and Health and Human Services, before finalizing its research agenda. Officials told us that they will also solicit public comments before the research plan is finalized.

In addition to engaging stakeholders, ETA has also established a formal research process. As we previously reported, ETA developed and documented its research process in 2007. The agency’s actions were in response to a request by the Office of Management and Budget (OMB) to establish more formal policies and procedures to guide its research—a request that came out of OMB’s concerns about the manner in which ETA’s research was being carried out. Prior to 2007, ETA lacked a documented research process, and its research was often conducted in an ad hoc manner. ETA’s current research process identifies the steps, activities, and time frames it uses to carry out its research. Figure 7 illustrates critical components of ETA’s 8-step research process.

ETA’s process contains several of the key elements identified by leading organizations as important for guiding research activities. For example, the process includes specific steps the agency should take to identify the types of evaluations it will perform, as well as the administrative steps it should take to develop evaluation plans and select the research projects to fund. In addition, the process also specifies key events and time frames, and provides for monitoring the implementation of the research. For example, the process stipulates that ETA should alert OMB of research reports that have not been approved for dissemination within 9 months of being submitted and allows contractors to publicly release their research reports within those same time frames.

Some Areas of ETA’s Research Program Merit Further Attention

Despite ETA’s efforts, more action is needed to improve its research program. While ETA has taken steps to document its research process, its process lacks specific details in some areas, creating ambiguities that could undermine efforts to adhere to a formal process. For example, as we previously reported, its process lacks clear criteria, such as a dollar threshold or a particular methodological design feature, for determining which projects require peer review. And while the process specifies the actions project officers should take if reports are not released in a timely manner, it does not specify the consequences for failing to do so. We previously recommended that ETA establish more specific processes, including time frames for disseminating research reports. ETA has taken some action, such as revising the performance standards for project officers to hold them accountable for meeting time frames, but these steps do not fully satisfy the recommendation because the changes are not yet reflected in the formal research process.

Moreover, ETA’s process is missing some critical elements that are needed to ensure that the current improvements become routine practices.

Consulting with the Chief Evaluation Officer. ETA’s process lacks a formal provision requiring consultation with the newly established Chief Evaluation Officer at important points in the research process. For example, it contains no provision for consulting with the Chief Evaluation Officer when developing its annual list of research projects or when determining how ETA will invest its research and evaluation resources. Such consultation could help Labor better coordinate its research and evaluation efforts and better leverage its research funding. Moreover, the process contains no provision for involving the Chief Evaluation Officer in the early stages of developing its research projects. In the recent past, Labor officials told us that ETA has had difficulty developing requests for research and evaluation proposals that can pass OMB technical reviews. In particular, OMB has been critical of ETA’s research designs because they failed to provide for adequate sample size and appropriate methodologies that are needed to obtain useful results. In addition, OMB has also expressed concerns with ETA’s reliance on process evaluations rather than focusing on outcomes. These difficulties have resulted in delays in the research process. ETA has begun to consult with the Chief Evaluation Officer; however, these consultations are not a routine component in the formal process.

Setting the research agenda. ETA’s current process, as documented, begins with phase two—selecting specific research studies—and misses the important first step of setting the overall research agenda. This first phase of the process should include the steps that ETA will take to establish its research priorities and to update them on a regular basis. It should also include provisions for ensuring critical issues are considered and internal and external stakeholders are included in developing the plan. Officials noted that they plan to incorporate the agenda-setting phase into its formal process, but have not yet done so. Setting the research agenda is key to ensuring that an appropriate mix of studies is included in future research. Failing to make this phase part of the formal process, including the specific steps to involve outside stakeholders that are currently under way, may leave ETA with little assurance that these efforts will continue in the future.

Beyond ETA’s process for conducting research, current research practices fall short of ensuring research transparency and accountability—essential elements of a sound research and evaluation program. The research program has few, if any, safeguards to protect it from undue influence. According to officials, at times in the past decade, many key research decisions have been made outside of the office that is responsible for research. For example, decisions about which research studies would and would not be publicly released were made at the highest levels within ETA, and the criteria used to make those decisions were unclear. Of the 34 reports that ETA released to the public in 2008, 20 had waited between 2 and 5 years to be approved for public release. Several reports that had experienced long delays had relatively positive and potentially useful findings for the workforce system, according to our analysis. Among the studies delayed by almost 5 years was an evaluation of labor exchange services in the one-stop system that found certain employment services to be highly cost-effective in some situations. Another study, delayed for about 3.5 years, was a compendium of past and ongoing experimental studies of the workforce system, including early findings and recommendations for future research.

In our previous report, we noted that ETA’s research and evaluation center lacked a specific mechanism to insulate it from undue influence. We reported that other federal agencies, such as the Department of Education’s Institute of Education Sciences and the National Science Foundation, engage advisory bodies in the research process. While not without tradeoffs in terms of additional time and effort, such an approach may serve to protect the research program from undue influence and improve accountability. ETA is currently involving outside experts in setting the research agenda for 2010 to 2015, but is not involving experts more broadly on research policy and practices.

ETA Has Recently Included More Random Assignment Studies in Its Research Program

ETA has recently begun to include more rigorous studies in its ongoing research. Of the 10 large, ongoing studies costing $2 million or more that began during the period of our review, three—the WIA Gold Standard Evaluation of the Adult and Dislocated Worker Programs, the Impact Evaluation of the Young Parents Demonstration, and the Evaluation of Project Growing America Through Entrepreneurship II (Project GATE II)—use experimental design with random assignment, as recommended by our experts. These ongoing studies—which range in cost from $2 million to nearly $23 million—have the potential to determine the effectiveness of some of the program services. Table 4 outlines some key characteristics of these three studies.

Experimental designs with random assignment are an important means to understand whether various program components or services are effective, but they are also often difficult to design and implement in real- world settings. For example, in doing evaluations of employment and training programs, researchers often have difficulty in recruiting sample sizes large enough to detect meaningful outcomes. Because employment and training services may vary by location, and participants and their socio-economic environments are diverse, researchers must find ways to standardize procedures and treatment or service options. This often means recruiting relatively large samples. However, studies can be intrusive, often requiring program sites to change how they operate or to increase the resources available to participants. As a result, recruiting sites and sufficient numbers of participants may be difficult.

Some of ETA’s ongoing research studies face challenges in recruiting sample sizes large enough to meet the studies’ objectives. For example, based on an OMB review, it was determined that the sample size for the Impact Evaluation of the Young Parents Demonstration had to be much larger in order to be able to assess the effectiveness of the program. At that time, ETA had already awarded two phases of grants. After consulting with the new Chief Evaluation Officer, ETA changed the number of participants required for the third phase from 100 to 400 to obtain a sample large enough to address OMB’s concerns and provide reliable estimates. However, grantees found it difficult to recruit even the 100 participants in the smaller sample, and it remains unclear whether they will be able to recruit all of the needed participants for the expanded design.

The WIA Gold Standard Evaluation of the Adult and Dislocated Worker Programs

The WIA Gold Standard Evaluation illustrates ETA’s difficulties in planning and executing large-scale, rigorous random assignment studies. WIA required that the Secretary of Labor conduct at least one multi-site control-group evaluation of the services and programs under WIA by the end of fiscal year 2005. ETA, however, delayed executing such a study, finally soliciting proposals in November 2007 and awarding the contract in June 2008. The contractor submitted the initial design report in January 2009 and provided ETA with design revisions in May 2010. Officials tell us researchers will soon begin randomly assigning participants. ETA expects to receive the first report (on implementation) during the winter of 2012- 2013 and the final report in 2015—10 years later than the WIA-mandated time frame.

An OMB-selected panel of government experts—a technical working group composed of experts chosen by ETA, the evaluation contractor, and OMB staff—reviewed the original design for this study. Reviewers agreed the design contained many strengths, including the selection of an experimental design and a net impact approach; the addition of a process or implementation study to evaluate differences among sites and other implementation and data collection issues; the use of administrative and survey data; the collection of information on services received by participants in the control group; and the collection of a wide range of outcome data for participants.

However, reviewers raised several concerns regarding the design. For example, they were skeptical that the researchers would be able to obtain a sufficiently large and representative sample to draw meaningful conclusions about the effectiveness of the national workforce system. In order to maximize participation, officials told us that the Assistant Secretary of ETA made personal phone calls to all selected sites to emphasize the importance of the study, offered an open door policy to site officials to discuss issues, and followed up with an appreciation letter. Furthermore, ETA required the evaluation contractor to provide reimbursement payments to each site to offset implementation costs.

Reviewers also had several other concerns regarding which groups would be included in the study and which groups would not. For example, some experts raised concerns about getting accurate information on the youth program because of the large, one-time infusion of funds the program received from the American Recovery and Reinvestment Act of 2009. Reviewers were further concerned about the appropriateness of the evaluation objectives, the adequacy of steps taken to account for the effect of variation in services across sites on evaluation outcomes, and the external validity or generalizability of the study. In order to address these concerns, ETA made substantial adjustments to the original design. Specifically, ETA officials told us that based on an agreement with OMB, they instructed the contractor to drop the youth component from the evaluation and to focus only on the Adult and Dislocated Worker programs. While we received information on the new design and time frames for the WIA Gold Standard Evaluation, a finalized design plan is not yet available. According to officials, a finalized design is being prepared and will be available in June 2011.

ETA Has Improved the Availability of Its Research but There Are Opportunities to Improve Its Search Page and Dissemination Methods

ETA Has Improved the Timeliness of Its Disseminated Research

ETA has recently improved the timeliness with which it disseminates its research reports. In our last review in January 2010, we found that 20 of the 34 reports that ETA disseminated in 2008 had been waiting 2 to 5 years to be publicly released. The 34 research reports published by ETA in 2008 took, on average, 804 days from the time the report was submitted to ETA until the time it was posted to ETA’s research database. By contrast, from 2009 through the first quarter of 2010, the average time between submission and public release was 76 days, which represents a more than 90 percent improvement in dissemination time compared with 2008. Additionally, there were no research reports in 2009 that were delayed for more than 6 months. Further, the average time to dissemination improved significantly even when we excluded such outliers as the 20 research reports that were delayed for 2 years or more. Without these outliers, average time to dissemination for reports in 2008 was 100 days, indicating that time to dissemination in 2009 through the first quarter of 2010 still improved by 24 percent.

ETA Has Improved Its Research Database but Lacks Plans for Assessing the Usability of Its Search Page

In 2010, ETA updated its online, Web-based search page in order to improve the usability of its research database—the primary tool for making ETA research available to policymakers and the general public. Officials told us that ETA’s old Web-based search page was so error-prone and difficult to use that they opted to substitute it with one that had not yet completed internal testing. Our review of the old Web-based search page confirmed that it had serious limitations and did not consistently return the same results. For example, when we searched the database by title for a known ETA research report titled Registered Apprenticeship, we successfully retrieved that report once. One month later, when we entered the exact same search terms, we were unable to retrieve the report. (For a more complete description of our analysis of ETA’s search capability, see app. II.)

In our review of the updated Web-based search page, we found that the updates make the research database more useable. Labor officials told us they have taken other steps, as well, in efforts to improve its Web-based search page. For example, they have developed a project plan that articulates the steps Labor will take to update ETA’s Web-based search page. In addition, they have assigned a database administrator whose responsibilities include performing daily quality control spot checks in order to monitor performance and address technical problems.

Although these changes have the potential to improve the usability of ETA’s database, Labor has not developed a formal plan for assessing the overall effectiveness of its Web-based search page, including user satisfaction. Labor has made a number of changes to the way the page operates, but it has not provided users with tips on how to use the search functions, even though it is an industry standard to do so. Even skilled users who were familiar with the old Web-based search page may need guidance on the exact meaning of new terms and functions now available on the new page. For example, the old Web-based search page gave users the option of searching by “key word,” which is no longer an option in the new page. Instead, “key word” searches have been replaced with a variety of other options, including the ability to search the full text or abstract of a research report. However, there is no guidance on the Web site on how to use these new search options. Industry best practices suggest that a Web site evaluation plan that incorporates data from routine reviews of Web site performance and that assesses user satisfaction can help agencies ensure the usability of their Web sites. ETA currently has no plans to do such assessments.

ETA Uses Various Methods to Disseminate Research, but Experts Suggest Additional Methods

At present, ETA’s research database is the primary method that ETA uses to make its research reports publicly available, according to officials. In order to call attention to new reports available in that database, ETA sends a Training and Employment Notice, also commonly known as a TEN, to an e-mail list of the more than 40,000 subscribers who have signed up to receive them. ETA’s research process specifies that for each new research report that is approved for dissemination, ETA must draft a TEN and an abstract before it is posted to ETA’s Web site. Beyond posting reports to its database, ETA also distributes hard copies of some of its research reports.

In addition to electronic distribution, ETA also organizes various presentations to disseminate its research findings. These presentations, however, are done on an ad hoc basis. As mentioned in our prior report, ETA hosted a research conference in 2009 to present some of its research findings, renewing a practice that had been discontinued in 2003. As ETA looks to the future, officials tell us they will plan and organize similar research conferences as resources permit. In addition to these research conferences, ETA’s regional offices occasionally hold smaller, regional conferences as well. Beyond these formal conferences, ETA also hosts an internal briefing series at Labor headquarters where research contractors present their findings to various officials. For each of these briefings, ETA has a list of stakeholders that it invites, including various Labor officials, outside agency officials, congressional staff, and other outside stakeholders.

Experts who participated in our virtual panel provided their views on the effectiveness of different methods for disseminating research reports, and several of those rated more highly are methods currently employed by ETA. (See fig. 8.)

Most of the experts (30 of the 39 respondents) in our panel reported that using e-mail notifications, a searchable database of ETA papers, and briefings at ETA for external audiences (including stakeholders and policymakers) would be very effective or extremely effective approaches for disseminating research. In addition, a majority of the experts (26 of the 39 respondents) in our panel reported that publishing one-page summaries of research findings, not currently done by ETA, would be very or extremely effective.

Conclusions

ETA plays an important role in developing workforce policies and helping to identify the most effective and efficient ways to train and employ workers for jobs in the twenty-first century. With the current economic crisis and high unemployment rates, ETA’s role has become even more critical. The agency has made some improvements in its research program, even since our last review a year ago. But officials can do more to ensure that the progress continues in the years to come.

ETA has taken a major step forward in establishing a formal research process—one that documents most actions that must be taken in the life cycle of a research or evaluation project. But, it is missing some key elements that could help ensure the continuation of current practices. While ETA is currently using outside advisory bodies to help it establish its research agenda, the formal process does not include the agenda-setting phase. Officials tell us they have plans to incorporate this phase in the future, and we urge them to do so. Without a formalized agenda- setting phase, ETA may miss opportunities to ensure that its research agenda addresses the most critical employment and training issues and that outside stakeholders are routinely involved. Moreover, ETA’s process has not formalized the now ad hoc advisory role of the Chief Evaluation Officer. Absent the routine involvement of the Chief Evaluation Officer at key steps in the process, ETA may find it difficult to ensure that research proposals are asking the right questions, are methodologically sound, and that they can quickly pass critical OMB reviews.

ETA’s research findings are now available to the public on its Web site in far less time than it took in 2008. Despite this clear improvement, ETA has not taken the necessary steps to ensure that research products remain readily available to the public. The decision regarding what and when to make research publicly available is left in the hands of too few, and the process lacks needed safeguards to ensure transparency and accountability. Absent safeguards, key research decisions may again be made in ways that harm the credibility of the program and prevent important research findings from being used to inform policy and practice.

ETA’s Web-based search page is the primary means ETA uses to make the research studies it funds readily available to the public. And, while ETA has improved the functionality of its Web site, no effort has been made to ensure that the problems that plagued the system in the past do not recur. Absent such efforts, ETA will have little assurance that its research findings are actually available to users.

Recommendations for Executive Action

To improve ETA’s research program, we recommend that the Secretary of Labor require ETA to take the following three actions: Formally incorporate into its research process the routine involvement of the Chief Evaluation Officer at key milestones, including at the development of ETA’s annual research agenda and spending priorities, as well as at the early stages of developing specific research projects.

Develop a mechanism to enhance the transparency and accountability of ETA’s research program. For example, such a mechanism might include involving advisory bodies or other entities outside ETA, in efforts to develop ETA’s research policies and processes.

Develop a formal plan for ensuring that ETA’s research products are easily accessible to stakeholders and to the general public through its searchable database. Such a plan could involve requiring Labor to assess the overall effectiveness of its Web-based search page, including user satisfaction with search features.

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Labor for review and comment. Labor provided written comments, which are reproduced in appendix VII. In addition, ETA provided technical comments, which we incorporated where appropriate.

In its response, Labor generally agreed with our findings and all of our recommendations, noting its ongoing efforts in support of the recommendations.

Regarding our recommendation to formally incorporate into its research process the routine involvement of the Chief Evaluation Officer at key research milestones, Labor noted that it is currently taking steps to do so. Officials reported that they have worked closely with this office in various aspects of its research, including discussing research, demonstration projects, and evaluations in the early stages of development and plans to continue this collaboration in the future. However, ETA’s comments did not discuss plans to update its documentation on the formal research process. We found in our review that involving the Chief Evaluation Officer was not an official component of ETA’s documented research process, and it occurred on an ad hoc basis. As ETA moves forward, we urge the agency to modify its current research process and document the involvement of the Chief Evaluation Officer at critical research milestones.

Regarding our recommendation for ETA to develop a mechanism to enhance the transparency and accountability of its research program, officials cited several steps they are taking to improve the program, including involving outside experts in the development of their 5-year research plan and establishing advisory and peer review groups to review major evaluations. While officials note they plan to engage outside experts in broader research policies and processes, we encourage ETA to formalize this involvement. Moreover, we encourage ETA to continue to move forward in its efforts to further clarify components of its research process that are not well defined, including, for example, the criteria to be used when deciding when a peer review should be performed.

Regarding our recommendation to develop a formal plan to ensure that disseminated research is easily accessible to stakeholders and the general public, officials cited specific steps the agency has taken to improve its Web-based research database. While these actions are a step in the right direction, we believe that it is still important for Labor to develop a formal and comprehensive plan to ensure that disseminated research continues to be accessible to the public.

Furthermore, Labor expressed concerns about how we characterized the agency’s budget for pilots, demonstrations, and research. Recognizing these concerns, we made changes to the report to better capture the amount of funding ETA has available for research.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Labor, and other interested parties. The report will also be available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-7215 or scottg@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VIII.

Appendix I: Status of Prior GAO Recommendations to the Department of Labor, as of January 2011

Appendix I: Status of Prior GAO Recommendations to the Department of Labor, as of January 2011 Department of Labor’s response The Department of Labor (Labor) does not agree with this recommendation as written. According to Labor officials, the Administrator of OPDR currently reports to the Deputy Assistant Secretary, not directly to ETA’s Assistant Secretary. However, Labor officials acknowledge that important functions such as research and evaluation should not have too many intermediary reporting layers. To facilitate communication, officials further noted that the OPDR Administrator, the Deputy Assistant Secretary, and the Chief Evaluation Officer meet on a monthly basis with the Assistant Secretary to discuss evaluations.

Labor agrees with this recommendation, but authority to make key decisions still resides with the Office of the Assistant Secretary for ETA. OPDR currently provides recommendations to this office regarding plans for conducting and disseminating research. In an effort to improve evaluations departmentwide, the Secretary of Labor recently established the Chief Evaluation Office to monitor evaluation efforts across the department. OPDR has begun to work informally with the Chief Evaluation Officer and the Chief Economist to design and implement research and evaluation projects.

Labor agrees with this recommendation. ETA reports that it has taken some steps to establish more specific processes regarding dissemination of research, citing changes in performance standards for project officers. However, our recommendation would make broader changes to their research process and no such changes are reflected in the documents the agency provided.

Labor’s actions do not completely satisfy recommendation … create an information system to track research projects at all phases to ensure timely completion and dissemination.

Labor agrees with this recommendation. Officials report that they have begun working on a centralized, electronic tracking system for its research projects. However, the work is still under way and no time frames have been provided for its completion. Currently, OPDR uses an Excel document to keep inventory of all research, demonstration, and evaluation projects.

Labor’s actions do not completely satisfy recommendation … instruct ETA’s research and evaluation center to develop processes to routinely involve outside experts in setting its research agenda and to the extent required, do so consistent with the Federal Advisory Committee Act.

Labor agrees with this recommendation. OPDR has taken steps to engage outside experts in setting its 5-year research plan for 2011 and collaborate with the research and evaluation centers of other federal agencies, such as the Departments of Education and Health and Human Services. OPDR also plans to convene an expert panel, solicit public comments, and incorporate feedback from its 2009 Reemployment Research Conference and its 2010 ETA Reemployment Summit. However, despite these current efforts, OPDR has not formally incorporated them in its standard research process.

Appendix II: Scope and Methodology

We were asked to review the Employment and Training Administration’s (ETA) research program to better understand its approach to conducting and disseminating research. Specifically, we answered the following research questions: (1) To what extent do ETA’s research priorities reflect key national employment and training issues and how useful were the studies funded under them? (2) What steps has ETA taken to improve its research program? (3) How has ETA improved, if at all, the availability of its research since our last review in January 2010 and what other steps could ETA take to further ensure its research findings are readily available?

To answer our research questions, we convened a virtual panel using a modified Delphi technique to obtain selected employment and training experts’ opinions on ETA’s research priorities and dissemination methods. We also visited two workforce agencies in Pennsylvania and Virginia that are implementing two of ETA’s ongoing research studies to learn about implementation issues and how research is being conducted. In addition, we reviewed 58 ETA-funded research and evaluation reports disseminated between January 2008 and March 2010 and assessed the methodological soundness of completed studies that cost $1 million or more. We also reviewed ETA’s ongoing studies that cost $2 million or more. To determine the availability of ETA’s research, we measured the time between when the final version of a research report was submitted to ETA’s Office of Policy Development and Research (ODPR) and when it was posted on ETA’s Web site. We also conducted a series of systematic searches to test the reliability of ETA’s research database. Furthermore, we interviewed Department of Labor (Labor) and ETA officials to better understand ETA’s research capacity, processes, and the use of research findings to inform policy and practice. Lastly, we reviewed relevant agency documents and policies, as well as relevant federal laws.

Web-Based Expert Panel

We convened a nongeneralizable Web-based virtual panel of 41 employment and training experts to obtain their opinions on ETA’s research priorities and dissemination methods. We employed a modified version of the Delphi method to organize and gather these experts’ opinions. To encourage participation by our experts, we promised that responses would not be individually identifiable and that results would generally be provided in summary form. To select the panel, we asked several employment and training experts, on the basis of their experience and expertise, to identify other experts who were knowledgeable of ETA and the research it conducts and disseminates. After receiving nominations from experts, we reviewed the list to ensure that it reflected a range of perspectives and backgrounds, including academics, researchers, and consultants.

Our Delphi process entailed two survey phases. (See app. V for a copy of our phase I and phase II questionnaires.) In phase I, which ran from June 22, 2010, to August 9, 2010, we asked the panel to respond to five open- ended questions about ETA’s research priorities and dissemination methods. We developed these questions based on our study objectives and pretested them with four experts by phone to ensure the questionnaire was clear, unbiased, and did not place an undue burden on respondents. All relevant changes were made before we deployed the first Web-based questionnaire to experts.

After the experts completed the open-ended questions in the first questionnaire, we performed a content analysis of the responses in order to identify the most important issues raised by our experts. Two members of our team categorized experts’ responses to each of the questions. Any disagreements were discussed until consensus was reached. Thirty-six of the 41 panelists selected completed phase I of the survey (about an 88 percent response rate). Those that did not complete phase I were allowed to participate in phase II. (For a list of experts who participated in phase I and phase II, see app. VI.)

The experts’ responses to phase I were used to create the questions for phase II. In phase II, we gathered more specific information on ETA’s research and dissemination practices. Phase II, which ran from October 29, 2010, to December 14, 2010, consisted of 16 follow-up questions where panelists were asked to either rank or rate the responses from phase I. We pretested the questionnaire for the second phase with three experts to ensure the clarity of the instrument. We conducted two of our expert pretests in-person and one by phone. Thirty-nine of the 41 experts completed phase II (about a 95 percent response rate).

Site Visits to Workforce Agencies Implementing ETA-Funded Research Studies

To further enhance our understanding of how ETA conducts its research, we visited two workforce agencies that are implementing ETA’s ongoing research studies. First, we visited the Lancaster County Workforce Investment Board in Lancaster, Pa., which received funding from ETA to implement the Young Parents Demonstration project. This project provides educational and occupational skills training to promote employment and economic self-sufficiency for mothers, fathers, and expectant mothers ages 16 to 24. Second, we visited the Northern Virginia Workforce Investment Board in Falls Church, Va., which received funding from ETA to implement the second round of the Project Growing America Through Entrepreneurship, also referred to as Project GATE II. This grant helps dislocated workers aged 50 and over obtain information, classroom training, one-to-one technical assistance, counseling, and financial assistance to establish new businesses in order to help them start and sustain successful self-employment.

We selected these workforce agencies because they were identified by ETA as having active research projects in the implementation stage. These sites also required minimum travel expenditure. During our site visits, we toured each workforce agencies’ facilities and used a semistructured interview protocol to interview the project director and staff about their role and responsibilities, the extent to which they communicate with ETA, and whether or not they face challenges with regards to implementation. At the Lancaster County site, we participated in an informal on-site lunch forum where local community programs that the agency partners with talked with us about their collaboration with the program. At the Northern Virginia GATE II site, we observed a focus group operated by the program to facilitate information-sharing among participants.

After our site visits, we conducted phone interviews with the contractors that received funding from ETA to evaluate the outcomes of two research projects. Specifically, we interviewed the Urban Institute, which evaluates the Young Parents Demonstration project, and IMPAQ International, which evaluates Project GATE II. Both projects include an experimental component with control and comparison groups to determine the effects of program interventions on participants. During our interviews we used a semistructured questionnaire and asked questions to better understand their roles and responsibilities for the project, the extent to which they communicate with ETA, and whether or not they experience methodological and implementation challenges.

Analysis of Methodological Characteristics of ETA

We reviewed the 58 research and evaluation reports that ETA disseminated between January 2008 and March 2010 and assessed the methodological soundness of 11 completed studies that cost $1 million or more. In addition, we reviewed 10 ongoing studies costing $2 million or more to determine if research practices or the soundness of research designs had changed over time. We categorized the 58 studies disseminated between January 2008 and March 2010 by study type, cost, and research area. For the larger studies costing $1 million or more, we analyzed key characteristics including design features, scope, generalizability, and the appropriateness of analytical approaches and statistical procedures. These studies were analyzed independently by two analysts and the agreement between their ratings was 100 percent. (For results of this analysis, see app. VI.)

Analysis of the Timeliness and Effectiveness of ETA’s Dissemination Activities

To evaluate the availability of ETA’s research, we measured the time between when the final version of a research report was submitted to ODPR and when it was posted on ETA’s Web site. Specifically, we measured the dissemination time frames for reports posted in 2008 and compared that with the dissemination time frames for reports issued between January 2009 through March 2010. In addition, we conducted a series of systematic searches to test the reliability of ETA’s Web-based research database. To perform our searches, we selected a random sample of 30 reports from the 312 reports available on ETA’s research database at the time of our review. Specifically, we tested a variety of search functions available at the time of our review to determine the extent to which research reports could be easily retrieved on ETA’s research database. These functions included searches by title, keywords, author, and/or dates. We classified a report as retrievable if it appeared anywhere in our search results. We conducted our initial searches between June 30, 2010, and July 6, 2010. A second round of searches was conducted between August 6, 2010, and August 10, 2010. Further, we interviewed Labor and ETA officials to learn more about the search capabilities of ETA’s research database and the processes used to address errors and implement changes. Finally, we interviewed officials to gather information about ETA’s dissemination methods, including its current techniques and future plans for disseminating research reports.

Interviews with Labor and ETA Officials

To better understand the agency’s research capacity, we interviewed ETA officials and reviewed relevant agency and budget documentation. Similarly, to obtain information on ETA’s research process and how research findings are used to inform employment and training policy and practice, we interviewed officials and reviewed agency documentation, including relevant policies and procedures that guide ETA’s research. We also reviewed relevant federal laws.

We conducted this performance audit from March 2009 through March 2011 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix III: The Panel’s Ratings of Key Employment and Training Issues, Populations, and Programs That ETA Should Address in Its Future Research

In our Delphi phase II Web-based questionnaire, we asked the panel of experts to rate and rank the key employment and training issues, populations, and programs that ETA should address in its future research. These issues were identified by the panel during phase I. For our analysis, we calculated basic descriptive statistics on these issues, which are presented in tables 5 through 7.

Appendix IV: Characteristics of Research Studies Disseminated between January 2009 and March 2010 That Cost $1 Million or More

Other (comparison v. treatment group using propensity score matching)

Increasing the labor market participation of underutilized populations characteristics collected before the intervention, but these were not used to make comparisons)

Integration of the workforce and regional economic development network analysis)

Integration of the workforce and regional economic development network analysis )

Experimental (randomized control trials)

Increasing the labor market participation of underutilized populations collected on characteristics upon entry and outcome characteristics collected after completion)

Appendix V: Delphi Phase I and Phase II Questionnaires

Appendix VI: Experts Who Agreed to Participate in GAO’s Delphi Panel

Trachtenberg School of Public Policy and Public Administration, George Washington University Abt Associates Inc.

Mathematica Policy Research, Inc.

W.E. Upjohn Institute for Employment Research National Bureau of Economic Research Robert M. LaFollette School of Public Affairs, University of Wisconsin-Madison W.E. Upjohn Institute for Employment Research Abt Associates Inc.

John J. Heldrich Center for Workforce Development, Rutgers, The State University of New Jersey Department of Economics, College of Arts and Sciences, American University Mathematica Policy Research, Inc.

Department of Economics, University of Missouri- Columbia Humphrey School of Public Affairs, University of Minnesota Minnesota Department of Employment and Economic Development W.E. Upjohn Institute for Employment Research Institute for Policy Studies, Johns Hopkins University Center for Law and Social Policy Peterson Institute for International Economics Mathematica Policy Research, Inc.

Appendix VII: Comments from the Department of Labor

Appendix VIII: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the contact listed above, Dianne Blank, Assistant Director, and Kathleen White, analyst-in-charge, managed all phases of the engagement. Ashanta Williams assisted in managing many aspects of the work and was responsible for final report preparation. Lucas Alvarez and Benjamin Collins made significant contributions to all aspects of this report. In addition, Amanda Miller assisted with study and questionnaire design; Joanna Chan performed the data analysis; Stephanie Shipman advised on evaluation approaches; James Bennett provided graphics assistance; David Chrisinger provided writing assistance; Alex Galuten and Sheila McCoy provided legal support; and Sheranda Campbell and Ryan Siegel verified our findings.

GAO Related Products

Program Evaluation: Experienced Agencies Follow a Similar Model For Prioritizing Research. GAO-11-176. Washington, D.C.: January 14, 2011.

Employment and Training Administration: Increased Authority and Accountability Could Improve Research Program. GAO-10-243. Washington, D.C.: January 29, 2010.

Workforce Investment Act: Labor Has Made Progress in Addressing Areas of Concern, but More Focus Needed on Understanding What Works and What Doesn’t. GAO-09-396T. Washington, D.C.: February 26, 2009.

Employment and Training Program Grants: Evaluating Impacts and Enhanced Monitoring Would Improve Accountability. GAO-08-486. Washington, D.C.: May 7, 2008.

Federal Research: Policies Guiding the Dissemination of Scientific Research from Selected Agencies Should Be Clarified and Better Communicated. GAO-07-653. Washington, D.C.: May 17, 2007.

Data Quality: Expanded Use of Key Dissemination Practices Would Further Safeguard the Integrity of Federal Statistical Data. GAO-06-607. Washington, D.C.: May 31, 2006.

Workforce Investment Act: Substantial Funds Are Used for Training, but Little Is Known Nationally about Training Outcomes. GAO-05-650. Washington, D.C.: June 29, 2005.

Program Evaluation: An Evaluation Culture and Collaborative Partnerships Help Build Agency Capacity. GAO-03-454. Washington, D.C.: May 2, 2003.

Workforce Investment Act: Improvements Needed in Performance Measures to Provide a More Accurate Picture of WIA’s Effectiveness. GAO-02-275. Washington, D.C.: February 1, 2002.