Unemployment Insurance: More Guidance and Evaluation of Worker-Profiling Initiative Could Help Improve State Efforts
Released: 2007-06-14T00:00:00

Letter

Background

Beginning in the mid-1970s, major structural changes took place in the American economy, as advances in technology, international competition, plant closings, and corporate streamlining resulted in the dislocation of thousands of workers from their jobs. Some of these individuals possessed skills that were no longer in demand; others suffered from a lack of job search skills. In the 1980s and early 1990s, demonstration projects were conducted in New Jersey, Nevada, Minnesota, and Washington. The New Jersey and Minnesota projects showed the efficacy of using statistical methods and administrative data to identify those who are likely to experience long periods of joblessness. For example, the New Jersey demonstration project screened claimants with various eligibility requirements and found that the screening allowed the state to direct services to those who generally faced reemployment difficulties. Further, results from all four states showed that providing more intensive job search assistance to this population reduced the duration of insured unemployment and UI expenditures.

In response to these events, the Clinton administration proposed legislation to implement worker profiling in 1993. In the same year, Congress enacted the Unemployment Compensation Amendments, amending the Unemployment Insurance program legislation. The law requires that states establish and utilize a system of profiling all new claimants for UI regular compensation. The system must identify those claimants that will be likely to exhaust regular compensation and refer them to reemployment services, such as job search assistance services. Typically, such claimants receive services at one of the local “one-stop” employment services centers that exist throughout the nation. States are required to collect information on the type of services claimants receive, their participation, and their subsequent employment outcomes. The last could include such information as whether referred claimants obtained new jobs and the related wage levels.

In 1994, Labor issued guidance to help states establish profiling tools and provide necessary reemployment services. In profiling claimants, Labor required that states consider factors that include whether the claimant has a date for being recalled to work, union status, first unemployment benefit payment, and previous industry or occupation of employment. Labor recommended also considering some additional factors such as claimants’ education, tenure at previous job, and the state unemployment rate.

Labor outlined recommended processes for providing reemployment services to referred claimants, including (1) an orientation session for claimants that would, among other things, explain the availability and benefit of reemployment services; (2) an assessment of the specific needs of each claimant, if appropriate; and (3) based on the assessment, development of an individual plan for services that would guide a claimant’s further services. (See fig. 1.) Under the law, states must also require that claimants who have been referred to reemployment services participate in those services as a condition of eligibility for receiving compensation.

Labor may withhold UI administrative grants from a state if Labor finds, after notice and an opportunity for a hearing, that a state has failed to comply with worker-profiling requirements. These include identifying claimants most likely to exhaust benefits, referring claimants to reemployment services, and collecting follow-up information on services received and subsequent employment outcomes.

The law required that Labor report to Congress on the operation and effectiveness of the profiling system within three years of its enactment. Labor issued a report to Congress in March 1997, and published a final report in 1999 on the program’s implementation and operation nationwide and the effectiveness in six early implementation states. Labor has published no studies on the effectiveness of the initiative since then. The agency’s strategic plan for fiscal years 2006 through 2011, in providing an overview of program evaluation, includes no ongoing or future research topics addressing the impact of the worker-profiling initiative. Labor has conducted impact evaluations as part of its program evaluations in the past. In fiscal year 2007, Labor was appropriated $17.7 million for pilots, demonstrations, and research.

Funding for the worker-profiling program is provided from a variety of sources. Federal funding for the creation and maintenance of profiling models can come from UI administrative funds, which are financed by a federal UI tax on employers. Reemployment services can be funded through a variety of sources. For example, states can use Wagner-Peyser Employment Services grants as well as other state sources of funding to provide reemployment services to profiled UI claimants. From 2001 to 2005, Labor also provided Reemployment Services grants to all states in order to enhance and target services to claimants through the nation’s network of one-stop employment service centers.

Most States Use Statistical Models to Identify Claimants Likely to Exhaust Benefits, but Many Have Not Updated Them to Account for Changing Economic Conditions

The large majority of states use statistical models to identify unemployment recipients who are most likely to exhaust benefits. However, many states have not recently adjusted their models, risking the possibility that these models may lose predictive accuracy over time. Forty-five states use statistical models to identify and rank clients by their likelihood to exhaust benefits, while 7 states use characteristic screens that do not rank claimants. One state—Florida—allows the local areas to decide whether to use statistical models or screening tools. Among the states using statistical models, the detailed specifications of these models vary considerably from state to state. Further, many states do not regularly update their models, a fact that can lead to a loss of predictive accuracy over time. A survey of the states reveals that many have not revised or updated their models in many years. Officials in states we contacted explained that they face a number of impediments to doing so.

Most States Use Statistical Models instead of More Limited Characteristic Screens

Under worker profiling, state UI agencies are to identify claimants who are most likely to exhaust benefits in two steps. States screen claimants in order to eliminate claimants who are unemployed but job-attached or would otherwise not qualify for referral to services from the profiling process. After the initial screening, states profile remaining claimants— that is, they consider a range of personal and economic variables related to a claimant and determine whether or not he or she is likely to exhaust benefits. Although states have considerable flexibility in determining what variables to use, Labor has recommended the use of five variables, as outlined in table 1.

We found that states used one of two methods to identify claimants who are most likely to exhaust benefits—the statistical model or characteristics screening. Both of these look at a range of personal and economic factors that help predict exhaustion.

Forty-five of the 53 states and territories use statistical models to identify clients likely to exhaust benefits. (See fig. 2.) Using various statistical techniques, these models consider the combined quantitative influence of various personal and labor market characteristics and produce a measurement of a claimant’s likelihood to exhaust. In statistical models, each characteristic—commonly referred to as a variable—is associated with a specific mathematical weight that quantifies the variable’s contribution to the claimant’s probability of exhaustion. If, for example, a claimant’s last job was in a steeply declining industry, the industry variable would have a positive effect on the score, indicating a claimant’s likelihood to exhaust. Conversely, if a claimant’s last job was in an expanding industry, it would have the opposite effect. Essentially a statistical model produces a weighted average of the effect of all the variables combined. As a result, states that use statistical models can rank claimants from greatest to least likelihood of exhaustion, and target reemployment services to claimants with the greatest likelihood of exhausting. According to an official of the Upjohn Institute for Employment Research, such models, if properly developed and maintained, are a powerful and effective means of identifying particular populations for a range of social service programs.

Seven of the 8 remaining states use characteristic screens that do not allow them to rank claimants. One state, Florida, delegates the selection of profiling tools to the local areas because state officials believe profiling can be done more accurately at that level. Like statistical models, characteristic screens may consider various factors associated with the likelihood to exhaust benefits, but treat them as yes-no decision points. Either the claimant has the attribute or does not. The relative importance of any one variable in relation to others is not considered. Claimants selected through this process must have each of the screening criteria. For example, the characteristic screen used by Delaware considers whether or not a claimant meets specific criteria relating to industry, occupation, and job tenure. In Delaware, a claimant passes the job tenure screen if he or she has 2 or more years of tenure with his or her last employer. However, since claimants cannot be ranked, states using screens must develop a method, such as random assignment, to refer identified claimants to services if they are unable to serve all claimants that pass the screens. For example, Delaware used to refer claimants who passed the screen on a random basis, but now refers all claimants who pass the screen. Labor encourages the use of statistical models over characteristic screens because they are more efficient and precise in identifying claimants likely to exhaust.

Statistical Models Vary in Key Elements

Although all statistical models are supposed to identify claimants who are likely to exhaust benefits, the states can vary in how they specifically define this exhaustion. The model originally proposed by Labor is designed to predict the probability of exhaustion as a yes-or-no outcome— exhaustion or no exhaustion—and the claimant’s profiling score would reflect the probability of the yes outcome. Most states have adopted this definition. However, as Labor explained in 1998 guidance, this approach does not distinguish between claimants who almost exhaust benefits and those who do not come close to exhausting benefits. This is significant, because the claimant with nearly exhausted benefits may be in greater need of reemployment services than the clamant who uses a comparatively small portion of his or her benefits. Consequently, some states predict exhaustion as the amount of benefits a claimant will potentially use. For example, the profiling score produced by Kentucky’s model produces a number between 1 and 20. A claimant with a score of 20 is likely to use 95 to 100 percent of benefits; a claimant with a score of 19 is likely to use 90 to 95 percent of benefits, and so on.

State models can differ considerably in how they define similar variables, including those corresponding to the factors recommended by Labor. For example, California uses six categories to measure the job tenure variable, ranging from 1 year or less on the low end to more than 15 years on the high end. In contrast, Texas uses only two categories—job tenure of less than 1 year on the low side and tenure of more than 10 years on the high end. The Kentucky model, on the other hand, measures job tenure on a continuous scale—specifically, the length of time that a claimant held his or her last job. The definitions of variables associated with education, industry, and other variables can also differ among state models. For example, Kentucky includes “completed vocational education” as part of its education variable, while Wisconsin does not.

The number and nature of the additional variables can also differ significantly by state. The large majority of states using statistical models (34 of 45) use models that consider factors in addition to the five factors recommended by Labor, while about one-quarter do not. (See fig. 2.) Among the 6 states that we contacted that use statistical models, the number of additional variables used ranged from 1 in California to over 50 in Kentucky. For example, 2 of the 7 states we contacted—Texas and Illinois—consider the time lapse between the loss of a job and the application for UI benefits. According to Texas officials and Labor, delays in filing a claim are indicative of a difficult job search, thus increasing the likelihood of benefit exhaustion.

Although Models Require Periodic Maintenance to Ensure Predictive Accuracy, Many States Have Not Updated Their Models since 2000 or Before

While Labor has recommended that states update models periodically to reflect changes in economic conditions, many states have not done so in many years. If not periodically updated, statistical models can lose predictive accuracy over time because of changes in the labor market, the general economy, or other factors. Labor has emphasized the importance of updating models, and noted in 1998 guidance that models represent the historical period in which they were developed, and that old models become increasingly unrealistic and less useful over time. Labor has further recommended that models be assessed, and if necessary adjusted, approximately every 3 years. Officials in some of the 7 states we contacted also stressed the importance of updating models from time to time. For example, Washington officials noted that although a 2002 analysis of their model update showed that it accurately identified the majority of claimants who exhausted, this adjustment of their model was based on data collected in 1999 and 2000, and subsequent changes in their labor market and the general economy have made the model outdated. Also, a 2003 California study found that the state’s model underestimated benefit exhaustion and recommended an update to the model. Similarly, an official of the Upjohn Institute for Employment Research told us that the institute’s analysis of 1 state’s model found that before the model was updated, its results were little better than random selection of claimants. Officials in Washington and California told GAO that the models would be updated in the next year.

Models can be adjusted by modifying the mathematical weights associated with specific variables, and by adding, deleting, or redefining variables to enhance a model’s predictive power. This is necessary over time because, although a particular variable—such as a claimant’s industry—can remain an important predictor of exhaustion, its relative importance in the model can change significantly. For example, if a variable’s weight was estimated based on data from a historical period of large changes of employment levels in a particular industry or industries, the model might produce misleading results if used in a period of greater industrial and employment stability. Similarly, a variable that once served as an important predictor in a model may lose predictive value as the labor market and economic circumstances change, and conversely, other variables that may not have been relevant in one time period may become important at another time. For example, Texas deleted education as a variable from the model used in that state. According to a Texas official, statistical work performed for the model update revealed that the education variable did not measurably add to its predictive power.

Factors other than the labor market and general economy can affect the reliability of models as well. For example, in the past 10 years standardized coding used to identify both industries and occupations has changed, and some of the states we contacted had not updated their model to reflect this change. Illinois’ analysis of its model showed that while the model had generally retained predictive accuracy, areas of concern existed. For example, as a result of outdated occupational codes, certain occupations associated with greater likelihood of exhaustion were no longer being targeted, while others not associated with exhaustion were.

Although Labor has taken a number of actions to encourage and assist states in updating their profiling models, some states have not done so for many years. Labor has noted the importance of updating models in written guidance, sponsored occasional seminars where best modeling practices are shared with state staff, and provides on-demand technical assistance to states. However, Labor has not established requirements for updating models, and has not undertaken ongoing monitoring of state models. A recent Labor-commissioned survey revealed that many states have not updated their profiling models in recent years. (See figs. 3 and 4.) For example, although 21 states reported taking actions such as adjusting variable weights since 2003, many others have not. Specifically, 18 states have not done so since 1999 or before, and 12 of these reported never having done so.

According to Labor’s survey results, states have been even less inclined to adjust their models by taking actions such as changing or redefining variables in the models. As figure 4 shows, 30 states reported that they had not made such changes since implementation, and 23 states reported having done so. Only 11 of these 23 states reported having done so since 2003.

Labor’s survey did not inquire about factors influencing the frequency with which states update their models, but our contacts with 7 states reveal a variety of reasons that some states have not updated their models. Officials in California said that they had more pressing priorities for UI administrative funds, and thus would have difficulty funding model updates. Wisconsin officials said that revising the models required expertise that they did not have, either in-house or from other sources, such as a state university. Although Labor provides technical guidance and advice, and has offered seminars on updating models, state officials indicated they still need more continuous access to expertise in order to keep models updated. A Texas official said that sometimes historical data needed to determine a variable’s impact on exhaustion of benefits are not available, and so the variable cannot be included in the model. Relatedly, if the necessary data on claimants are not collected, or cannot be transmitted and used by the model, the related variable cannot be used. For example, a Texas official told us that certain variables, such as the number of a claimant’s dependents or spousal income, might be good predictive variables. However, the standard Texas application form for UI benefits does not ask about the number of dependents or spousal income, so these variables cannot be used.

Most Study States Did Not Take the In-Depth Approach Recommended by Labor to Ensure That Profiled Claimants Obtain Reemployment Services

Labor data provide a limited picture of states’ implementation of worker profiling, and some aspects of these data were not reliable. Further, 6 of our 7 study states did not offer the in-depth approach to services prescribed by Labor. These states generally referred claimants to services, held them accountable for attending the services, and provided them with an orientation and some instruction on job search skills. However, 6 of the 7 states did not adhere to Labor’s guidance recommending an in-depth individual needs assessment and a tailored reemployment service plan for referred UI claimants.

Data Collected by Labor Provided a Limited Picture of States’ Implementation of Reemployment Services

Between 2002 and 2006, about 94 percent of the UI claimants who received a first payment were profiled. To the extent that reemployment services are available, Labor requires that states refer profiled claimants to these services. Of those profiled, an average of 15 percent were referred to services, with states ranging from 5-year averages of 1 percent (Wyoming) to 52 percent (Washington) (See fig. 5.) While 3 states referred between 29 and 52 percent of profiled claimants to services, 28 states referred 14 percent or fewer. Further, of those claimants profiled, an average of 11 percent completed services, with states ranging from 1 percent (Arkansas, Colorado, Idaho, Michigan, and Wyoming) to 39 percent (Texas). (See fig. 6.) In 2 states, more than 27 percent of profiled claimants completed services. However, in 33 states, 13 percent or fewer of claimants did so.

See appendix II for the average percentages of profiled claimants referred to and completing services by state from 2002 to 2006.

Labor’s data are not sufficiently reliable to provide any information on the specific services provided to claimants—such as orientation, counseling, job search workshops, or job clubs. Specifically, Labor and state officials told us that definitions of these services can vary across states and within states over time as they change the content of their programs. For example, California officials told us that the state’s definitions of services provided were established over 10 years ago and that the nature of the services may have changed since then.

Six of Seven States We Studied Referred Claimants and Enforced Compliance with Referrals

We found that 6 of the 7 study states had, as required by Labor, referred profiled claimants to services and made claimants ineligible for benefits if they failed to attend reemployment services. In contrast, officials in 1 state told us that referrals had been delegated to local workforce areas, and that they did not know whether claimants were being referred to services statewide. We subsequently contacted some local workforce development offices in this state and learned that several had not been referring UI profilees to reemployment services for years. In addition, officials in this state told us that there are no consequences for those who fail to attend reemployment services. They further said they do not track information at the state level on whether claimants attend services. While Labor requires that states hold claimants ineligible for benefits for any week in which they fail to attend services, Delaware goes further and holds the UI benefits of claimants who do not attend services until they reschedule.

Some of the study states took additional steps to ensure compliance with service referrals, while others did not. Of the states that referred claimants to services, Delaware and Washington required that claimants reschedule if they failed to attend required services, while Texas and Wisconsin attempted to reschedule claimants in some cases and the remaining states did not do so. Officials in Delaware reported that they go so far as to have staff call claimants early during the week of their scheduled orientation to remind them to attend; officials in Washington said that some local workforce centers do this. Officials cited the large flow of claimants into the program, the complexity of the rescheduling process, and the scarcity of staff resources as reasons they did not reschedule referred claimants.

Six of Seven States We Studied Provided Limited Reemployment Services and Did Not Develop Individual Assessments and Service Plans

The reemployment services offered in the states we contacted generally did not conform to the robust service process originally outlined by Labor. Labor’s 1994 guidance states that after initial orientation, the service provider should determine the specific needs of each worker through an assessment process, such as vocational testing. Only one of our study states, Delaware, required that case managers conduct an initial assessment to determine what services claimants might need, such as Workforce Investment Act (WIA) training, depending on their job readiness level. Washington and Wisconsin required that claimants complete a self-assessment. For example, claimants at one one-stop center were expected to complete a one-page self-assessment that asks questions, including what educational level they attained, whether they had a current résumé, and whether they had difficulty filling out a job application. The 4 other states we studied required no assessment of any kind.

According to state officials, our study states also generally did not require, as recommended by Labor, that local offices develop or document a reemployment services plan that could serve as the basis for determining satisfactory participation. Only Delaware required case managers to develop service plans and meet with claimants on a monthly basis after each claimant’s assessment. In California and Wisconsin, claimants developed their own plans, which involved selecting an additional service session on a topic the claimants felt would be most helpful. For example, California required that UI claimants attend an orientation and choose an additional service, such as a WIA service or job club, that would constitute their individual reemployment plan.

All 7 study states cited lack of or declining funding as an issue that affected the provision of reemployment services. Specifically, some states mentioned the loss of Labor’s Reemployment Services grants, which had been awarded to all states between 2001 and 2005 to enhance and target integrated core services to claimants through the one-stop centers and were used by some states to fund program-related services. A Wisconsin official said that when the grant funds end in summer 2007, the state would only be providing worker profiling services in 6 to 12 of its 75 local workforce development offices. State officials also mentioned continuing declines in Wagner-Peyser, or Employment Services, funding. A local workforce manager in Washington said that there is a vast gap between the need for services and the resources and that the state only has resources for about 5 percent of the 50,000 to 60,000 UI claimant population. In order to help address this issue, officials in Washington told us that a special surtax is applied to UI taxes, and a small portion of this is diverted to worker-profiling service activities. While state officials were concerned with the availability of funding, Labor officials said that the purpose of the worker-profiling initiative is to target the funding that does exist to those claimants who need it most and that the program does not mandate that states serve any claimants they did not serve prior to its implementation.

Officials also cited various day-to-day challenges in providing effective reemployment services. A single services session can include claimants ranging from former upper management employees to construction and factory production workers, according to a Kentucky official. The same official said that pitching the class so that it is effective for both types of claimants can be difficult. Claimants’ language skills also can be a challenge. However, California addresses this by excusing non-English speakers from the session, and directing them to job service centers or community-based partners that provide reemployment services in their own language, unless the orientation is available in their native language.

Little Is Known about Program’s Effectiveness because There Are No Current Studies and Labor’s Data Are of Limited Usefulness

Little is known about the current effectiveness of the worker-profiling initiative. Research studies, while generally finding that profiling and a referral to services had a positive impact on claimants, used data from the early implementation of the initiative—1994 to 1996. Although Labor collects data on the outcomes of those profiled and referred to services, we found portions of it to be unreliable. In addition, state officials said they do not use Labor’s data for evaluation purposes.

Early Research Showed Some Positive Outcomes for Those Referred to Services, but There Are No Current Studies

Five methodologically sound studies looking at the impacts of the worker- profiling initiative after it was first implemented found that the program had some desired effects. Examining data from 1994 to 1996, the studies generally indicated that a referral to services under worker profiling led to a reduction in claimants’ duration on UI, a reduction in the amount of UI benefits that were paid out, and an increase in subsequent employment earnings. Though the methodologies varied, all the studies evaluated the impacts of the referral to services using statistical analyses to compare the outcomes of claimants who were referred to services against those of claimants who were not. As table 3 indicates, these studies cover a total of only 7 states, and no national study exists. Further, no study using current data exists. Labor sponsored the two multistate studies published in 1997 and 1999, but has not published any subsequent studies. According to Labor officials, the agency has no current plans to study the effects of profiling. Because data in all the studies were from the period when worker profiling was first implemented, the profiling process and reemployment services provided then may not reflect what states are currently offering.

While these early studies showed positive impacts for referred claimants with regard to reducing duration, reducing amount of UI benefits, and increasing employment earnings, there were mixed results for whether the program reduced the percentage of claimants who exhausted their benefits or improved subsequent employment rates (See table 4.) According to the studies, claimants who were referred to services had a decreased UI duration and received lower total amounts of UI benefits. Most of the studies found that claimants who were referred to services increased earnings in the year following the UI claim. However, the largest multistate study was unable to draw any conclusions about the impact on earnings because of contradictory data. Evidence that a referral to services reduced the percentage of claimants who exhausted their UI benefits was mixed. For example, one study showed a decrease in the percentage of claimants who exhausted their UI benefits in 3 states, but an increase in 2 states. The effect of a referral to services on employment rates was also inconclusive. According to the two multistate studies, the effect was minimally positive for one state, but the other 6 states showed insignificant or contradictory results. Most of the studies, however, did not examine subsequent employment rates.

Research studies of other work search programs corroborate the generally favorable results found in the impact evaluation studies of the worker- profiling initiative. Though the methodologies varied, these studies demonstrated that work search assistance reduced the duration claimants received UI benefits, among other beneficial impacts. In two demonstration projects, UI claimants who received job search assistance received fewer weeks of UI benefits. The reemployment services offered in these demonstration projects, however, were more robust; for example, in one study, claimants were required to attend an orientation, testing, a job search workshop, and a one-on-one assessment interview. As such, they may not reflect what is offered through the states’ worker-profiling programs currently.

Even though they were unable to provide supporting data, officials from our study states said that worker profiling was a useful program for UI claimants. They said it had enabled states to advertise their job search and training services and target claimants who are most likely to exhaust their UI benefits. In the process of referring claimants to services, states are also educating the community on the many services and resources available at the one-stop service centers. They also said the initiative was a way to focus resources on those who would benefit from job search assistance the most.

Outcomes Data Collected by Labor Are Limited and Not Consistently Used for Evaluation Purposes

Due to reliability issues, Labor’s claimant outcomes data are of limited value. Labor’s claimant outcomes data were sufficiently reliable for us to report only certain outcomes, including benefits exhaustion, weeks of benefit receipt, and reemployment. Those data showed that less than half of profiled claimants exhausted benefits, that on average they received benefits for about two-thirds of the typical maximum time allowed, and that about half found employment within 1 year of the referral to services (see table 5).

In addition to reliability issues, other characteristics, such as the lack of a comparison group and long time lags, limit the usefulness of both the reemployment services data and claimant outcomes data for states. First, the outcomes data reflected only the experience of those who were referred to services, and did not include an adequate point of comparison. It was therefore impossible to know if these outcomes were different than they would have been had the claimants not been referred to or completed reemployment services. Second, according to Labor officials, the data were originally intended for states to evaluate the effectiveness of the worker-profiling initiative. However, we found that neither Labor nor the states used the data for this purpose. Several state officials said the time lag and aggregated nature of the data were insufficient for program management purposes. The claimant outcomes data were not reported for more than a year after claimants were referred to services, and some state officials said they needed more timely data. Both the reemployment services and claimant outcomes data were aggregated to the state level, and some state officials said that local-level data would better meet their management needs. Four of our seven study states indicated that they did not utilize the reemployment services data or claimant outcomes data, and some only reported them because it was required by Labor; the remaining states said they used the reemployment services data for nonevaluative purposes, such as determining how many services were provided to claimants or the volume of claimants served under the worker-profiling program.

In light of these data limitations, several state officials said they developed their own program performance measures and reports instead of using the reemployment services data and claimant outcomes data. For example, Washington developed its own data warehouse system that links data on UI benefits, reemployment services, and claimant wages. According to officials, on a monthly basis they review performance indicators, such as the number of UI claimants that find employment and the amount of time it takes before finding employment.

Conclusions

Our findings suggest that although states continue to profile and refer claimants to reemployment services, the worker-profiling initiative is not a high priority at the federal level or in many states. In the past Labor has set out broad guidelines for states on the design and maintenance of profiling models. However, our analyses indicate that these have been inadequate. Labor’s 2006 survey of state profiling techniques revealed that many states had not updated their profiling models for many years. As a result, it is possible that many models have lost predictive accuracy, and are referring claimants to services who are not in need of them, or failing to refer claimants that are in need of them. However, the worker-profiling program is required by law, and if there is to be a continued federal mandate, it may be that a more assertive federal role is necessary to ensure the integrity of those models.

A long time has passed since Labor articulated its vision of reemployment services, and our review of seven states indicates that what is being practiced is a diminished version of that vision. While the states we studied indicated they provided orientation sessions that seemed to convey important information, including job search skills, Labor’s guidance implies a more tailored and in-depth approach to services. It may be that the original vision is no longer realistic or perhaps, in the states’ experience, necessary. Absent clarification at the federal level, it will remain unclear what Labor expects from the states.

The national data on the worker-profiling initiative is of very limited usefulness as a measure of program activity, outcomes, and effectiveness. Many of the data are not usable because of inconsistent or incorrect reporting, and neither Labor nor the states we contacted use the data for evaluating the worker-profiling initiative. Further, even if all the outcomes data were reported consistently and accurately, these data cannot, by themselves, be used to measure the impact of the program. In the end, by requiring the submittal of data that are of such limited reliability and value, Labor is potentially wasting both its own and the states’ resources. Finally, absent information about the program’s current impact, Labor may find it more difficult to make decisions regarding the best means for returning the unemployed to work more quickly.

Recommendations for Executive Action

To better ensure that claimants who need and could benefit from reemployment services are referred, and to ensure that resources are not unnecessarily expended on claimants not needing them, we recommend that the Secretary of Labor: 1. Reevaluate the agency’s worker-profiling data collection to determine whether it is sufficient for its intended purpose. The agency might assess gaps in data, evaluate data consistency, confer with states on what data would be beneficial to them, determine the purpose of the data collection and for whose benefit the data are collected, and modify what Labor requires states to collect. 2. Ensure that the Employment and Training Administration takes a more active role to help ensure the accuracy of the state profiling models. The agency might track states’ management of their models and actively encourage review and updating of models in specific states where there have been no efforts to adjust the model for a number of years. The agency could also assess whether an expanded technical assistance effort is needed, and, if so, take the lead in developing one. 3. Encourage states to adhere to Labor’s vision for in-depth reemployment services, such as conducting individualized needs assessments and developing individual service plans, or issue updated guidance if this original vision would be too burdensome for the states. 4. Evaluate the impact of the worker-profiling program on the reemployment of UI recipients to ensure the benefits are commensurate with the resources invested.

Agency Comments

We provided a draft of this report to Labor for review and comment. In general, Labor agreed with our findings and recommendations. Labor’s formal comments are reproduced in appendix V.

Labor also provided technical comments on the draft report, which we have incorporated where appropriate.

We are sending copies of the report to interested congressional committees and members, and the Secretary of Labor. We will also make copies available to others upon request. In addition, our report will be available at no charge on GAO’s Web site at http://www.gao.gov.

A list of related GAO products is included at the end of the report. If you or your staff has any questions about this report, please contact me at (202) 512-7215. You may also reach me by e-mail at nilsens@gao.gov. Key contributors to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to answer the following questions: 1. How do states identify unemployment claimants who are most likely to exhaust benefits? 2. To what extent do states provide reemployment services as recommended by Labor? 3. What is known about the effectiveness of the worker-profiling initiative in accelerating the reemployment of unemployment insurance claimants?

To answer the first question, we reviewed Labor’s guidance about the worker-profiling initiative, and reviewed literature and interviewed experts with the Department of Labor and the Upjohn Institute for Employment Research regarding profiling techniques. We also obtained and analyzed the results of a 2006 Department of Labor-sponsored survey of the 53 states and territories. This survey made numerous inquiries about the structural and operational aspects of the profiling tools—such as statistical models or characteristic screens—in use in the states. Finally, we contacted officials in 7 states—California, Delaware, Illinois, Kentucky, Texas, Washington, and Wisconsin. We selected some states to ensure that we included certain aspects of worker profiling; for example, we selected Kentucky because it had a very complex statistical model with numerous variables, and we selected Delaware because it was one of the few states that profiled claimants using a characteristic screen instead of a statistical model. We also selected these states because they ensured geographic dispersion and a range of populations sizes. In each of these states, we reviewed documents describing the profiling model that the state uses, and interviewed knowledgeable officials about the variables used in the model, the degree to which the model has been assessed and updated, and other matters.

To answer the second question, we reviewed Labor guidance regarding reemployment services provided to Unemployment Insurance (UI) claimants referred through the worker-profiling initiative, and obtained and analyzed national data collected by the Department of Labor from states on the Employment and Training Administration (ETA) 9048 Worker Profiling and Reemployment Services Activity report. In this report, states submit to Labor, by quarter, information such as the number of UI claimants profiled, referred to services, and completing services. During our contacts with the 7 states mentioned above, we also obtained and reviewed state documents describing policies about referral and reemployment services for claimants profiled under the worker-profiling initiative. We also interviewed knowledgeable state officials about these policies, including referral and notification of claimants, enforcement of participation requirements, and the type of reemployment services that are offered to claimants. In 6 of these states, we also contacted officials at local one-stop offices or regional offices to discuss how reemployment services are managed and delivered. In 4 of these states, we also attended the initial reemployment services session for claimants referred through the worker-profiling initiative and recorded our observations on a standard template.

To answer the third question, we identified and reviewed six research studies that evaluated the impact of profiling and the referral to services on claimant outcomes. All the studies used regression techniques to estimate the impact of a referral to services on a claimant’s UI claims experience or the subsequent earnings and employment activities. A GAO economist reviewed these studies and determined whether each study's findings were generally reliable by evaluating the methodological soundness of the studies and validity of the results and conclusions that were drawn. On the basis of this assessment, we determined that five of the six studies were methodologically rigorous enough to use in this report. We confirmed with Labor and national experts on unemployment insurance that these remaining five studies constituted the definitive work done to date on the impact of the worker-profiling initiative. Additionally, we reviewed these studies to assess the reemployment services offered under the worker-profiling initiative. Finally, we reviewed several studies on other work search programs that also evaluated impacts on claimant outcomes. We also obtained and analyzed national data collected by Labor from states via the ETA 9049 Worker Profiling and Reemployment Services Outcomes report. In this report, states report to Labor on a quarterly basis information on the outcomes of referred claimants, such as the average duration claimants received UI benefits and the number of claimants that found employment in the year following referral. Finally, in our contacts with the 7 states mentioned above, we interviewed knowledgeable officials regarding the data collected by Labor and their general views about the worker-profiling initiative, and in particular whether they believed the initiative was having the intended outcomes.

We conducted a data reliability assessment on the ETA 9048 and ETA 9049 reports data, which included electronically checking the data and interviewing Labor and state officials on the reliability of the data. On the basis of our reliability assessment and interviews, we found that some of the ETA 9048 and ETA 9049 reports had missing or inaccurate data. As a result, we took the following actions to ensure the accuracy of the data. First, because Labor instituted data edit checks starting in 2002, we limited the time frame of our analysis to 2002 to the most recent available, September 2006 and March 2005 for the ETA 9048 and ETA 9049, respectively. Second, we disregarded data from states that had excessive amounts of missing data reports. Specifically, from the ETA 9048, we excluded Louisiana, New Mexico, Puerto Rico, and the Virgin Islands, and for the ETA 9049, we also excluded Idaho and New Jersey, in addition to those states dropped for the ETA 9048. Third, we estimated data values, if possible, for states that had sporadically missing reports or data that were anomalous or illogical, for example, when the number of claimants who found employment exceeded the number referred to services. Of the data we reported from the ETA 9048 and ETA 9049, we estimated approximately 1 percent of these data; because of this small proportion, we believe that any errors arising from our estimation process did not significantly affect the state and national averages we reported. Some possible issues resulting from our estimation process were the following: We utilized logical relationships between data to estimate values, and at times, these values were based on other estimated data. Any errors resulting from the previous estimation would be carried over to the following estimated value.

Some states had volatile data, and as our estimation process was based on the existing state data, it is uncertain how accurate our estimates were.

At times, our estimated values were the highest or the lowest in the data series, and it is possible that the estimation procedure resulted in an inaccurate value.

Fourth, we excluded data from states that we confirmed were reported incorrectly. Specifically, for the ETA 9049, California and Georgia were excluded from calculations using the number of claimants who become employed, and Illinois was dropped from all analyses of both the ETA 9048 and ETA 9049 data. Last, we did not use any of the detailed reemployment services data, such as the number of claimants that completed an orientation, assessment, and so forth, because both Labor and state officials said these data were not comparable within and between states.

Appendix II: Average Percentage of Claimants Profiled, Referred to, and Completing Services for 2002-2006 and Average Claimant Outcomes for 2002-2005, by State

Length of time referred claimants receive UI benefits (2002-2006) (2002-2006) (2002-2005)

Referred claimants who become employed (2002-2005)

Length of time referred claimants receive UI benefits (2002-2006) (2002-2006) (2002-2005)

Referred claimants who become employed (2002-2005) (weeks) (2002- 2005)

GAO analysis based on 47 states and the District of Columbia.

Appendix III: Bibliography of Research Studies on the Worker-Profiling Initiative— Exhaustive List Identified from the Literature Review

Black, Dan A., Jeffrey A. Smith, Mark C. Berger, and Brett J. Noel. “Is the Threat of Reemployment Services More Effective than the Services Themselves? Evidence from Random Assignment in the UI System.” The American Economic Review, Vol. 93, No. 4 (November 2003).

Black, Dan A., Jose Galdo, and Jeffrey A. Smith. “Evaluating the Worker Profiling and Reemployment Services System Using a Regression Discontinuity Approach.” Paper presented at the American Economic Association conference in January 2007. Submitted to The American Economic Review for the May 2007 Papers and Proceedings Issue.

Dickinson, Katherine P., Suzanne D. Kreutzer, and Paul T. Decker. “Evaluation of Worker Profiling and Reemployment Services Systems: Report to Congress.” U.S. Department of Labor, Employment and Training Administration (March 1997).

Dickinson, Katherine P., Suzanne D. Kreutzer, Richard W. West, and Paul T. Decker. “Evaluation of Worker Profiling and Reemployment Services: Final Report.” U.S. Department of Labor, Employment and Training Administration Research and Evaluation Report Series 99-D (1999).

Noel, Brett J. “Two Essays on Unemployment Insurance: Claimant Responses to Policy Changes.” Dissertation submitted for the degree of Doctor of Philosophy at the Graduate School of the University of Kentucky, UMI Number: 9922624 (1998).

Appendix IV: Summary of the Impact of Referral to Services on Claimant Outcomes from the Literature Review

CT, ME, NJ: Reduced by 1.4 to 4.3 percentage points SC, KY: Increased by 1.1 to 4.1 percentage points Dates indicate when claimants filed their UI claim or received their first UI benefit payment.

Unpublished dissertation.

Delaware was included in this study, but its sample size was too small to detect any significant impacts.

Appendix V: Comments from the Department of Labor

Appendix VI: GAO Contacts and Acknowledgments

Staff Acknowledgments

Patrick di Battista, Assistant Director, and Michael Hartnett, managed this engagement.

Shannon Groff, Rosemary Torres Lerma, and Winchee Lin also made significant contributions throughout the engagement. Susan Bernstein helped develop the report’s message. Jay Smale, Stuart Kaufman, Rhiannon Patterson, Robert Dinkelmeyer, and Greg Dybalski contributed to the analysis of Labor data and reviews of external studies. Jessica Botsford provided legal support.

Related GAO Products

Workforce Investment Act: Employers Found One-Stop Centers Useful in Hiring Low-Skilled Workers; Performance Information Could Help Gauge Employer Involvement. GAO-07-167. Washington, D.C.: December 22, 2006.

Unemployment Insurance: States’ Tax Financing Systems Allow Costs to Be Shared among Industries. GAO-06-769. Washington, D.C.: July 26, 2006.

Unemployment Insurance: Enhancing Program Performance by Focusing on Improper Payments and Reemployment Services. GAO-06-696T. Washington, D.C.: May 4, 2006.

Unemployment Insurance: Factors Associated with Benefit Receipt and Linkages with Reemployment Services for Claimants. GAO-06-484T. Washington, D.C.: March, 15, 2006.

Unemployment Insurance: Factors Associated with Benefit Receipt. GAO-06-341. Washington, D.C.: March 7, 2006.

Workforce Investment Act: Labor and States Have Taken Actions to Improve Data Quality, but Additional Steps Are Needed. GAO-06-82. Washington, D.C.: November 14, 2005.

Unemployment Insurance: Better Data Needed to Assess Reemployment Services to Claimants. GAO-05-413. Washington, D.C.: June 24, 2005.

Workforce Investment Act: Labor Should Consider Alternative Approaches to Implement New Performance and Reporting Requirements. GAO-05-539. Washington, D.C.: May 27, 2005.

Unemployment Insurance: Information on Benefit Receipt. GAO-05-291. Washington, D.C. March 17, 2005.

Workforce Investment Act: Employers Are Aware of, Using, and Satisfied with One-Stop Services, but More Data Could Help Labor Better Address Employers’ Needs. GAO-05-259. Washington, D.C.: February 18, 2005.

Workforce Investment Act: States and Local Areas Have Developed Strategies to Assess Performance, but Labor Could Do More to Help. GAO-04-657. Washington, D.C.: June 1, 2004.

Workforce Investment Act: One-Stop Centers Implemented Strategies to Strengthen Services and Partnerships, but More Research and Information Sharing Is Needed. GAO-03-725. Washington, D.C.: June 18, 2003.