Unemployment Insurance: More Guidance and Evaluation of Worker-Profiling Initiative Could Help Improve State Efforts
Published: 2007-06-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Changes to the U.S. economy have led to longer-term unemployment. Many unemployed workers receive Unemployment Insurance (UI), which provided about $30 billion in benefits in 2006. In 1993, Congress established requirements--now known as the Worker Profiling and Reemployment Services (WPRS) initiative--for state UI agencies to identify claimants who are most likely to exhaust their benefits, and then refer such claimants to reemployment services. To assess the implementation and effect of the initiative, GAO examined (1) how states identify claimants who are most likely to exhaust benefits, (2) to what extent states provide reemployment services as recommended by the Department of Labor (Labor), and (3) what is known about the effectiveness of the initiative in accelerating reemployment. To answer these questions, we used a combination of national data; review of seven states, including visits to local service providers in four states; and existing studies and interviews with Labor and subject matter experts.
section_title: What GAO Found
paragraphs: Forty-five of the 53 states and territories use statistical models that facilitate the ranking of claimants by their likelihood to exhaust benefits, while 7 states use more limited screening tools that do not facilitate a ranking. Florida delegates the selection of profiling tools to local areas in the state. Factors used to determine the probability of exhaustion include a claimant's education, occupation, and job tenure. Many states have not regularly maintained their models, and as a result, the models in some states may not be accurately identifying claimants who are likely to exhaust benefits. Although Labor data provide a limited picture of states' implementation of the worker-profiling initiative, 6 of the 7 states we studied did not provide the in-depth approach to services as recommended by Labor. Overall, an average of 15 percent of profiled UI claimants were referred to reemployment services, and 11 percent completed these services between 2002 and 2006. Six of the 7 states we contacted referred claimants to services, held them accountable for attending the services, and provided an orientation. However, only 1 of the 7 states provided individualized needs assessments, and developed service plans, as recommended. Little is known about the effectiveness of the worker-profiling initiative as it is currently operating. Although studies using data from the 1990s generally indicated that claimants who were referred to services had reduced reliance on UI, there are no more up-to-date studies. Further, some of the program data collected by Labor are not reliable, and the data are not being used by Labor or states to evaluate the initiative.

section_title: Letter
paragraphs: Changes to the U.S. economy—including the contraction of entire  Changes to the U.S. economy—including the contraction of entire  industries as a result of changes in technology and overseas competition— industries as a result of changes in technology and overseas competition— have led to increases in the length of unemployment. Unemployed workers  have led to increases in the length of unemployment. Unemployed workers  are now less likely to be rehired by their previous employers and are at a  are now less likely to be rehired by their previous employers and are at a  greater risk of long-term unemployment than in the past. Over the past five  greater risk of long-term unemployment than in the past. Over the past five  decades, the average duration of unemployment has been gradually  decades, the average duration of unemployment has been gradually  increasing, so that during 2006, periods of unemployment grew to an  increasing, so that during 2006, periods of unemployment grew to an  average of 15 weeks, compared with 11 weeks during the 1950s. Many  average of 15 weeks, compared with 11 weeks during the 1950s. Many  unemployed workers receive temporary, partial wage replacement  unemployed workers receive temporary, partial wage replacement  through the Unemployment Insurance (UI) program. Under most state  through the Unemployment Insurance (UI) program. Under most state  programs, claimants can obtain regular benefits for up to 26 weeks. In  programs, claimants can obtain regular benefits for up to 26 weeks. In  order to help facilitate workers’ return to work, Congress established  order to help facilitate workers’ return to work, Congress established  requirements now known as the Worker Profiling and Reemployment  requirements now known as the Worker Profiling and Reemployment  Services initiative in 1993. Under this initiative, state UI agencies are  Services initiative in 1993. Under this initiative, state UI agencies are  required to identify those who are most likely to exhaust their benefits, a  required to identify those who are most likely to exhaust their benefits, a  process known as profiling, and refer them to reemployment services.  process known as profiling, and refer them to reemployment services.
Statistics on the UI program underscore the importance of addressing  Statistics on the UI program underscore the importance of addressing  benefit exhaustion. In 2006 about 7 million claimants received UI  benefit exhaustion. In 2006 about 7 million claimants received UI  payments, totaling about $30 billion. Of those claimants, about 35 percent  payments, totaling about $30 billion. Of those claimants, about 35 percent  used all the benefits available to them. If they had used 1 week less of  used all the benefits available to them. If they had used 1 week less of  benefits, it would have saved the state UI trust funds roughly $600 million.  benefits, it would have saved the state UI trust funds roughly $600 million.
You asked us to assess how the worker-profiling initiative has been  You asked us to assess how the worker-profiling initiative has been  implemented, and its effect on shortening the length of unemployment.  implemented, and its effect on shortening the length of unemployment.  Specifically, you wanted to know:  Specifically, you wanted to know:  1.  How do states identify unemployment claimants who are most  1.  How do states identify unemployment claimants who are most  likely to exhaust benefits?  likely to exhaust benefits?  2.  To what extent do states provide reemployment services as  recommended by Labor?  3.  What is known about the effectiveness of the worker-profiling  initiative in accelerating the reemployment of unemployment  insurance claimants?
To answer these questions, we used a combination of national data, in- depth site visits, existing studies, and interviews with subject matter  experts. We analyzed national data collected by the Department of Labor  (Labor) from states on the worker-profiling initiative, including data on the  50 states, as well as the District of Columbia, Puerto Rico, and the Virgin  Islands from a 2006 Labor-sponsored survey on models states use to  profile. We also analyzed data on profiling, reemployment services, and  outcomes that states report to Labor using the reemployment service  activity and outcomes reports maintained by Labor’s Employment and  Training Administration. In addition, we interviewed state officials in  seven states: California, Delaware, Illinois, Kentucky, Texas, Washington,  and Wisconsin. We selected the seven states to include the range of  approaches states take to identify and serve those likely to exhaust  benefits. We visited local service providers in four of these states. Our site  visit states were selected to provide a range of state experiences with the  worker-profiling initiative and to ensure variation in geography and  population size. We identified six studies examining the impact of the  worker-profiling programs and, after evaluating the methodological  soundness and the validity of the results and conclusions, determined that  five of the six studies were sufficiently rigorous to use in this report.  Further, we interviewed Labor officials and other experts on worker  profiling and UI and reviewed other reports, including academic and Labor  research on profiling systems, best practices, and the outcomes of profiled  UI claimants. We performed our work in accordance with generally  accepted government auditing standards between August 2006 and April  2007.
subsections: section_title: Background
paragraphs: Beginning in the mid-1970s, major structural changes took place in the  American economy, as advances in technology, international competition,  plant closings, and corporate streamlining resulted in the dislocation of  thousands of workers from their jobs. Some of these individuals possessed  skills that were no longer in demand; others suffered from a lack of job  search skills. In the 1980s and early 1990s, demonstration projects were  conducted in New Jersey, Nevada, Minnesota, and Washington. The New  Jersey and Minnesota projects showed the efficacy of using statistical  methods and administrative data to identify those who are likely to  experience long periods of joblessness. For example, the New Jersey  demonstration project screened claimants with various eligibility  requirements and found that the screening allowed the state to direct  services to those who generally faced reemployment difficulties. Further,  results from all four states showed that providing more intensive job  search assistance to this population reduced the duration of insured  unemployment and UI expenditures.
In response to these events, the Clinton administration proposed  legislation to implement worker profiling in 1993. In the same year,  Congress enacted the Unemployment Compensation Amendments,  amending the Unemployment Insurance program legislation. The law  requires that states establish and utilize a system of profiling all new  claimants for UI regular compensation. The system must identify those  claimants that will be likely to exhaust regular compensation and refer  them to reemployment services, such as job search assistance services.  Typically, such claimants receive services at one of the local “one-stop”  employment services centers that exist throughout the nation. States are  required to collect information on the type of services claimants receive,  their participation, and their subsequent employment outcomes. The last  could include such information as whether referred claimants obtained  new jobs and the related wage levels.
In 1994, Labor issued guidance to help states establish profiling tools and  provide necessary reemployment services. In profiling claimants, Labor  required that states consider factors that include whether the claimant has  a date for being recalled to work, union status, first unemployment benefit  payment, and previous industry or occupation of employment. Labor  recommended also considering some additional factors such as claimants’  education, tenure at previous job, and the state unemployment rate.
Labor outlined recommended processes for providing reemployment  services to referred claimants, including (1) an orientation session for  claimants that would, among other things, explain the availability and  benefit of reemployment services; (2) an assessment of the specific needs  of each claimant, if appropriate; and (3) based on the assessment,  development of an individual plan for services that would guide a  claimant’s further services. (See fig. 1.) Under the law, states must also  require that claimants who have been referred to reemployment services  participate in those services as a condition of eligibility for receiving  compensation.
Labor may withhold UI administrative grants from a state if Labor finds,  after notice and an opportunity for a hearing, that a state has failed to  comply with worker-profiling requirements.  These include identifying  claimants most likely to exhaust benefits, referring claimants to  reemployment services, and collecting follow-up information on services  received and subsequent employment outcomes.
The law required that Labor report to Congress on the operation and  effectiveness of the profiling system within three years of its enactment.  Labor issued a report to Congress in March 1997, and published a final  report in 1999 on the program’s implementation and operation nationwide  and the effectiveness in six early implementation states. Labor has  published no studies on the effectiveness of the initiative since then. The  agency’s strategic plan for fiscal years 2006 through 2011, in providing an  overview of program evaluation, includes no ongoing or future research  topics addressing the impact of the worker-profiling initiative. Labor has  conducted impact evaluations as part of its program evaluations in the  past. In fiscal year 2007, Labor was appropriated $17.7 million for pilots,  demonstrations, and research.
Funding for the worker-profiling program is provided from a variety of  sources. Federal funding for the creation and maintenance of profiling  models can come from UI administrative funds, which are financed by a  federal UI tax on employers. Reemployment services can be funded  through a variety of sources. For example, states can use Wagner-Peyser  Employment Services grants as well as other state sources of funding to  provide reemployment services to profiled UI claimants. From 2001 to  2005, Labor also provided Reemployment Services grants to all states in  order to enhance and target services to claimants through the nation’s  network of one-stop employment service centers.
section_title: Most States Use Statistical Models to Identify Claimants Likely to Exhaust Benefits, but Many Have Not Updated Them to Account for Changing Economic Conditions
paragraphs: The large majority of states use statistical models to identify  unemployment recipients who are most likely to exhaust benefits.  However, many states have not recently adjusted their models, risking the  possibility that these models may lose predictive accuracy over time.  Forty-five states use statistical models to identify and rank clients by their  likelihood to exhaust benefits, while 7 states use characteristic screens  that do not rank claimants. One state—Florida—allows the local areas to  decide whether to use statistical models or screening tools. Among the  states using statistical models, the detailed specifications of these models  vary considerably from state to state. Further, many states do not regularly  update their models, a fact that can lead to a loss of predictive accuracy  over time. A survey of the states reveals that many have not revised or  updated their models in many years. Officials in states we contacted  explained that they face a number of impediments to doing so.
subsections: section_title: Most States Use Statistical Models instead of More Limited Characteristic Screens
paragraphs: Under worker profiling, state UI agencies are to identify claimants who are  most likely to exhaust benefits in two steps. States screen claimants in  order to eliminate claimants who are unemployed but job-attached or  would otherwise not qualify for referral to services from the profiling  process. After the initial screening, states profile remaining claimants— that is, they consider a range of personal and economic variables related  to a claimant and determine whether or not he or she is likely to exhaust  benefits. Although states have considerable flexibility in determining what  variables to use, Labor has recommended the use of five variables, as  outlined in table 1.
We found that states used one of two methods to identify claimants who  are most likely to exhaust benefits—the statistical model or  characteristics screening. Both of these look at a range of personal and  economic factors that help predict exhaustion.
Forty-five of the 53 states and territories use statistical models to identify  clients likely to exhaust benefits. (See fig. 2.) Using various statistical  techniques, these models consider the combined quantitative influence of  various personal and labor market characteristics and produce a  measurement of a claimant’s likelihood to exhaust. In statistical models,  each characteristic—commonly referred to as a variable—is associated  with a specific mathematical weight that quantifies the variable’s  contribution to the claimant’s probability of exhaustion. If, for example, a  claimant’s last job was in a steeply declining industry, the industry variable  would have a positive effect on the score, indicating a claimant’s  likelihood to exhaust. Conversely, if a claimant’s last job was in an  expanding industry, it would have the opposite effect. Essentially a  statistical model produces a weighted average of the effect of all the  variables combined. As a result, states that use statistical models can rank  claimants from greatest to least likelihood of exhaustion, and target  reemployment services to claimants with the greatest likelihood of  exhausting. According to an official of the Upjohn Institute for  Employment Research, such models, if properly developed and  maintained, are a powerful and effective means of identifying particular  populations for a range of social service programs.
Seven of the 8 remaining states use characteristic screens that do not  allow them to rank claimants. One state, Florida, delegates the selection of  profiling tools to the local areas because state officials believe profiling  can be done more accurately at that level. Like statistical models,  characteristic screens may consider various factors associated with the  likelihood to exhaust benefits, but treat them as yes-no decision points.  Either the claimant has the attribute or does not. The relative importance  of any one variable in relation to others is not considered. Claimants  selected through this process must have each of the screening criteria. For  example, the characteristic screen used by Delaware considers whether or  not a claimant meets specific criteria relating to industry, occupation, and  job tenure. In Delaware, a claimant passes the job tenure screen if he or  she has 2 or more years of tenure with his or her last employer. However,  since claimants cannot be ranked, states using screens must develop a  method, such as random assignment, to refer identified claimants to  services if they are unable to serve all claimants that pass the screens. For  example, Delaware used to refer claimants who passed the screen on a  random basis, but now refers all claimants who pass the screen. Labor  encourages the use of statistical models over characteristic screens  because they are more efficient and precise in identifying claimants likely  to exhaust.
section_title: Statistical Models Vary in Key Elements
paragraphs: Although all statistical models are supposed to identify claimants who are  likely to exhaust benefits, the states can vary in how they specifically  define this exhaustion. The model originally proposed by Labor is  designed to predict the probability of exhaustion as a yes-or-no outcome— exhaustion or no exhaustion—and the claimant’s profiling score would  reflect the probability of the yes outcome. Most states have adopted this  definition. However, as Labor explained in 1998 guidance, this approach  does not distinguish between claimants who almost exhaust benefits and  those who do not come close to exhausting benefits. This is significant,  because the claimant with nearly exhausted benefits may be in greater  need of reemployment services than the clamant who uses a  comparatively small portion of his or her benefits. Consequently, some  states predict exhaustion as the amount of benefits a claimant will  potentially use. For example, the profiling score produced by Kentucky’s  model produces a number between 1 and 20. A claimant with a score of   20 is likely to use 95 to 100 percent of benefits; a claimant with a score of  19 is likely to use 90 to 95 percent of benefits, and so on.
State models can differ considerably in how they define similar variables,  including those corresponding to the factors recommended by Labor. For  example, California uses six categories to measure the job tenure variable,  ranging from 1 year or less on the low end to more than 15 years on the  high end. In contrast, Texas uses only two categories—job tenure of less  than 1 year on the low side and tenure of more than 10 years on the high  end. The Kentucky model, on the other hand, measures job tenure on a  continuous scale—specifically, the length of time that a claimant held his  or her last job. The definitions of variables associated with education,  industry, and other variables can also differ among state models. For  example, Kentucky includes “completed vocational education” as part of  its education variable, while Wisconsin does not.
The number and nature of the additional variables can also differ  significantly by state. The large majority of states using statistical models  (34 of 45) use models that consider factors in addition to the five factors  recommended by Labor, while about one-quarter do not. (See fig. 2.)  Among the 6 states that we contacted that use statistical models, the  number of additional variables used ranged from 1 in California to over   50 in Kentucky. For example, 2 of the 7 states we contacted—Texas and  Illinois—consider the time lapse between the loss of a job and the  application for UI benefits. According to Texas officials and Labor, delays  in filing a claim are indicative of a difficult job search, thus increasing the  likelihood of benefit exhaustion.
section_title: Although Models Require Periodic Maintenance to Ensure Predictive Accuracy, Many States Have Not Updated Their Models since 2000 or Before
paragraphs: While Labor has recommended that states update models periodically to  reflect changes in economic conditions, many states have not done so in  many years. If not periodically updated, statistical models can lose  predictive accuracy over time because of changes in the labor market, the  general economy, or other factors. Labor has emphasized the importance  of updating models, and noted in 1998 guidance that models represent the  historical period in which they were developed, and that old models  become increasingly unrealistic and less useful over time. Labor has  further recommended that models be assessed, and if necessary adjusted,  approximately every 3 years. Officials in some of the 7 states we contacted  also stressed the importance of updating models from time to time. For  example, Washington officials noted that although a 2002 analysis of their  model update showed that it accurately identified the majority of  claimants who exhausted, this adjustment of their model was based on  data collected in 1999 and 2000, and subsequent changes in their labor  market and the general economy have made the model outdated. Also, a  2003 California study found that the state’s model underestimated benefit  exhaustion and recommended an update to the model. Similarly, an  official of the Upjohn Institute for Employment Research told us that the  institute’s analysis of 1 state’s model found that before the model was  updated, its results were little better than random selection of claimants.  Officials in Washington and California told GAO that the models would be  updated in the next year.
Models can be adjusted by modifying the mathematical weights associated  with specific variables, and by adding, deleting, or redefining variables to  enhance a model’s predictive power. This is necessary over time because,  although a particular variable—such as a claimant’s industry—can remain  an important predictor of exhaustion, its relative importance in the model  can change significantly. For example, if a variable’s weight was estimated  based on data from a historical period of large changes of employment  levels in a particular industry or industries, the model might produce  misleading results if used in a period of greater industrial and employment  stability. Similarly, a variable that once served as an important predictor in  a model may lose predictive value as the labor market and economic  circumstances change, and conversely, other variables that may not have  been relevant in one time period may become important at another time.  For example, Texas deleted education as a variable from the model used  in that state. According to a Texas official, statistical work performed for  the model update revealed that the education variable did not measurably  add to its predictive power.
Factors other than the labor market and general economy can affect the  reliability of models as well. For example, in the past 10 years  standardized coding used to identify both industries and occupations has  changed, and some of the states we contacted had not updated their  model to reflect this change. Illinois’ analysis of its model showed that  while the model had generally retained predictive accuracy, areas of  concern existed. For example, as a result of outdated occupational codes,  certain occupations associated with greater likelihood of exhaustion were  no longer being targeted, while others not associated with exhaustion  were.
Although Labor has taken a number of actions to encourage and assist  states in updating their profiling models, some states have not done so for  many years. Labor has noted the importance of updating models in written  guidance, sponsored occasional seminars where best modeling practices  are shared with state staff, and provides on-demand technical assistance  to states. However, Labor has not established requirements for updating  models, and has not undertaken ongoing monitoring of state models. A  recent Labor-commissioned survey revealed that many states have not  updated their profiling models in recent years. (See figs. 3 and 4.) For  example, although 21 states reported taking actions such as adjusting  variable weights since 2003, many others have not. Specifically, 18 states  have not done so since 1999 or before, and 12 of these reported never  having done so.
According to Labor’s survey results, states have been even less inclined to  adjust their models by taking actions such as changing or redefining  variables in the models. As figure 4 shows, 30 states reported that they had  not made such changes since implementation, and 23 states reported  having done so. Only 11 of these 23 states reported having done so since  2003.
Labor’s survey did not inquire about factors influencing the frequency with  which states update their models, but our contacts with 7 states reveal a  variety of reasons that some states have not updated their models.  Officials in California said that they had more pressing priorities for UI  administrative funds, and thus would have difficulty funding model  updates. Wisconsin officials said that revising the models required  expertise that they did not have, either in-house or from other sources,  such as a state university. Although Labor provides technical guidance and  advice, and has offered seminars on updating models, state officials  indicated they still need more continuous access to expertise in order to  keep models updated. A Texas official said that sometimes historical data  needed to determine a variable’s impact on exhaustion of benefits are not  available, and so the variable cannot be included in the model. Relatedly, if  the necessary data on claimants are not collected, or cannot be  transmitted and used by the model, the related variable cannot be used.  For example, a Texas official told us that certain variables, such as the  number of a claimant’s dependents or spousal income, might be good  predictive variables. However, the standard Texas application form for UI  benefits does not ask about the number of dependents or spousal income,  so these variables cannot be used.
section_title: Most Study States Did Not Take the In-Depth Approach Recommended by Labor to Ensure That Profiled Claimants Obtain Reemployment Services
paragraphs: Labor data provide a limited picture of states’ implementation of worker  profiling, and some aspects of these data were not reliable. Further, 6 of  our 7 study states did not offer the in-depth approach to services  prescribed by Labor. These states generally referred claimants to services,  held them accountable for attending the services, and provided them with  an orientation and some instruction on job search skills. However, 6 of the  7 states did not adhere to Labor’s guidance recommending an in-depth  individual needs assessment and a tailored reemployment service plan for  referred UI claimants.
subsections: section_title: Data Collected by Labor Provided a Limited Picture of States’ Implementation of Reemployment Services
paragraphs: Between 2002 and 2006, about 94 percent of the UI claimants who received  a first payment were profiled. To the extent that reemployment services  are available, Labor requires that states refer profiled claimants to these  services. Of those profiled, an average of 15 percent were referred to  services, with states ranging from 5-year averages of 1 percent (Wyoming)  to 52 percent (Washington) (See fig. 5.) While 3 states referred between   29 and 52 percent of profiled claimants to services, 28 states referred 14  percent or fewer. Further, of those claimants profiled, an average of 11  percent completed services, with states ranging from 1 percent (Arkansas,  Colorado, Idaho, Michigan, and Wyoming) to 39 percent (Texas). (See fig.  6.) In 2 states, more than 27 percent of profiled claimants completed  services. However, in 33 states, 13 percent or fewer of claimants did so.
See appendix II for the average percentages of profiled claimants referred  to and completing services by state from 2002 to 2006.
Labor’s data are not sufficiently reliable to provide any information on the  specific services provided to claimants—such as orientation, counseling,  job search workshops, or job clubs. Specifically, Labor and state officials  told us that definitions of these services can vary across states and within  states over time as they change the content of their programs. For  example, California officials told us that the state’s definitions of services  provided were established over 10 years ago and that the nature of the  services may have changed since then.
section_title: Six of Seven States We Studied Referred Claimants and Enforced Compliance with Referrals
paragraphs: We found that 6 of the 7 study states had, as required by Labor, referred  profiled claimants to services and made claimants ineligible for benefits if  they failed to attend reemployment services. In contrast, officials in 1 state  told us that referrals had been delegated to local workforce areas, and that  they did not know whether claimants were being referred to services  statewide. We subsequently contacted some local workforce development  offices in this state and learned that several had not been referring UI  profilees to reemployment services for years. In addition, officials in this  state told us that there are no consequences for those who fail to attend  reemployment services. They further said they do not track information at  the state level on whether claimants attend services. While Labor requires  that states hold claimants ineligible for benefits for any week in which  they fail to attend services, Delaware goes further and holds the UI  benefits of claimants who do not attend services until they reschedule.
Some of the study states took additional steps to ensure compliance with  service referrals, while others did not. Of the states that referred claimants  to services, Delaware and Washington required that claimants reschedule  if they failed to attend required services, while Texas and Wisconsin  attempted to reschedule claimants in some cases and the remaining states  did not do so. Officials in Delaware reported that they go so far as to  have staff call claimants early during the week of their scheduled  orientation to remind them to attend; officials in Washington said that  some local workforce centers do this. Officials cited the large flow of  claimants into the program, the complexity of the rescheduling process,  and the scarcity of staff resources as reasons they did not reschedule  referred claimants.
section_title: Six of Seven States We Studied Provided Limited Reemployment Services and Did Not Develop Individual Assessments and Service Plans
paragraphs: The reemployment services offered in the states we contacted generally  did not conform to the robust service process originally outlined by Labor.  Labor’s 1994 guidance states that after initial orientation, the service  provider should determine the specific needs of each worker through an  assessment process, such as vocational testing. Only one of our study  states, Delaware, required that case managers conduct an initial  assessment to determine what services claimants might need, such as  Workforce Investment Act (WIA) training, depending on their job  readiness level. Washington and Wisconsin required that claimants  complete a self-assessment. For example, claimants at one one-stop center  were expected to complete a one-page self-assessment that asks questions,  including what educational level they attained, whether they had a current  résumé, and whether they had difficulty filling out a job application. The   4 other states we studied required no assessment of any kind.
According to state officials, our study states also generally did not require,  as recommended by Labor, that local offices develop or document a  reemployment services plan that could serve as the basis for determining  satisfactory participation. Only Delaware required case managers to  develop service plans and meet with claimants on a monthly basis after  each claimant’s assessment. In California and Wisconsin, claimants  developed their own plans, which involved selecting an additional service  session on a topic the claimants felt would be most helpful. For example,  California required that UI claimants attend an orientation and choose an  additional service, such as a WIA service or job club, that would constitute  their individual reemployment plan.
All 7 study states cited lack of or declining funding as an issue that  affected the provision of reemployment services. Specifically, some states  mentioned the loss of Labor’s Reemployment Services grants, which had  been awarded to all states between 2001 and 2005 to enhance and target  integrated core services to claimants through the one-stop centers and  were used by some states to fund program-related services. A Wisconsin  official said that when the grant funds end in summer 2007, the state  would only be providing worker profiling services in 6 to 12 of its 75 local  workforce development offices. State officials also mentioned continuing  declines in Wagner-Peyser, or Employment Services, funding. A local  workforce manager in Washington said that there is a vast gap between  the need for services and the resources and that the state only has  resources for about 5 percent of the 50,000 to 60,000 UI claimant  population. In order to help address this issue, officials in Washington told  us that a special surtax is applied to UI taxes, and a small portion of this is  diverted to worker-profiling service activities. While state officials were  concerned with the availability of funding, Labor officials said that the  purpose of the worker-profiling initiative is to target the funding that does  exist to those claimants who need it most and that the program does not  mandate that states serve any claimants they did not serve prior to its  implementation.
Officials also cited various day-to-day challenges in providing effective  reemployment services. A single services session can include claimants  ranging from former upper management employees to construction and  factory production workers, according to a Kentucky official. The same  official said that pitching the class so that it is effective for both types of  claimants can be difficult. Claimants’ language skills also can be a  challenge. However, California addresses this by excusing non-English  speakers from the session, and directing them to job service centers or  community-based partners that provide reemployment services in their  own language, unless the orientation is available in their native language.
section_title: Little Is Known about Program’s Effectiveness because There Are No Current Studies and Labor’s Data Are of Limited Usefulness
paragraphs: Little is known about the current effectiveness of the worker-profiling  initiative. Research studies, while generally finding that profiling and a  referral to services had a positive impact on claimants, used data from the  early implementation of the initiative—1994 to 1996. Although Labor  collects data on the outcomes of those profiled and referred to services,  we found portions of it to be unreliable. In addition, state officials said  they do not use Labor’s data for evaluation purposes.
subsections: section_title: Early Research Showed Some Positive Outcomes for Those Referred to Services, but There Are No Current Studies
paragraphs: Five methodologically sound studies looking at the impacts of the worker- profiling initiative after it was first implemented found that the program  had some desired effects. Examining data from 1994 to 1996, the studies  generally indicated that a referral to services under worker profiling led to  a reduction in claimants’ duration on UI, a reduction in the amount of UI  benefits that were paid out, and an increase in subsequent employment  earnings. Though the methodologies varied, all the studies evaluated the  impacts of the referral to services using statistical analyses to compare the  outcomes of claimants who were referred to services against those of  claimants who were not. As table 3 indicates, these studies cover a total  of only 7 states, and no national study exists. Further, no study using  current data exists. Labor sponsored the two multistate studies published  in 1997 and 1999, but has not published any subsequent studies.  According to Labor officials, the agency has no current plans to study the  effects of profiling. Because data in all the studies were from the period  when worker profiling was first implemented, the profiling process and  reemployment services provided then may not reflect what states are  currently offering.
While these early studies showed positive impacts for referred claimants  with regard to reducing duration, reducing amount of UI benefits, and  increasing employment earnings, there were mixed results for whether the  program reduced the percentage of claimants who exhausted their  benefits or improved subsequent employment rates (See table 4.)  According to the studies, claimants who were referred to services had a  decreased UI duration and received lower total amounts of UI benefits.  Most of the studies found that claimants who were referred to services  increased earnings in the year following the UI claim. However, the largest  multistate study was unable to draw any conclusions about the impact on  earnings because of contradictory data. Evidence that a referral to  services reduced the percentage of claimants who exhausted their UI  benefits was mixed. For example, one study showed a decrease in the  percentage of claimants who exhausted their UI benefits in 3 states, but an  increase in 2 states. The effect of a referral to services on employment  rates was also inconclusive. According to the two multistate studies, the  effect was minimally positive for one state, but the other 6 states showed  insignificant or contradictory results. Most of the studies, however, did not  examine subsequent employment rates.
Research studies of other work search programs corroborate the generally  favorable results found in the impact evaluation studies of the worker- profiling initiative. Though the methodologies varied, these studies  demonstrated that work search assistance reduced the duration claimants  received UI benefits, among other beneficial impacts. In two  demonstration projects, UI claimants who received job search assistance  received fewer weeks of UI benefits. The reemployment services offered  in these demonstration projects, however, were more robust; for example,  in one study, claimants were required to attend an orientation, testing, a  job search workshop, and a one-on-one assessment interview. As such,  they may not reflect what is offered through the states’ worker-profiling  programs currently.
Even though they were unable to provide supporting data, officials from  our study states said that worker profiling was a useful program for UI  claimants. They said it had enabled states to advertise their job search and  training services and target claimants who are most likely to exhaust their  UI benefits. In the process of referring claimants to services, states are  also educating the community on the many services and resources  available at the one-stop service centers. They also said the initiative was a  way to focus resources on those who would benefit from job search  assistance the most.
section_title: Outcomes Data Collected by Labor Are Limited and Not Consistently Used for Evaluation Purposes
paragraphs: Due to reliability issues, Labor’s claimant outcomes data are of limited  value. Labor’s claimant outcomes data were sufficiently reliable for us to  report only certain outcomes, including benefits exhaustion, weeks of  benefit receipt, and reemployment. Those data showed that less than half  of profiled claimants exhausted benefits, that on average they received  benefits for about two-thirds of the typical maximum time allowed, and  that about half found employment within 1 year of the referral to services  (see table 5).
In addition to reliability issues, other characteristics, such as the lack of a  comparison group and long time lags, limit the usefulness of both the  reemployment services data and claimant outcomes data for states. First,  the outcomes data reflected only the experience of those who were  referred to services, and did not include an adequate point of comparison.  It was therefore impossible to know if these outcomes were different than  they would have been had the claimants not been referred to or completed  reemployment services. Second, according to Labor officials, the data  were originally intended for states to evaluate the effectiveness of the  worker-profiling initiative. However, we found that neither Labor nor the  states used the data for this purpose. Several state officials said the time  lag and aggregated nature of the data were insufficient for program  management purposes. The claimant outcomes data were not reported for  more than a year after claimants were referred to services, and some state  officials said they needed more timely data. Both the reemployment  services and claimant outcomes data were aggregated to the state level,  and some state officials said that local-level data would better meet their  management needs. Four of our seven study states indicated that they did  not utilize the reemployment services data or claimant outcomes data, and  some only reported them because it was required by Labor; the remaining  states said they used the reemployment services data for nonevaluative  purposes, such as determining how many services were provided to  claimants or the volume of claimants served under the worker-profiling  program.
In light of these data limitations, several state officials said they developed  their own program performance measures and reports instead of using the  reemployment services data and claimant outcomes data. For example,  Washington developed its own data warehouse system that links data on  UI benefits, reemployment services, and claimant wages. According to  officials, on a monthly basis they review performance indicators, such as  the number of UI claimants that find employment and the amount of time  it takes before finding employment.
section_title: Conclusions
paragraphs: Our findings suggest that although states continue to profile and refer  claimants to reemployment services, the worker-profiling initiative is not a  high priority at the federal level or in many states. In the past Labor has set  out broad guidelines for states on the design and maintenance of profiling  models. However, our analyses indicate that these have been inadequate.   Labor’s 2006 survey of state profiling techniques revealed that many states  had not updated their profiling models for many years. As a result, it is  possible that many models have lost predictive accuracy, and are referring  claimants to services who are not in need of them, or failing to refer  claimants that are in need of them. However, the worker-profiling program  is required by law, and if there is to be a continued federal mandate, it may  be that a more assertive federal role is necessary to ensure the integrity of  those models.
A long time has passed since Labor articulated its vision of reemployment  services, and our review of seven states indicates that what is being  practiced is a diminished version of that vision. While the states we  studied indicated they provided orientation sessions that seemed to  convey important information, including job search skills, Labor’s  guidance implies a more tailored and in-depth approach to services. It may  be that the original vision is no longer realistic or perhaps, in the states’  experience, necessary. Absent clarification at the federal level, it will  remain unclear what Labor expects from the states.
The national data on the worker-profiling initiative is of very limited  usefulness as a measure of program activity, outcomes, and effectiveness.  Many of the data are not usable because of inconsistent or incorrect  reporting, and neither Labor nor the states we contacted use the data for  evaluating the worker-profiling initiative. Further, even if all the outcomes  data were reported consistently and accurately, these data cannot, by  themselves, be used to measure the impact of the program. In the end, by  requiring the submittal of data that are of such limited reliability and value,  Labor is potentially wasting both its own and the states’ resources. Finally,  absent information about the program’s current impact, Labor may find it  more difficult to make decisions regarding the best means for returning  the unemployed to work more quickly.
section_title: Recommendations for Executive Action
paragraphs: To better ensure that claimants who need and could benefit from  reemployment services are referred, and to ensure that resources are not  unnecessarily expended on claimants not needing them, we recommend  that the Secretary of Labor:  1.  Reevaluate the agency’s worker-profiling data collection to  determine whether it is sufficient for its intended purpose. The  agency might assess gaps in data, evaluate data consistency, confer  with states on what data would be beneficial to them, determine  the purpose of the data collection and for whose benefit the data  are collected, and modify what Labor requires states to collect.  2.  Ensure that the Employment and Training Administration takes a  more active role to help ensure the accuracy of the state profiling  models. The agency might track states’ management of their  models and actively encourage review and updating of models in  specific states where there have been no efforts to adjust the  model for a number of years. The agency could also assess whether  an expanded technical assistance effort is needed, and, if so, take  the lead in developing one.   3.  Encourage states to adhere to Labor’s vision for in-depth  reemployment services, such as conducting individualized needs  assessments and developing individual service plans, or issue  updated guidance if this original vision would be too burdensome  for the states.   4.  Evaluate the impact of the worker-profiling program on the  reemployment of UI recipients to ensure the benefits are  commensurate with the resources invested.
section_title: Agency Comments
paragraphs: We provided a draft of this report to Labor for review and comment. In  general, Labor agreed with our findings and recommendations. Labor’s  formal comments are reproduced in appendix V.
Labor also provided technical comments on the draft report, which we  have incorporated where appropriate.
We are sending copies of the report to interested congressional  committees and members, and the Secretary of Labor. We will also make  copies available to others upon request. In addition, our report will be  available at no charge on GAO’s Web site at http://www.gao.gov.
A list of related GAO products is included at the end of the report. If you  or your staff has any questions about this report, please contact me at  (202) 512-7215. You may also reach me by e-mail at nilsens@gao.gov. Key  contributors to this report are listed in appendix VI.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to answer the following questions:  1.  How do states identify unemployment claimants who are most  likely to exhaust benefits?  2.  To what extent do states provide reemployment services as  recommended by Labor?  3.  What is known about the effectiveness of the worker-profiling  initiative in accelerating the reemployment of unemployment  insurance claimants?
To answer the first question, we reviewed Labor’s guidance about the  worker-profiling initiative, and reviewed literature and interviewed  experts with the Department of Labor and the Upjohn Institute for  Employment Research regarding profiling techniques. We also obtained  and analyzed the results of a 2006 Department of Labor-sponsored survey  of the 53 states and territories. This survey made numerous inquiries  about the structural and operational aspects of the profiling tools—such  as statistical models or characteristic screens—in use in the states. Finally,  we contacted officials in 7 states—California, Delaware, Illinois, Kentucky,  Texas, Washington, and Wisconsin. We selected some states to ensure that  we included certain aspects of worker profiling; for example, we selected  Kentucky because it had a very complex statistical model with numerous  variables, and we selected Delaware because it was one of the few states  that profiled claimants using a characteristic screen instead of a statistical  model. We also selected these states because they ensured geographic  dispersion and a range of populations sizes. In each of these states, we  reviewed documents describing the profiling model that the state uses,  and interviewed knowledgeable officials about the variables used in the  model, the degree to which the model has been assessed and updated, and  other matters.
To answer the second question, we reviewed Labor guidance regarding  reemployment services provided to Unemployment Insurance (UI)  claimants referred through the worker-profiling initiative, and obtained  and analyzed national data collected by the Department of Labor from  states on the Employment and Training Administration (ETA) 9048  Worker Profiling and Reemployment Services Activity report. In this  report, states submit to Labor, by quarter, information such as the number  of UI claimants profiled, referred to services, and completing services.  During our contacts with the 7 states mentioned above, we also obtained  and reviewed state documents describing policies about referral and  reemployment services for claimants profiled under the worker-profiling  initiative. We also interviewed knowledgeable state officials about these  policies, including referral and notification of claimants, enforcement of  participation requirements, and the type of reemployment services that are  offered to claimants. In 6 of these states, we also contacted officials at  local one-stop offices or regional offices to discuss how reemployment  services are managed and delivered. In 4 of these states, we also attended  the initial reemployment services session for claimants referred through  the worker-profiling initiative and recorded our observations on a  standard template.
To answer the third question, we identified and reviewed six research  studies that evaluated the impact of profiling and the referral to services  on claimant outcomes. All the studies used regression techniques to  estimate the impact of a referral to services on a claimant’s UI claims  experience or the subsequent earnings and employment activities. A GAO  economist reviewed these studies and determined whether each study's  findings were generally reliable by evaluating the methodological  soundness of the studies and validity of the results and conclusions that  were drawn. On the basis of this assessment, we determined that five of  the six studies were methodologically rigorous enough to use in this  report. We confirmed with Labor and national experts on unemployment  insurance that these remaining five studies constituted the definitive work  done to date on the impact of the worker-profiling initiative. Additionally,  we reviewed these studies to assess the reemployment services offered  under the worker-profiling initiative. Finally, we reviewed several studies  on other work search programs that also evaluated impacts on claimant  outcomes. We also obtained and analyzed national data collected by Labor  from states via the ETA 9049 Worker Profiling and Reemployment Services  Outcomes report. In this report, states report to Labor on a quarterly basis  information on the outcomes of referred claimants, such as the average  duration claimants received UI benefits and the number of claimants that  found employment in the year following referral. Finally, in our contacts  with the 7 states mentioned above, we interviewed knowledgeable  officials regarding the data collected by Labor and their general views  about the worker-profiling initiative, and in particular whether they  believed the initiative was having the intended outcomes.
We conducted a data reliability assessment on the ETA 9048 and ETA 9049  reports data, which included electronically checking the data and  interviewing Labor and state officials on the reliability of the data. On the  basis of our reliability assessment and interviews, we found that some of  the ETA 9048 and ETA 9049 reports had missing or inaccurate data. As a  result, we took the following actions to ensure the accuracy of the data.  First, because Labor instituted data edit checks starting in 2002, we limited  the time frame of our analysis to 2002 to the most recent available,  September 2006 and March 2005 for the ETA 9048 and ETA 9049,  respectively. Second, we disregarded data from states that had excessive  amounts of missing data reports. Specifically, from the ETA 9048, we  excluded Louisiana, New Mexico, Puerto Rico, and the Virgin Islands, and  for the ETA 9049, we also excluded Idaho and New Jersey, in addition to  those states dropped for the ETA 9048. Third, we estimated data values, if  possible, for states that had sporadically missing reports or data that were  anomalous or illogical, for example, when the number of claimants who  found employment exceeded the number referred to services. Of the data  we reported from the ETA 9048 and ETA 9049, we estimated  approximately 1 percent of these data; because of this small proportion,  we believe that any errors arising from our estimation process did not  significantly affect the state and national averages we reported. Some  possible issues resulting from our estimation process were the following:    We utilized logical relationships between data to estimate values,  and at times, these values were based on other estimated data. Any  errors resulting from the previous estimation would be carried over  to the following estimated value.
Some states had volatile data, and as our estimation process was  based on the existing state data, it is uncertain how accurate our  estimates were.
At times, our estimated values were the highest or the lowest in the  data series, and it is possible that the estimation procedure resulted  in an inaccurate value.
Fourth, we excluded data from states that we confirmed were reported  incorrectly. Specifically, for the ETA 9049, California and Georgia were  excluded from calculations using the number of claimants who become  employed, and Illinois was dropped from all analyses of both the ETA 9048  and ETA 9049 data. Last, we did not use any of the detailed reemployment  services data, such as the number of claimants that completed an  orientation, assessment, and so forth, because both Labor and state  officials said these data were not comparable within and between states.
section_title: Appendix II: Average Percentage of Claimants Profiled, Referred to, and Completing Services for 2002-2006 and Average Claimant Outcomes for 2002-2005, by State
paragraphs: Length of  time referred  claimants receive  UI benefits  (2002-2006) (2002-2006) (2002-2005)
Referred  claimants  who become  employed  (2002-2005)
Length of  time referred  claimants receive  UI benefits  (2002-2006) (2002-2006) (2002-2005)
Referred  claimants  who become  employed  (2002-2005) (weeks) (2002- 2005)
GAO analysis based on 47 states and the District of Columbia.
section_title: Appendix III: Bibliography of Research Studies on the Worker-Profiling Initiative— Exhaustive List Identified from the Literature Review
paragraphs: Black, Dan A., Jeffrey A. Smith, Mark C. Berger, and Brett J. Noel. “Is the  Threat of Reemployment Services More Effective than the Services  Themselves? Evidence from Random Assignment in the UI System.” The  American Economic Review, Vol. 93, No. 4 (November 2003).
Black, Dan A., Jose Galdo, and Jeffrey A. Smith. “Evaluating the Worker  Profiling and Reemployment Services System Using a Regression  Discontinuity Approach.” Paper presented at the American Economic  Association conference in January 2007. Submitted to The American  Economic Review for the May 2007 Papers and Proceedings Issue.
Dickinson, Katherine P., Suzanne D. Kreutzer, and Paul T. Decker.  “Evaluation of Worker Profiling and Reemployment Services Systems:  Report to Congress.” U.S. Department of Labor, Employment and Training  Administration (March 1997).
Dickinson, Katherine P., Suzanne D. Kreutzer, Richard W. West, and Paul  T. Decker. “Evaluation of Worker Profiling and Reemployment Services:  Final Report.” U.S. Department of Labor, Employment and Training  Administration Research and Evaluation Report Series 99-D (1999).
Noel, Brett J. “Two Essays on Unemployment Insurance: Claimant  Responses to Policy Changes.” Dissertation submitted for the degree of  Doctor of Philosophy at the Graduate School of the University of  Kentucky, UMI Number: 9922624 (1998).
section_title: Appendix IV: Summary of the Impact of Referral to Services on Claimant Outcomes from the Literature Review
paragraphs: CT, ME, NJ:  Reduced by   1.4 to 4.3  percentage  points SC, KY:  Increased by   1.1 to 4.1  percentage  points   Dates indicate when claimants filed their UI claim or received their first UI benefit payment.
Unpublished dissertation.
Delaware was included in this study, but its sample size was too small to detect any significant  impacts.
section_title: Appendix V: Comments from the Department of Labor
section_title: Appendix VI: GAO Contacts and Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: Patrick di Battista, Assistant Director, and Michael Hartnett, managed this  engagement.
Shannon Groff, Rosemary Torres Lerma, and Winchee Lin also made  significant contributions throughout the engagement. Susan Bernstein  helped develop the report’s message. Jay Smale, Stuart Kaufman,  Rhiannon Patterson, Robert Dinkelmeyer, and Greg Dybalski contributed  to the analysis of Labor data and reviews of external studies. Jessica  Botsford provided legal support.
section_title: Related GAO Products
paragraphs: Workforce Investment Act: Employers Found One-Stop Centers Useful in  Hiring Low-Skilled Workers; Performance Information Could Help  Gauge Employer Involvement. GAO-07-167. Washington, D.C.: December  22, 2006.
Unemployment Insurance: States’ Tax Financing Systems Allow Costs to  Be Shared among Industries. GAO-06-769. Washington, D.C.: July 26,  2006.
Unemployment Insurance: Enhancing Program Performance by  Focusing on Improper Payments and Reemployment Services.   GAO-06-696T. Washington, D.C.: May 4, 2006.
Unemployment Insurance: Factors Associated with Benefit Receipt and  Linkages with Reemployment Services for Claimants. GAO-06-484T.  Washington, D.C.: March, 15, 2006.
Unemployment Insurance: Factors Associated with Benefit Receipt.  GAO-06-341. Washington, D.C.: March 7, 2006.
Workforce Investment Act: Labor and States Have Taken Actions to  Improve Data Quality, but Additional Steps Are Needed. GAO-06-82.  Washington, D.C.: November 14, 2005.
Unemployment Insurance: Better Data Needed to Assess Reemployment  Services to Claimants. GAO-05-413. Washington, D.C.: June 24, 2005.
Workforce Investment Act: Labor Should Consider Alternative  Approaches to Implement New Performance and Reporting  Requirements. GAO-05-539. Washington, D.C.: May 27, 2005.
Unemployment Insurance: Information on Benefit Receipt. GAO-05-291.  Washington, D.C. March 17, 2005.
Workforce Investment Act: Employers Are Aware of, Using, and Satisfied  with One-Stop Services, but More Data Could Help Labor Better Address  Employers’ Needs. GAO-05-259. Washington, D.C.: February 18, 2005.
Workforce Investment Act: States and Local Areas Have Developed  Strategies to Assess Performance, but Labor Could Do More to Help.  GAO-04-657. Washington, D.C.: June 1, 2004.
Workforce Investment Act: One-Stop Centers Implemented Strategies to  Strengthen Services and Partnerships, but More Research and  Information Sharing Is Needed. GAO-03-725. Washington, D.C.: June   18, 2003.