Veterans Justice Outreach Program: VA Could Improve Management by Establishing Performance Measures and Fully Assessing Risks
Published: 2016-04-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Most veterans transition to civilian life trouble-free. For those who struggle with their transition to the point that they are arrested and jailed, VA created the VJO Program, which connects veterans with supports and services to help avoid re-incarceration. The program relies on VJO specialists to link veterans to treatment. GAO was asked to review the management of the VJO Program.
This report examines 1) how the program delivers services and the number and characteristics of veterans in the program, 2) the extent to which VA uses performance assessment of the program, and 3) the key challenges VA has identified and the extent to which VA has developed mitigation strategies. GAO obtained VA data on program participants for fiscal years 2012 through 2015; reviewed documents; interviewed VA officials and staff from nine areas served by a VA medical center and selected for their geographic diversity and differences in the structures of local criminal justice systems; and in three of the areas interviewed criminal justice system stakeholders and veterans. While information from these interviews cannot be generalized, they provide insights on program challenges and operations.
section_title: What GAO Found
paragraphs: The Veterans Justice Outreach (VJO) Program—created by the Department of Veterans Affairs (VA)—operates through VA medical centers to provide services to veterans involved in local criminal justice systems, and in fiscal year 2015 served about 46,500 veterans, mostly men and many diagnosed with mental health or substance abuse problems. Officials from VA medical centers manage more than 260 VJO Program specialists who identify veterans in jails and local courts, assess their health and social needs, and link them to supports and services. VJO specialists monitor veterans' services and treatment in courts dedicated to veteran offenders. According to VA data, the number of veterans served by the program increased 72 percent from fiscal years 2012 - 2015. In addition, many veterans involved in the program were Post-9/11 veterans; about two-thirds were diagnosed with one or more mental health problems.
VA has taken some steps to incorporate a performance assessment system into the VJO Program, one component of effective program management (see figure). Specifically, VA developed strategic goals and plans to conduct evaluations. However, VA has not established performance goals with related targets, timeframes, and performance measures for any of the program's five broad strategic goals. VA officials told GAO they have not taken this step, in part, because VA medical centers have flexibility in determining the activities of VJO specialists. GAO's past work has highlighted strategies that agencies can use in this situation, such as developing measures based on common activities. Best practices call for agencies to establish performance goals and associated performance measures. Until VA incorporates performance goals and measures, it will lack a systematic way to obtain ongoing information to identify possible underperforming areas for improvements.
VA identified several key challenges—most of which were related to the demand for services outpacing the program's resources—but has not fully developed appropriate mitigation strategies. One key challenge, for example, is addressing increased program demand as jail administrators more widely use VA's online system that better identifies incarcerated veterans. In addition, a major reason for the demand-resource imbalance is the heavier workload of VJO specialists serving veterans in an expanding number of courts dedicated to veterans, according to VA officials and stakeholders that GAO interviewed. However, GAO found that VA did not comprehensively identify and assess risks posed by each of the key challenges it identified, contrary to federal internal control standards. Absent a comprehensive risk assessment, VA is not well-positioned to develop appropriate strategies to mitigate the greatest risks, which may limit its ability to help justice-involved veterans receive assistance and avoid re-incarceration.
section_title: What GAO Recommends
paragraphs: To improve program management, VA should establish performance goals and measures and conduct a comprehensive risk assessment. In commenting on a draft of this report, VA agreed with the recommendations and discussed actions it plans to take to implement them.

section_title: Letter
paragraphs: While most veterans transition to civilian life relatively trouble-free, others  face difficulties, such as mental illness, substance abuse, unemployment,  homelessness, and other serious issues. For some veterans, these  issues may go unaddressed and may lead them to commit crimes and  other offenses. To help veterans who become involved with their local  criminal justice systems (referred to as “justice-involved veterans”), the  Department of Veterans Affairs (VA) established the Veterans Justice  Outreach (VJO) Program in 2009. The VJO Program is designed to  connect veterans to supports and services to reduce and prevent re- incarceration and homelessness. The program relies on VJO specialists  across the United States to forge and maintain relationships with jails and  municipal courts at the local level (i.e., local criminal justice systems).
Given the importance of helping veterans readjust to civilian life, you  asked us to examine the management of the VJO Program. This report  provides information on the following questions:  1.  How does the Veterans Justice Outreach Program deliver services,  and what are the number and characteristics of veterans served by  the program?  2.  To what extent has VA used performance assessment to help  manage the Veterans Justice Outreach Program?  3.  What key challenges, if any, has VA identified and to what extent has  the agency developed mitigation strategies, as necessary?
To address our objectives, we reviewed relevant federal laws and  regulations, as well as documents from VA’s central office, including the  VJO Program fiscal year 2012-2016 strategic plan (the most recent  available), policies, procedures, guidance, and program fact sheets. We  did not independently verify the actions described in such documents. We  obtained data about the number and characteristics of veterans served by  the VJO Program from VA for fiscal years 2012 through 2015 (the most  recent data available). We determined that VA’s compilation of data about  veterans served by the VJO Program were sufficiently reliable to include  in our report. Specifically, we obtained and assessed official  documentation such as user guides, frequently asked questions, and  disclaimers; and we discussed our planned use of the data and any  limitations with VA officials. To assess the degree to which VA uses  program performance assessment—setting program goals, using  performance measures, and evaluating programs—to manage the VJO  Program, we compared VA’s use of performance assessment against  best practices for assessing program performance and federal standards  for internal control. We compared VA’s efforts to identify challenges and  respective mitigation strategies with criteria established in the federal  standards for internal control. Specifically, we chose to use risk  assessment—one of five key components in standards for internal  control—because analyzing risk provides the basis for developing  appropriate mitigation strategies.
We also conducted semi-structured interviews with stakeholders in nine  areas served by VA medical centers. They are: (1) Baltimore, Maryland;  (2) Bedford, Massachusetts; (3) Chicago, Illinois; (4) Fargo, North Dakota;  (5) Houston, Texas; (6) Orlando, Florida; (7) Salt Lake City, Utah; (8) San  Diego, California; and (9) Seattle, Washington. In all areas we conducted  interviews with VJO specialists and Veterans Integrated Service Network  (VISN) officials responsible for overseeing the VA medical centers  included in our review. In addition, for three areas—Baltimore, Maryland;  Orlando, Florida; and San Diego, California—we also interviewed VA  medical center officials; local criminal justice system stakeholders,  including court coordinators and judges, as well as jail administrators; and  held discussion groups with a small nonprobability sample of veterans  participating in the VJO Program. We selected the nine areas based on  the size of their veteran populations, proximity to veterans treatment  courts, geographic diversity, and VA central office officials’  recommendations. The interviews we conducted in the nine areas are  nongeneralizable but provided insights on the challenges facing the  program and its operations. We also interviewed relevant VA central  office officials.
Additional information about our methodology and the selection criteria for  areas in our review is provided in appendix I.
We conducted this performance audit from February 2015 to April 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Description of Local Criminal Justice Systems
paragraphs: Local jurisdictions (i.e., counties and municipalities) administer criminal  courts and permit pretrial detention of defendants accused of serious  offenses and deemed to be dangerous to prevent them from committing  crimes prior to trial, according to the Bureau of Justice Statistics. Many  local jurisdictions have begun adopting alternatives to incarceration,  which are intended to improve public health and safety while reducing  costs. Such alternatives generally represent a shift in emphasis away  from prosecuting those associated with lower-level crimes toward  providing treatment for underlying mental health or substance abuse  disorders and include programs for individuals in jail or court who screen  positive for mental illness. Veterans treatment courts are one such  alternative. These courts are typically local courts dedicated to handling  criminal cases involving veterans with mental health or substance abuse  problems.
According to VA, veterans treatment courts share several general  characteristics but vary in their specific policies and procedures because  of, among other things, differences in local jurisdictions and criminal  justice system practices. Veterans treatment courts are modeled after  adult drug courts, which are specialized courts that target criminal  offenders who have drug addiction and dependency problems. As in the  case of adult drug courts, judges preside over veterans treatment court  proceedings and monitor veterans’ progress with treatment in  collaboration with a team that usually includes a court coordinator,  prosecutor, public defender, and probation officer. Additionally, the team  includes a VJO specialist. Veterans treatment courts vary in terms of  criteria for taking a case, such as the types or levels of criminal offenses.
section_title: Involvement of Veterans in Local Criminal Justice Systems
paragraphs: As was the case with past generations of veterans, the transition from  military life to civilian life can be challenging for Post-9/11 veterans. Most  veterans are not involved with local criminal justice systems, but some  veterans—particularly if their mental health, family readjustment, or other  needs remain unmet—may become justice-involved. The Bureau of  Justice Statistics reported that about 7 percent (about 50,000) of the total  population of inmates in jail between February 2011 and May 2012 were  veterans. According to the Bureau of Justice Statistics report, this  estimate represents a 25 percent decrease from the number of veterans  in jail in 2004. Moreover, veterans were incarcerated in jails at lower rates  than nonveterans between February 2011 and May 2012, according to  the Bureau of Justice Statistics.
While many veterans who served in the military have successfully  readjusted to civilian life with minimal difficulties, researchers and  policymakers have identified concerns about how the experience of Post- 9/11 servicemembers may affect the incarceration rates among these  veterans. According to VA, military experience (particularly combat) has  been an underlying factor in behavior that prompts a response from law  enforcement, such as domestic conflicts. During the last 14 years of U.S.  military operations, many servicemembers have experienced numerous  deployments, which can increase the risk of developing post-traumatic  stress disorder (PTSD) and traumatic brain injury. According to VA, a  strong relationship exists between PTSD and substance abuse.  Research has demonstrated that justice-involved veterans have high  rates of mental illness, substance abuse, homelessness, and other health  issues. At the same time, some veterans may be unwilling or unable to  access the supports and services they need. For example, we have  previously reported that some veterans do not seek mental health  treatment due to concerns about negative career outcomes, lack of  understanding or awareness of treatment, and logistical challenges to  accessing care. Left unaddressed, a combination of homelessness,  unemployment, mental-health, or substance abuse issues can place  veterans experiencing a difficult transition at higher risk of committing a  crime.
section_title: Description of the Veterans Justice Outreach Program
paragraphs: VA initiated the VJO Program in 2009. The mission of the program is to  reduce and prevent criminal justice recidivism and homelessness among  veterans by linking justice-involved veterans with appropriate supports  and services. According to VA, incarceration is a strong predictor of  veteran homelessness, and recidivism can limit VA’s ability to provide  continuous care for mental health and other issues.
The VJO Program’s fiscal year 2012-2016 strategic plan contains five  broad strategic goals for the program as well as a number of related  objectives (see table 1).
section_title: Importance of Program Performance Assessment
paragraphs: Evaluative information helps the executive branch and congressional  committees make decisions about the programs they oversee; that is,  evaluative information tells them whether and why a program is working  well or not. We have previously reported that an important system to  obtain such information is through program performance assessment.  (See fig. 1.) A program performance assessment system is an important  component of effective program management and contains three key  elements:  1.  Program goals communicate what the agency proposes to  accomplish and allow agencies to assess or demonstrate the degree  to which those desired results were achieved.
Strategic goals and related objectives are long-term goals that set  a general direction for a program’s efforts.
Performance goals are the specific results an agency expects its  program to achieve in the near term.  2.  Performance measures are concrete, objective, observable  conditions that permit the assessment of progress made toward the  agency’s goals. Performance measures show the progress the  agency is making in achieving performance goals.  3.  Program evaluations are individual systematic studies using  performance measures and other information to answer specific  questions about how well a program is meeting its objectives.
section_title: The VJO Program Operates Through VA Medical Centers and Has Linked Many Post-9/11 Veterans to Supports and Services
subsections: section_title: VJO Program Operates Through VA Medical Centers to Respond to Local Needs
paragraphs: The VJO Program operates through VA medical centers. VA’s central  office provides each VA medical center the flexibility to determine how to  best respond to the needs of justice-involved veterans within the local  community. More specifically, VA established broad program parameters  that allow VA medical center officials to set VJO specialists’ activities to  meet the needs of veterans in local criminal justice systems. VA issued  guidelines on the operations of the VJO Program through a series of  memorandums to program officials. According to VA, the program is to  serve veterans as they interact with the local criminal justice system at  multiple points and settings—from initial contact with local law  enforcement to release from jail. VA guidelines define a “justice-involved  veteran” as a veteran who is (1) arrested by local law enforcement who  can be appropriately diverted from arrest into treatment; (2) incarcerated  in a local jail, and who either has a pending trial or is serving a sentence  after a conviction; or (3) involved in adjudication or monitoring by a court.  (See fig. 2.)
The Veterans Health Administration (VHA), which operates VA’s health  care system, administers the VJO Program. VA policy requires each of  the 167 VA medical centers around the country to provide outreach to  justice-involved veterans. VA prioritizes veterans charged with nonviolent  crimes, but VA must consider a veteran’s current legal circumstances— and not legal history alone—to determine whether the program can meet  the individual veteran’s needs while maintaining safety, according to VA  policy. The program does not provide legal representation, nor does it  accept legal custody of a veteran.
Specifically, VHA staff in Veterans Integrated Service Networks (VISN)  help oversee the VJO Program and provide technical assistance to VJO  specialists. Further, VA’s central office, its VISNs, and VA medical  centers manage different aspects of program operations. VA central office  officials directed each medical center to have at least one full-time VJO  specialist position, most of which are funded by central office funds,  according to VA. Some VA medical centers employ one VJO specialist  position while others have multiple positions. VA central office officials  also train new VJO specialists and manage national data collection  efforts. VJO specialists report directly to VA medical center officials,  typically in the homelessness prevention or mental health services.
According to VA central office officials, as of September 2015, VA  employed 261 full-time VJO specialists, who are mostly social workers,  and seven other VJO specialists who play a hybrid role between the VJO  and the Health Care for Re-entry Veterans programs.
To accomplish the program mission of serving justice-involved veterans,  VA tasks VJO specialists to perform three core functions: identify, assess,  and link justice-involved veterans to appropriate supports and services.
subsections: section_title: Identification
paragraphs: Experience of Veterans Who Have  Participated in the VJO Program  All of the veterans participating in our group  discussions were appreciative of the VJO  Program. At one location, veterans said that  the VJO specialists understood their needs,  educated them about available VA supports  and services, assisted them in determining  their eligibility, and helped them obtain VA  supports and services.  An example from a VJO Specialist  To illustrate the effect of the program on one  veteran’s life, VJO specialists we interviewed  in one of our selected areas provided the  following anecdote. A homeless veteran—who  had served in Operation Enduring Freedom— was arrested for trespassing for sleeping  behind a VA building. The veteran had  substance abuse issues. When he appeared  before a veterans treatment court, he met a  VJO specialist who assessed his needs and  assisted him in enrolling in VA housing and  substance abuse treatment. When the veteran  was stable, the VJO specialist was able to link  him to permanent housing and VA vocational  training. After several years of treatment and  assistance, the veteran was hired by VA and  purchased a home with a VA loan. helps jail administrators identify incarcerated veterans by comparing the  names of inmates with VA’s list of veterans.
section_title: Assessment
paragraphs: After they identify justice-involved veterans, VJO specialists determine  the veterans’ treatment needs by assessing their mental health, social  well-being, appearance, and attitude. VJO specialists also collect  information on employment history, current housing situation, military  service, and discharge status. According to VA staff, upon completing the  assessment, VJO specialists develop a treatment plan to meet the  veteran’s needs. A treatment plan typically includes recommendations for  medical or mental health services, housing, or other services, according  to VJO specialists in five of the nine selected areas.
section_title: Linkage
paragraphs: After they identify and assess justice-involved veterans, VJO specialists  link veterans to VA or community supports and services. VJO  specialists do not directly provide treatment to justice-involved veterans.  VA staff said once VJO specialists refer and link veterans to the  appropriate supports and services, they perform follow-up visits with  veterans to ensure they are receiving them. For example, VJO specialists  may assist veterans to find adequate housing or with any transportation  issues. Some veterans in jails may receive one follow-up visit while others  may receive more, depending on their needs according to a VJO  specialist. In contrast, VJO specialists work with veterans who participate  in veterans treatment courts for 1 to 2 years, depending on the local  criminal justice system and on the amount of time a veteran is required to  participate in the court program.
VJO specialists refer and link veterans to the VHA for health care, mental  health, substance abuse treatment, or housing services where the  overseeing staff determine the type of treatment and services provided.  VJO specialists also link veterans to the Veterans Benefits Administration  for disability compensation, pension benefits, or vocational rehabilitation  to be determined by overseeing staff based on a veteran’s eligibility,  according to VA (see fig. 3). In addition, VJO specialists link veterans to  community service providers when the VA-provided treatment is too far  away for a veteran to participate, or when a veteran is not eligible for VA  health care services or benefits according to VA.
Within these broad program parameters, a VA medical center determines  the type and amount of investment it makes in serving justice-involved  veterans. VJO specialists’ activities vary based on whether they are  working in courts, jails, or both. For example:
Specialty courts: VJO specialists provide services to veterans in a  variety of specialty courts, such as veterans treatment courts, drug  courts, and mental health courts. However, VJO specialists work  most often with veterans treatment courts compared to other specialty  courts, according to VA central office officials.
Local criminal justice systems: In local criminal justice systems  without established veteran-focused programs, VJO specialists spend  significant time attempting to develop or plan these types of programs,  according to VISN officials. These activities involve but are not limited  to planning veterans treatment court (or other alternative court)  programs and negotiating the terms of access to conduct outreach in  jail facilities. In other areas, VJO specialists work with local criminal  justice officials who may be unaware of veteran-focused programs. In  such cases, VJO specialists focus their work on educating local law  enforcement about VA resources available to veterans in crisis and  working with veterans in jails, according to two VISN officials with  whom we spoke.
Jail administrators: VJO specialists in two of the nine areas we  selected for interviews worked with local jail administrators to set up a  program that provides veteran-specific housing units. These housing  units are a designated block of jail cells only for veterans. Within these  units, local community service organizations provide incarcerated  veterans with treatment and services while they are in jail, and VJO  specialists assess and develop community re-entry plans for housing,  VA treatment, or other necessary services. In another area we visited,  VJO specialists developed a program to help “high-risk” veterans  released from jail. VJO specialists used information from the veterans’  clinical assessment to determine which veterans were high-risk for re- offending, dropping out of treatment, or becoming homeless. One  VJO specialist at that VA medical center said the program serves  these veterans by providing weekly follow-up services within the first  month of their release from jail, and offers regular follow-up services  and support after the first month, based on the individual veteran’s  needs.
section_title: In fiscal year 2015, the VJO Program Served about 46,500 Veterans, Many of Whom Served in Post-9/11 Conflicts and Reported Substance Abuse Problems
paragraphs: In fiscal year 2015, the VJO Program served about 46,500 justice- involved veterans, and the program has experienced steady growth in the  number of veterans served, according to data from VA. During fiscal  years 2012 through 2015, the number of justice-involved veterans  annually served by the VJO Program increased from about 27,000 to  46,500, a 72 percent increase (see fig. 4).
Justice-involved veterans who were served by the VJO Program in fiscal  year 2015 had the following characteristics:
Most were young men. Nearly 95 percent of the veterans served by  the program were male and 52 percent of justice-involved veterans  were between the ages of 18 and 44.
Many reported not working full-time during the past 3 years.  Approximately 40 percent of the veterans served by the program  reported working part time, working irregularly, or having been  unemployed during the past 3 years. Another one-third (33 percent)  reported being retired or disabled.
Almost three-fourths reported serving during a U.S. military  intervention. Of veterans who received services through the VJO  Program in fiscal year 2015, 73 percent had served during a military  intervention (see fig.5).
Two-thirds reported mental health problems and the vast  majority had substance abuse problems. During the clinical  assessment performed by VJO specialists, about two-thirds (68  percent) of the veterans reported one or more mental health  problems, and 69 percent had substance abuse problems.
Public order offenses were the most common type of reported  criminal charges. The types of criminal offenses that veterans were  charged with during fiscal year 2015 included, but were not limited to,  public order offenses (33 percent), drug offenses (22 percent),  property offenses (16 percent), and probation violations (12 percent).
More than 40 percent reported either being homeless, losing a  home, or living in an unstable housing environment (see fig. 6).  VJO specialists we interviewed in two of the nine areas we selected  said that some veterans become homeless once they leave jail  because they are unable to pay their rent while incarcerated.
Additional information about the characteristics of veterans in the VJO  Program is provided in appendix II.
section_title: VA Has Taken Steps toward Partially Incorporating Key Elements of a Performance Assessment System
subsections: section_title: VA Is at the Beginning Stage of Planning VJO Program Evaluations
paragraphs: VA has taken some steps toward evaluating the VJO Program via a  longitudinal program evaluation that examines the extent to which  veterans are being linked to services for which they are referred. VA  planned to obtain information through interviews with 1,500 veterans.  However, after the contractor submitted an interim report that detailed  problems it encountered recruiting veterans to interview as part of the  study, VA central office officials terminated the contract. As an alternative,  VA plans to use administrative data to complete this evaluation. However,  at the time of our review, VA central office officials were in the process of  determining how to complete such a re-tooled evaluation.
VA also developed a research agenda to evaluate the outcomes for  veterans served by the VJO Program. For example, one research topic is  to identify outcomes for veterans who participate in veterans treatment  courts. In addition, officials said they plan to evaluate the extent to which  veterans served by the VJO Program avoided incarceration and  homelessness.
section_title: VA Has Not Established Performance Goals or Performance Measures
paragraphs: Although VA has developed five broad strategic goals for the VJO  Program and taken steps to evaluate it, VA has not fully measured  progress toward any of the strategic goals because it has developed  neither performance goals nor performance measures, contrary to leading  practices for managing programs. Performance measures focus on  whether a program has achieved measurable standards. They allow  agencies to monitor and report program accomplishments on an ongoing  basis.
VA collects ongoing information about program operations and veterans  served by the program, but this information does not help VA measure its  progress toward accomplishing any of the five goals outlined in its  strategic plan. According to VA central office officials, VA collects some  information, such as (1) the number of VJO specialists and vacancies at  each VA medical center; (2) VJO specialists’ workload, including the  number of veterans that VJO specialists serve and their nonclinical  activities, such as the number of trainings they conduct; and (3)  information on veterans served in the program, such as demographic  characteristics and medical histories. This information is useful for some  aspects of program management. For example, VA central office officials  told us that supervisors need information about the clinical and  administrative workloads of VJO specialists to assess the need for  additional staff and to monitor productivity. However, VA does not have a  way to use the information it currently collects to compare actual program  performance against expected results, or to analyze significant  differences, contrary to federal standards for internal control. We have  previously reported that performance measurement gives managers  crucial information to identify gaps in program performance and plan any  needed improvements. The information that VA currently collects does  not allow VA to fully answer key questions, such as the reasons for  observed performance, what are effective approaches to program  implementation, and how to improve program performance.
VA central office officials cited the need to be responsive to local  conditions as the main reason why they have not set or used  performance goals and measures for the VJO Program. Specifically, the  areas served by each VA medical center have their own unique  circumstances, they said, such as having or not having a veterans  treatment court, being located in an urban or rural area, or being located  in an area with a large or small population of veterans. Due to these  unique local conditions, the officials stated, the program needs to be  flexible. VA medical centers have discretion in determining the activities  of VJO specialists. VA central office officials added that a broad measure,  such as the number of veterans served by each VJO specialist per  month, would not be appropriate because it could be misleading to  compare VJO specialists since their work circumstances vary. For  example, officials said that given VJO specialists work in different criminal  justice systems, it would not be reasonable to set a national goal for the  number of veterans they reach out to each month. While some VJO  specialists conduct outreach to a large number of veterans in jails, others  may focus their attention on a smaller number of veterans in veterans  treatment courts. Similarly, they pointed out that while some VJO  specialists in VA medical centers in large urban areas may be able to  serve a large number of veterans, others in rural areas may need to drive  several hours to see a veteran and can only see a few in any given day.
We recognize that measuring progress toward goals within the VJO  Program’s decentralized service delivery model poses challenges in  designing performance goals and measures. Our past work has also  acknowledged the challenges in developing national performance  measures when flexible programs vary in their activities so as to meet  local needs. Nonetheless, we have found one approach agencies used to  successfully overcome this challenge was to develop common national  measures. For example, our past work found that to assess the  performance of the Expanded Food and Nutrition Education Program—a  program that assists low-income families in acquiring skills to improve  their family diet—the U.S. Department of Agriculture assessed local  offices on common activities used by all offices. Another approach used  by agencies with flexible programs was to encourage local projects to  evaluate progress toward their own performance goals. For example, the  National Tobacco Control Program—a program operated by the Centers  for Disease Control and Prevention—has four goals connected to its  mission to reduce tobacco-related diseases and death, including reducing  youth’s tobacco use. To accomplish its goals, the agency allows states to  direct their own activities. The agency provided states with guidance  about identifying short term, intermediate, and long term outcomes, and  encouraged states to assess their own individual efforts.
VA central office officials we interviewed also said that they have not  established performance goals or measures because some of the  outcomes that veterans experience are influenced by factors outside the  program’s control. For example, VJO Program-related outcomes—such  as mental health recovery or criminal recidivism—can often depend on  factors such as whether a community has a veterans treatment court or if  a veteran is willing to adhere to his treatment plan, according to VA  central office officials.
We have highlighted strategies in our past work that agencies can use  when faced with the challenge of having limited control over external  factors that can affect a program’s outcomes. These strategies include:  selecting a mix of outcome goals over which the agency has varying  levels of control; using data on external factors to statistically adjust for  their effect on the desired outcome; and disaggregating goals for distinct  target populations for which the agency has different expectations. For  example, our past work found that to measure progress toward its  strategic goal to eliminate transportation related deaths, injuries, and  property damage, the National Highway Traffic Safety Administration  measured an intermediate outcome—the rate of front-seat safety belt  use—and an end outcome—the rate of transportation related injuries. The  agency also statistically adjusted the results of its performance measures  by using the ratio of fatalities per vehicle mile driven to control for the  simple fact that if more miles are driven, then more crashes are likely to  result. Our past work also found that the Natural Resources Conservation  Service created separate performance goals for each type of habitat (e.g.,  croplands, watersheds, wetlands, and grazing lands) so it could measure  progress toward its strategic goal of maintaining healthy and productive  land. Additionally, to help interpret the results of performance measures,  we have also emphasized in our past work the importance of  communicating adequate contextual information, such as factors inside or  outside the agency’s control that might affect performance.
VA central office officials said they plan to update the VJO Program’s  strategic plan, which contains the program’s broad strategic goals, by the  end of fiscal year 2016, but they do not intend to include performance  measures. Leading practices demonstrate that developing ways to  measure VJO Program efforts could help VJO specialists, VA central  office officials, and Congress measure and monitor the extent to which  program efforts are achieving their intended results, make needed  improvements, and make funding decisions. Further, a fundamental  element in an organization’s efforts to manage for results is its ability to  set performance goals with specific targets and time frames that reflect  strategic goals and to measure progress toward them as part of its  strategic-planning efforts. Our previous work has highlighted  characteristics of successful performance measures that could be  helpful. Without a robust performance assessment system—which  includes both performance measures and evaluations—officials  managing the VJO Program lack a full picture of its success and of  potentially underperforming areas for improvements.
section_title: VA Has Identified Capacity Challenges but Has Not Comprehensively Assessed Related Risks or Developed Mitigation Strategies
subsections: section_title: VA Identified Key Challenges Related to Justice-involved Veterans’ Demand for Services Outpacing VJO Program Resources
paragraphs: VA identified several key challenges related to the demand for services  outpacing the VJO Program’s resources, which could limit the program’s  capacity to serve all justice-involved veterans. These demand-resource  imbalances were identified through VA’s strategic planning process and in  consultation with VJO specialists and VISN officials. In developing its list  of five challenges, which it reported out in 2012, VA assumed that  demands on the VJO Program would continue to increase. Specifically,  two of the five challenges VA identified relate to demand for services: (1)  demand may increase due to greater use of VA’s Veterans Re-entry  Search Service system and result in the program not being able to  properly serve all eligible veterans; and, (2) demand from homeless  veterans may increase if economic conditions worsen. The remaining  three challenges VA identified relate to resources; (3) the program may  not be able to link veterans to treatment due to shortages in VA clinical  programs; (4) the program lacks direct control over information  technology on which it is highly dependent; and, (5) the program may  face funding cuts.
The range of stakeholders we interviewed—VISN officials, VJO  specialists, justice-involved veterans, and local criminal justice system  officials—affirmed many of the challenges identified in the VJO Program  strategic plan. In general, many stakeholders said that demand for VJO  specialist assistance is outpacing the program’s ability to serve all  potentially eligible veterans, and the gap may worsen over time. For  example, VJO specialists and VISN officials we spoke with in five of the  nine areas we selected for interviews told us workload challenges have  intensified in recent years.
In addition to the high demand for the program cited by program  stakeholders, the increase in veterans treatment courts is further  increasing demand for program services, according to stakeholders and  VA internal documents. In particular:
The number of veterans treatment courts is growing. While these  courts can help improve veterans’ mental health and sobriety, the  increase in the number of these courts is a major reason for the VJO  Program’s workload challenges, according to VJO specialists, VISN  officials, VA central office officials, and justice system partners we  interviewed. According to these stakeholders, VJO specialists had  already been working in many local jails and traditional courts across  the country, and the expansion of veterans treatment courts added to  their existing workload. Specifically, the number of veterans treatment  courts nationwide grew from 65 in fiscal year 2010 to 360 in fiscal  year 2015, according to VA data. Moreover, hundreds of additional  veterans treatment courts are in the planning stages, according to an  organization that advocates on behalf of veterans treatment courts.  During fiscal years 2012 through 2015, justice-involved veterans  served by the VJO Program and who participated in veterans  treatment courts increased from about 1,900 to about 3,900.
Working with veterans in veterans treatment courts is more time  consuming than working with veterans in jails. According to VA  central office officials, working with individual veterans participating in  veterans treatment courts typically requires more of VJO specialists’  time than working with veterans in jail. VJO specialists we interviewed  in seven of our nine selected areas also said working with veterans  treatment courts is more time consuming than working in jails. For  example, VJO specialists meet regularly with veterans and their  treatment providers and apprise other members of the court team of a  veteran’s adherence to court-ordered treatment. A veteran in a  veterans treatment court generally participates in the VJO Program for  12 to 24 months. In contrast, VJO specialists we interviewed in seven  of the nine areas said they generally work with veterans in jails for  much shorter durations.
According to VA central office officials, VA attempts to ease workload  demands by hiring additional VJO specialists, as funds allow, but this has  not fully addressed demand for program services. VA increased staffing  for the VJO Program from 43 specialists in fiscal year 2010 to 261 in  fiscal year 2015, according to VA data. VA central office officials said they  have used an annual hiring process to allocate new VJO specialist  positions. Through this process they added 13 VJO specialists in 13  locations in fiscal year 2015. VA central office officials said that the  process for hiring additional VJO specialists consists of collecting  information from VA medical centers about the current clinical and  administrative activities of VJO specialists, and about any imminent  workload demands, including those attributable to a new veterans  treatment court, to determine which VA medical centers receive additional  specialist positions. In fiscal year 2015, 54 of the 167 VA medical centers  requested 1 or more of the 13 new VJO specialist positions. However, VA  central office officials, VJO specialists, and VISN officials acknowledged  that despite additional positions, VJO specialists are at capacity and are  not able to fully address the demand for VJO Program services.
VA central office officials also told us that they try to address the  challenge of limited existing workload capacity by advising VJO  specialists and VA medical center officials to avoid overcommitting  program resources. For example, VA central office officials told us they  advised VJO specialists that they should consider their existing workload  and commitments before deciding to work with new veterans treatment  courts or visit additional jails. Some VJO specialists we spoke with chose  to focus on serving veterans in jails while others decided that serving  those in veterans treatment courts was more effective. For example, VJO  specialists and two criminal justice system officials in one area expressed  concerns about the workload and resources associated with serving  veterans treatment courts. In their view, a more cost-effective approach is  to focus the program’s efforts on providing case management to justice- involved veterans on probation. However, VA central office officials  acknowledge that without VA involvement, these courts likely would not  function and proliferate. VJO specialists in five areas with veterans  treatment courts told us they decided to prioritize working with veterans in  treatment courts over incarcerated veterans given their workloads and  because veterans participating in these courts are mandated to seek  treatment. Specifically, veterans in these courts are generally held  accountable for remaining in treatment for longer periods of time than  those released from jails, which tends to produce more stable behavioral  changes, according to these VJO specialists. A consequence, however, is  that the specialists may see fewer veterans in veterans treatment courts  or have less time to spend serving those in jails and other criminal justice  settings, according to these VJO specialists we interviewed. In addition,  almost every justice-involved veteran (13 of 14) we interviewed said that  although they have benefitted from the program, VJO specialists do not  frequent the jails enough to fully meet veterans’ needs.
VA central office officials we interviewed also said they recognize that  demand for the VJO Program could increase beyond existing capacity if  its Veterans Re-entry Search Service becomes more widely used by jail  administrators. VA developed the Veterans Re-entry Search Service in  2013 to meet a key strategic goal to identify justice-involved veterans  more effectively. This online system improves the jails’ identification  process compared with the self-identifying process currently used by  most jail administrators. According to VA central office officials, jail  administrators who are using the self-identification process are reporting  a much smaller percentage of veterans. For example, an administrator in  one local jail reported that 23 inmates self-identified as veterans, but the  Veterans Re-entry Search Service revealed that there were 64  incarcerated veterans in the jail during the same time-period. In another  jail system, the administrator reported that 220 inmates self-identified as  veterans, but the online search revealed there were 400 incarcerated  veterans.
VA central office officials we interviewed recognize the risk the Veterans  Re-entry Search Service poses to existing capacity. VA central office  officials told us they plan to promote the new system to jail officials in  small phases because they recognize that this system may overtax  existing program capacity. VA central office officials told us that they  decided to initially promote VA’s new system to 180 of the largest jails  rather than roll it out to jail systems (over 3,000 in the United States) so  as to avoid overwhelming VJO specialists with an unmanageable influx of  new veterans. Once VA completes its outreach to these larger jail  systems, it plans to promote the service to other jail systems, and VA  central office officials expect about 1,000 jails to eventually use the  service.
In addition to challenges related to capacity, VJO specialists and VISN  officials in several of our selected areas told us that shortages in clinical  programs—especially residential drug treatment programs—can be a  challenge because it limits treatment options for veterans. They also  reported challenges associated with shortages in public housing and  public transportation, the latter of which can impede veterans’ ability to  readily attend court and medical appointments for treatment. These  capacity challenges include:
Limited capacity in residential substance abuse programs. VJO  specialists and VISN officials in several of our selected areas reported  that the high demand for residential substance abuse treatment— which provide appropriate housing to facilitate sobriety—limits the  extent to which these programs are a viable option for justice-involved  veterans. When openings are unavailable, VJO specialists said they  refer veterans to the next best available treatment option. However,  officials from one of the veterans treatment courts we interviewed said  the next best available treatment option may not always fully meet a  veteran’s treatment needs. For example, officials we interviewed at  one veterans treatment court said that a veteran in the program has a  heroin addiction and has tested positive for drug use. Thus, they  would expect this veteran to be in a residential substance abuse  treatment program, which provides greater monitoring. In this case,  however, VA has recommended out-patient substance abuse  treatment due to limited availability of residential slots.
Limited housing options for sex offenders. VJO specialists and  VISN officials in several of our selected areas also reported that  housing options available for individuals who are required to register  as sex offenders are limited. VA central office officials said that they  are aware of this issue, which cuts across many of VA’s homeless  programs.
Limited public transportation options. VJO specialists and VISN  officials in several of our selected areas also reported that many  justice-involved veterans—some of whom have lost their driving  licenses due to their offenses—rely on public transportation to take  them to court and VA- or community-provided treatment. VJO  specialists told us that the public transportation can be slow, limited,  or nonexistent. For example, VJO specialists in one area we visited  said it can take a full day for a veteran to travel to and from treatment.  VJO specialists in another rural location said they have tried to  address this challenge by hiring veterans who have received VJO  Program assistance to help drive other veterans to their medical  appointments.
section_title: VA Has Not Comprehensively Assessed Program Risks, which Would Allow It to Develop Strategies to Mitigate Their Effects
paragraphs: VA has not performed a comprehensive assessment of risks posed by the  challenges the VJO Program faces, which is inconsistent with federal  standards for internal control and one component of effective program  management. These standards call for agencies to both identify all  relevant risks and analyze risks that may prevent them from achieving  their goals. Further, this assessment should include an estimation of a  risk’s significance, an examination of the likelihood of any risk’s  occurrence, and a decision as to what actions should be taken to manage  the risk. Risk assessment is important because it also informs an entity’s  policies, planning, and priorities. Without understanding risks, an agency  may not be able to set appropriate policies, plans, and priorities because  it would not be able to account for events that can adversely affect its  ability to implement its objectives.
In 2014, VHA leadership requested that each homelessness prevention  program, including the VJO Program, identify risks that may affect  veterans’ access to VA supports and services and identify possible  actions that could be taken to mitigate those risks. To identify risks, VJO  Program central office officials examined their broad strategic goals. In an  October 2014 assessment report, VJO Program officials identified six  risks. Identified risks included, for example, lack of an automated  interface between the Veterans Re-entry Search Service and correctional  facilities’ information technology systems to identify incarcerated  veterans; reassignment of VJO specialists to other programs by VA  medical centers; and prioritizing clinical services for veterans already  receiving treatment over new justice-involved veterans by VA medical  centers, thus limiting timely access to appropriate services.
VA central office officials, as part of this 2014 effort, also estimated the  significance and likelihood of the risks, consistent with federal standards  for internal control. For example, regarding the significance if VA medical  centers were to de-prioritize outreach to veterans, VA found that the  program’s relationship with its criminal justice system partners could be  weakened and justice-involved veterans may suffer. In addition, the  assessment indicated actions VA would take to address each of the six  identified risks, such as collecting additional information to monitor  identified risks.
VA did not, however, identify or analyze the risks posed by each one of  the challenges identified in its strategic plan. Notably absent were the  workload challenges facing the VJO Program. VA central office officials  we interviewed acknowledged that the challenges identified in the  strategic plan still exist and could affect the operations of the program. VA  expects the VJO Program’s workloads to increase and its funding and  workforce to remain level, raising questions about how VA can best  deploy its resources and align its policies to best meet increasing  demand. Not identifying all relevant risks limits VA’s ability to effectively  compare and prioritize risks faced by the program.
In addition, VA’s lack of performance goals, as previously discussed,  negatively affects its ability to effectively identify and assess risk. As  demonstrated in our previous work, federal standards for internal control  state that a precondition to risk assessment is the establishment of clear  performance goals (see fig. 7). Translating its broad strategic goals into  measurable performance goals would allow VA to target the risks that  may impede achievement of the program’s objectives.
VA plans to conduct another risk assessment as part of its upcoming  strategic planning efforts, using the same methodology it used in its risk  assessment in 2014. As VA completes future assessments, it is important  that the agency incorporates all of the elements of risk assessment  detailed in federal standards for internal control, including identifying all  relevant risks posed to achieving performance goals. Lacking  comprehensive risk assessments may limit VA’s ability to target areas  posing the greatest risks and, in turn, develop appropriate mitigation  strategies.
section_title: Conclusions
paragraphs: Perhaps some of the most challenging veterans to serve in this country  are veterans who have committed a crime or other offense and who face  significant long-term consequences for their actions. If VA intervention is  timely and targeted, then these justice-involved veterans and their  families have a better chance to avoid a detrimental future. A relatively  new VA program, the VJO Program is designed to help justice-involved  veterans—who often have high rates of mental illness, substance abuse,  and other issues that may stem from their military service—avoid re- incarceration and homelessness. In designing the program, VA intended  for support to respond to local conditions. While this flexibility can help the  program cater to veterans’ needs in local communities, VA lacks a  complete national perspective necessary to strategically manage the  program. We are encouraged that VA has taken some initial steps toward  assessing program performance and risks. However, as VA moves  forward, it is especially important that the agency fully incorporate key  practices for assessing performance and risks to strategically guide  program decisions. Incorporating these key practices will allow VA to  compare the program’s actual performance against expected results and  comprehensively assess risks to develop appropriate mitigation  strategies. More specifically, without developing a national perspective  that comprehensively considers program performance and the greatest  risks to the program’s goals, VA cannot fully target its resources toward  efforts that achieve the intended results. As a result, justice-involved  veterans may not receive the proper supports and services needed to re- establish healthy lives and avoid re-incarceration. If this lack of a national  perspective persists, it will be difficult for VA to know whether its  resources are being used to serve justice-involved veterans in the best  manner possible.
section_title: Recommendations for Executive Action
paragraphs: To improve management of the Veterans Justice Outreach Program, we  recommend that the Secretary of Veterans Affairs direct the  Undersecretary for Health to take the following two actions:  1.  Establish performance goals with specific targets, time frames, and  related performance measures that are linked to strategic goals to  provide a basis for comparing actual program performance against  expected results; and  2.  Conduct a comprehensive assessment of the risks that challenges  pose to achieving the program’s strategic and performance goals, and  develop, as necessary, applicable mitigation strategies.
section_title: Agency Comments
paragraphs: We provided a draft of this report to the Department of Veterans Affairs  (VA) for review and comment. In written comments, which are reproduced  in appendix III, VA agreed with our recommendations and noted steps it  plans to take to address them. Specifically, VA agreed to develop a plan  linking VJO Program strategic goals with performance goals that sets  specific targets at the VA medical center level. With regard to risk  assessment, VA agreed to conduct a comprehensive assessment of risks  using federal standards for internal control and begin collecting  information from VA medical centers about workload challenges to fully  inform any efforts to redefine how it delivers services. VA did not provide  technical comments.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Veterans Affairs, and the Under Secretary  for Health. In addition, the report is available at no charge on GAO’s  website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202)-512-7215 or bertonid@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this review were to examine: (1) how the Veterans  Justice Outreach (VJO) Program delivers services, and the number and  characteristics of veterans served by the program; (2) the extent the  Department of Veterans Affairs (VA) has used performance assessment  to help manage the Veterans Justice Outreach Program; and (3) what key  challenges, if any, VA has identified and to what extent the agency has  developed mitigation strategies, as necessary.
To address all three objectives, we conducted semi-structured interviews  and reviewed documentation from nine areas served by VA medical  centers. These areas are: (1) Baltimore, Maryland; (2) Bedford,  Massachusetts; (3) Chicago, Illinois; (4) Fargo, North Dakota; (5)  Houston, Texas; (6) Orlando, Florida; (7) Salt Lake City, Utah; (8) San  Diego, California; and (9) Seattle, Washington. For all nine areas we  conducted interviews with VJO specialists and Veterans Integrated  Service Network (VISN) officials responsible for overseeing the VA  medical centers included in our review. In addition, for three of these  areas—Baltimore, Maryland; Orlando, Florida; and San Diego,  California—we also interviewed VA medical center officials; local criminal  justice system stakeholders, including court coordinators and judges, and  jail administrators; and held discussion groups with a small nonprobability  sample of veterans participating in the VJO Program. The interviews we  conducted with officials and stakeholder at the nine areas are  nongeneralizable but provided insights on the challenges facing the  program and its operations. We also interviewed relevant VA central  office officials.
subsections: section_title: Selection Criteria for Areas Served by VA Medical Centers in our Review
paragraphs: We selected the nine areas in our review based on the size of the  population of veterans in the area, geographic diversity, VA officials’  recommendations, and proximity to veterans treatment courts. (See table  2.) Specifically:
To identify areas with veterans treatment courts, we obtained a list of  veterans treatment courts from Justice for Vets, an organization that  advocates on behalf of veterans treatment courts. We considered a  VA medical center to be located near a veterans treatment court if the  court is located in the same state and is within 40 miles of the center.  Our selection process included some VA medical centers located near  a court and some which were not.
We obtained information about the population of veterans from VA’s  Veteran Population Projection Model 2014, an actuarial model  developed by VA. This model projects the veteran population from  fiscal years 2014 through 2043 by using data through fiscal year  2013. We used this model to select VA medical centers in counties  with varying numbers of veterans, ranging from approximately 11,000  (in Cass County, North Dakota, which includes Fargo) to 232,000 (in  San Diego, California).
In addition we used these cross-question methodologies:
To better understand how the VJO Program delivers services to  veterans, we reviewed relevant federal laws and regulations, VA  policies, procedures, guidance, program fact sheets, the VJO  Program’s fiscal year 2012-2016 strategic plan, documents from VJO  specialists, and other types of documentation that describe the types  of program activities and services used to serve justice-involved  veterans. We did not independently verify the actions described in  such documents. To describe the number and characteristics of the  veterans served by the VJO Program, we obtained data from VA.  This data included summary information on the number of veterans in  the program from fiscal year 2012 through fiscal year 2015 (the most  recent data available) as well as sex, age, race, and types of criminal  offense. We also obtained fiscal year 2015 data from VA about  veterans’ education level, marital status, the number of years since  the veterans separated from the military, and mental health status.  We determined that VA’s compilation of data about veterans served  by the VJO Program was sufficiently reliable to include in our report  by reviewing related documentation and interviewing knowledgeable  agency officials. Specifically, we obtained and assessed official  documentation such as users’ guides, frequently asked questions,  and disclaimers, and we discussed our planned use of the data and  any limitations with VA officials.
We assessed the degree to which VA uses program performance  assessment—setting program goals, evaluating programs and using  performance measures—to manage the VJO Program by reviewing  VA reports and documents, including the program’s fiscal year 2012- 2016 strategic plan, which is the most recent plan; program evaluation  plans and preliminary results; and reports on veteran receipt of  program services. We also interviewed VA central office officials and  other knowledgeable individuals about VA’s current efforts to evaluate  the VJO Program and about the program’s goals and efforts to  measure progress toward those goals. We compared VA’s use of  performance assessment against best practices for assessing  program performance and federal standards for internal control.
We obtained information on the challenges and VA’s respective  mitigation strategies by reviewing VA documents, including a 2014  report assessing program risks. We also interviewed VA central office  officials about the challenges and associated risks, how they are  addressing the challenges, and what strategies, if any, they have  developed to mitigate risks to achieving the program’s goals. We  compared VA’s approach for assessing risks with criteria established  in the federal standards for internal control. Specifically, we chose to  use risk assessment—one of five key components in standards for  internal control—because analyzing risk provides the basis for  developing appropriate mitigation strategies. We also obtained data  from VA on the number of veterans treatment courts and other  veteran-focused courts the VJO Program serves, and on the number  of full-time VJO specialists from fiscal year 2012 through fiscal year  2015. We compared this list of veterans treatment courts with other  authoritative sources and discussed any limitations with VA. We found  this number to be sufficiently reliable for our purposes.
section_title: Appendix II: Demographics of Veterans Served by the VJO Program during Fiscal Year 2015
paragraphs: Table 3 provides data from VA’s Homeless Operations Management and  Evaluation System. The selected demographics of veterans include  characteristics, military history, criminal justice system involvement, living  situation, employment and income, and clinical impression of veterans  served by the VJO program during fiscal year 2015.
Table 4 provides data on the number of justice-involved veterans by  states and the District of Columbia, who were served by the VJO Program  during fiscal year 2015. The highest concentrations of justice-involved  veterans were in Florida (1,433), California (1,202), Ohio (1,062), Texas  (991), and New York (886).
section_title: Appendix III: Comments from the Department of Veterans Affairs
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact mentioned above, the following staff members  made significant contributions to this report: Brett Fallavollita (Assistant  Director), James Whitcomb (Analyst in Charge), Hedieh Fusfield, and  John Lack. In addition, key support was provided by James Bennett, Joy  Booth, Valarie Caracelli, David Chrisinger, Alexander Galuten, Benjamin  Licht, Amy Moran Lowe, Paul Schearf, Stephanie Shipman, Almeta  Spencer, and Andrew Stavisky.