Veterans Crisis Line: Additional Testing, Monitoring, and Information Needed to Ensure Better Quality Service
Published: 2016-05-26T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: VA established the VCL in July 2007 to provide support to veterans in emotional crisis. Between fiscal years 2008, its first full year of operation, and 2015, the number of calls received by the VCL increased almost 700 percent, exceeding VA's expectations. As VA began to address increasing numbers of requests for assistance, reports of dissatisfaction with VCL's service periodically appeared in the media.
GAO was asked to review VA's administration of the VCL. This report, among other issues, examines (1) the extent to which VA meets response-time goals for VCL calls and text messages, (2) how VA monitors VCL primary center call center operations, and (3) how VA works with VCL service partners to help ensure veterans receive high-quality service. GAO visited the VCL's primary center and two backup call centers; tested VCL response time through a generalizable sample of covert telephone calls and a nongeneralizable sample of text messages in July and August 2015; reviewed internal reports and policies and plans; and interviewed VA and SAMHSA officials.
section_title: What GAO Found
paragraphs: GAO found that the Department of Veterans Affairs (VA) did not meet its call response time goals for the Veterans Crisis Line (VCL), although extended call wait times were not common. VA's goal is to answer 90 percent of VCL calls at the VCL primary center within 30 seconds. Currently, calls not answered within 30 seconds route to VCL backup call centers; however, for 5 months of fiscal year 2015, calls were routed to VCL backup call centers after 60 seconds. VA officials told GAO that VA data show about 65 to 75 percent of VCL calls were answered at the VCL primary center in fiscal year 2015 within either 30 or 60 seconds. GAO's covert testing in July and August 2015 confirms VA's data. Specifically, 119 covert test calls show that an estimated 73 percent of calls made during this period were answered within 30 seconds. GAO also estimates that 99 percent of all VCL calls during this period were answered within 120 seconds. GAO also covertly tested the VCL's text messaging services and found that 4 of 14 GAO test text messages did not receive responses. VA officials said they do not monitor or test the timeliness and performance of the VCL text message system and instead rely solely on the VCL's text messaging provider for these functions. VA officials told GAO that the provider had not reported any issues with the system, but the provider told GAO that routine testing of the VCL system is not conducted. Without routinely testing its text messaging system or ensuring that its provider does so, VA cannot identify limitations to this service.
While VA has taken a number of steps to improve its monitoring of the VCL primary center operations, VA has not developed measurable targets and time frames for its key performance indicators, such as the program's percentage of abandoned calls. VA established a permanent VCL call center evaluation team and created a mechanism for tracking complaints about the performance of the VCL primary center from VCL callers or external parties. However, GAO found that VA has not specified quantifiable or otherwise measurable targets and has not included dates for when it would expect the VCL to complete actions covered by each key performance indicator. This is inconsistent with guidance provided by the Office of Management and Budget. As a result, VA cannot ensure that the VCL is providing consistent, high-quality services to callers and cannot effectively track and publicly report progress or results.
VA established an interagency agreement with its service partner, the Department of Health and Human Services' (HHS) Substance Abuse and Mental Health Services Administration (SAMHSA), to manage the shared operations of the VCL and the National Suicide Prevention Lifeline (Lifeline), which include a single national toll-free number used by both. Despite these efforts to coordinate, VA and SAMHSA do not collect information needed to assess how often and why callers intending to reach the VCL do not follow voice prompts and instead reach Lifeline local crisis centers. VA officials told GAO that the type of information that would be needed to do so is not collected because VA has focused on addressing the concerns of those callers who did reach the VCL. In addition, SAMHSA officials said that they do not require Lifeline local crisis centers to collect this type of information, noting that it would be possible to collect it. As a result, VA and SAMHSA do not know the extent to which this occurs and cannot determine the underlying causes that may need to be addressed.
section_title: What GAO Recommends
paragraphs: GAO recommends that VA regularly test VCL's text messaging system and document targets and time frames for key performance indicators. GAO also recommends that VA and SAMHSA collect information on how often and why callers reach Lifeline when intending to reach the VCL, review this information, and, if necessary, develop plans to address the causes. VA and HHS concurred with GAO's recommendations and described planned actions to address them.

section_title: Letter
paragraphs: Upon returning home from deployments in Afghanistan, Iraq, Vietnam,  and other locations, many servicemembers have struggled with mental  health issues, including post-traumatic stress disorder, depression, and  substance abuse. Several of these mental health issues have been  identified as risk factors for suicide among veterans. The Department of  Veterans Affairs (VA) provides an array of health care services, including  mental health services and other benefits to over 21 million veterans and  their families. As part of the continuum of services it provides, VA  established the Veterans Crisis Line (VCL) in July 2007. The VCL  supports veterans in emotional crisis and helps implement VA’s goal of  improving mental health outcomes for servicemembers, veterans, and  their families through a number of actions—including reducing barriers to  seeking mental health treatment and expanding access to VA services.
Veterans can access the VCL by calling a national toll-free number—  1-800-273-TALK (8255)—and following instructions to press “1” to be  connected with a VCL responder. The VCL and the National Suicide  Prevention Lifeline (Lifeline)—a Substance Abuse and Mental Health  Services Administration (SAMHSA) grantee—share this national toll-free  number through an interagency agreement between VA and SAMHSA.  Currently, the VCL operates through a VA-operated primary center  staffed with VA-employed responders and five backup call centers that  provide additional responders and other services through a backup call  coverage contract. The VCL is distinct from Lifeline, which operates  through a network of private, nonprofit providers working independently of  one another while maintaining agreed-upon clinical standards. In  addition to responding to calls, the VCL can also be accessed via online  chat and text message.
Demand for the VCL’s services has exceeded VA’s expectations.  Specifically, in its first full year of operation in fiscal year 2008, the VCL  received about 67,000 calls compared to about 534,000 calls received in  fiscal year 2015, an almost 700 percent increase. In response, VA  steadily increased the VCL’s spending from about $3 million to $30 million  from fiscal year 2008 through fiscal year 2015, devoting additional staff  and resources to the VCL over time. As VA endeavored to address  increasing numbers of requests for assistance, reports of dissatisfaction  with VCL service periodically appeared in the media, and the VA Office of  Inspector General was asked to investigate complaints about the VCL’s  lack of timely response to callers. The Inspector General identified gaps  in the VCL quality-assurance process, including challenges associated  with supervisory review, tracking of issues, and collection and analysis of  data from VCL backup call centers. In addition, the Inspector General  found that in some cases callers did not receive immediate assistance  from responders.
In response to accounts of dissatisfaction with VCL services, you asked  us to assess the effectiveness of VA’s policies and procedures designed  to provide timely assistance to veterans through the VCL. In this report,  we examine  1.  the extent to which VA meets response-time goals for calls, online  chats, and text messages received through the VCL;  2.  how VA monitors the performance of the VCL primary center  responders and call center operations;  3.  how VA works with VCL service partners—backup call centers and  SAMHSA—to help ensure veterans receive high-quality service from  responders; and  4.  what plans VA has to improve the VCL operations.
To examine the extent to which VA meets response-time goals for calls,  online chats, and text messages received through the VCL, we made  covert test telephone calls, text messages, and online chats to measure  the VCL’s response times. The test calls included a generalizable sample  of 119 calls that can be used to describe all callers’ wait times when  calling the VCL during July and August of 2015. We also sent a  nongeneralizable sample of 15 test online chats to the VCL and 14 test  text messages during the same period. For these tests, we measured the  length of time between sending the outgoing message to the VCL and  receiving a response from a VCL responder. In addition, we examined  telephone call, online chat, and text message data and summary reports  from January 2013 through December 2015 that VA maintains related to  the timeliness of the VCL’s operations. To determine the validity of these  data, we interviewed officials responsible for managing them and  compared our test call response data with VA’s reported telephone data  for July and August 2015. On the basis of these actions, we found these  data to be sufficiently reliable for the purposes of describing the quantity  of requests for services reaching the VCL. We used these data to  evaluate the timeliness of the VCL’s call response and compared the data  to the department’s own goals. Finally, we spoke with the VCL text  messaging service provider to learn about its operations.
To examine how VA monitors the performance of VCL primary center  responders and call center operations, we reviewed relevant  documents—including VCL policies and procedures—and interviewed VA  officials. In addition, we reviewed data and information that resulted from  VCL monitoring. Finally, we compared VA’s use of key performance  indicators to the Office of Management and Budget’s guidance on  performance goals, which are consistent with the Government  Performance and Results Modernization Act of 2010.
To examine how VA works with VCL service partners—backup call  centers and SAMHSA—to help ensure veterans receive high-quality  service from responders, we conducted site visits to the VCL primary  center and two of the five VCL backup call centers, where we observed  call center operations and interviewed VA officials and backup call center  representatives. We also interviewed SAMHSA officials and  representatives of the remaining three VCL backup call centers by  teleconference. In addition, we interviewed officials from the Veterans  Health Administration’s (VHA) Office of Mental Health Operations  (OMHO)—the VA Central Office entity responsible for overseeing the  VCL—and representatives from the contractor that manages VCL backup  call coverage. Additionally, we reviewed VA’s contract that provides  backup call coverage and VA’s interagency agreement with SAMHSA.  Finally, we conducted a nongeneralizable sample of 34 calls in August  2015 in which we mimicked the experience of veterans who do not follow  the instructions of a voice prompt to press “1” to reach the VCL.
To examine what plans VA has to improve the VCL operations, we  reviewed VA’s documented improvement plans and interviewed VA  officials responsible for planning and implementing those improvements.  For additional details about the scope and methodology used in this  report, see appendix I.
We conducted this performance audit from February 2015 to May 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. We conducted our related  investigative work from July through August 2015 in accordance with the  standards prescribed by the Council of the Inspectors General on Integrity  and Efficiency.
subsections: section_title: Background
subsections: section_title: VCL Structure
paragraphs: In 2007, VA established the VCL, a 24-hour crisis line staffed by  responders trained to assist veterans in emotional crisis. Through an  interagency agreement, VA collaborated with SAMHSA to use a single,  national toll-free number for crisis calls that serves both Lifeline and the  VCL. Through this interagency agreement, VA and SAMHSA set out to  establish a seamless crisis-management system through a collaborative  and cooperative relationship between the agencies that would provide  consistent suicide-prevention techniques to callers.
The national toll-free number presents callers with choices. Callers are  greeted by a recorded message that explains the function of the crisis line  and prompts individuals to press “1” to reach the VCL. Callers who do not  press “1” by the end of the message are routed to one of Lifeline’s 164  local crisis centers. All callers who press “1” are routed first to the VCL  primary center. Calls that are not answered at the VCL primary center  within 30 seconds of the time that the caller presses “1” during the Lifeline  greeting are automatically routed to one of five VCL backup call centers.  If a call is not answered by the VCL backup call center that initially  receives it, the call may be sent to another VCL backup call center. VA  entered into a contract with a firm to oversee the operations of the VCL  backup call centers. There are a total of 164 Lifeline local crisis centers, 5  of which also serve the VCL. (See fig. 1.)
section_title: VCL Workload and Funding Obligations
paragraphs: The number of calls reaching the VCL has increased substantially since  the VCL’s first full year of operation. Increases in the number of VCL calls  received have corresponded with increased annual funding obligations for  the VCL. (See fig. 2.)
VA added online chat and text message capabilities to the VCL in fiscal  years 2009 and 2012, respectively. The number of online chats and text  messages handled by the VCL generally increased every year, though  the number of online chats decreased in fiscal year 2015. (See fig. 3.)
section_title: Extended Call Wait Times Were Uncommon in July and August 2015, but VA Did Not Meet Its Call Response Time Goals and Some Text Messages Did Not Receive Responses
subsections: section_title: VA Responded to Most Calls within 30 Seconds in July and August 2015, but Did Not Meet Its Goal to Answer 90 Percent of Calls within 30 Seconds at the VCL Primary Center
paragraphs: To determine how well VA performed against its goal for responding to  VCL callers, we covertly tested the VCL’s call response time in July and  August 2015. During this testing we found that it was uncommon for VCL  callers to wait an extended period before reaching a responder since all  of our calls that reached the VCL were answered in less than 4 minutes.  According to VA officials, VA established a goal that the VCL primary  center would answer 90 percent of calls to the VCL within 30 seconds.  Our test included a generalizable sample of 119 test calls that can be  used to describe all callers’ wait times when calling the VCL during this  period. On the basis of our test calls, we estimate that during July and  August 2015 about 73 percent of all VCL calls were answered at the VCL  primary center within 30 seconds. (See fig. 4.) VA officials told us that,  during fiscal year 2015, about 65 to 75 percent of VCL calls were  answered at the VCL primary center and about 25 to 35 percent of VCL  calls were answered at the backup call centers. These VA-reported  results indicate that about 65 to 75 percent of VCL calls were answered  within either 30 or 60 seconds. These results are consistent with our  test results for July and August 2015. According to VA officials, VA  attempts to maximize the percentage of calls answered at the VCL  primary center because these responders have additional resources— including access to veterans’ VA electronic medical records—that are  unavailable to VCL backup call center responders. All responders receive  specialized training to assist callers in crisis.
section_title: To Help Achieve Response-Time Goals, VA Implemented Changes at the VCL Primary Center
paragraphs: To improve its performance toward meeting the goal of answering 90  percent of calls at the VCL primary center within 30 seconds, VA  implemented two changes in fiscal year 2015—namely, staggered work  shifts for responders and new call-handling procedures.
Staggered work shifts. VA implemented staggered shifts for responders  at the VCL primary center on September 6, 2015. Staggered shifts are  work schedules that allow employees to start and stop their shifts at  different times as a way to ensure better coverage during peak calling  periods. Specifically, it helps schedule more employees to work when call  volume is highest and fewer employees to work when call volume is  lowest. Additionally, staggered shifts help limit disruptions in service as  responders begin and end their shifts. By comparing VCL telephone call  data from September through December of 2014 to that of September  through December of 2015, we found that VA’s implementation of  staggered shifts at the VCL primary center had mixed results.
Overall: The average percentage of calls answered per hour at the  VCL primary center from September through December 2015—after  staggered shifts were implemented—was 75 percent, slightly less  than the average of 79 percent answered during the corresponding  period in 2014 before staggered shifts were implemented. However,  the VCL received an average of about 1.3 more calls per hour during  this period in 2015 than it received during the corresponding period in  2014 and, according to VA officials, the VCL primary center employed  fewer responder staff in 2015 than 2014.
By day of the week: The average percentage of calls answered per  hour at the VCL primary center increased on Mondays to 89 percent  and Tuesdays to 83 percent after VA implemented staggered shifts,  up from 78 percent and 79 percent, respectively, during the  corresponding period in 2014. These increases suggest that  staggered shifts may have helped VA answer more calls at the VCL  primary center on these days because VCL call data from our analysis  indicate that these days of the week typically experienced the highest  number of calls prior to implementing staggered shifts, and VA  officials told us that they used the implementation of staggered shifts  to schedule more responders on these days. However, the average  percentage of calls answered per hour at the VCL primary center  decreased on Saturdays to 61 percent and Sundays to 70 percent  after VA implemented staggered shifts, down from 78 percent and 80  percent, respectively, during the corresponding period in 2014.
By hours of the day: VA answered a higher percentage of calls at the  VCL primary center during the mid-day and evening hours after the  implementation of staggered shifts. Specifically, from 11:00 a.m. to  4:00 p.m. and from 9:00 p.m. to 11:00 p.m. the VCL primary center  answered a higher percentage of calls compared with the  corresponding periods in 2014. However, VA answered a lower  percentage of calls at the VCL primary center during overnight  hours—midnight to 9:00 a.m.—and in the early evening—5:00 p.m. to  8:00 p.m.—compared to corresponding periods in 2014.
To address staffing limitations and align the number of responders  available for each staggered shift according to demand, VA officials said  that VA planned to hire 63 additional responders for the VCL primary  center in fiscal year 2016 and assign these new responders to weekend  and evening shifts. This change would likely help improve the mixed  results we identified in our analysis of VA’s implementation of staggered  shifts for responders. As of February 2016, VA officials said that 22  applicants had accepted employment offers and that VA planned to  extend employment offers to an additional 15 applicants. These officials  also noted that recent attrition at the VCL primary center was largely due  to VCL primary center responders being promoted into new positions at  the VCL primary center or to VCL primary center responders leaving  because their work with the VCL did not qualify as clinical hours required  for licensure in their specialties.
New call-handling procedures. VA implemented new call handling  procedures at the VCL primary center beginning in June 2015 that  provided responders with specific guidance to more efficiently handle  “noncore” callers—those callers who were not seeking crisis assistance  but rather seeking help with other issues, such as help with veterans’  benefits questions. For example, if a caller reached the VCL with a  question about VA disability benefits, the VCL primary center responder  would verify that the caller was not in crisis and transfer the caller to the  Veterans Benefits Administration to address the question. VCL telephone  call data provided by VA suggest that the average time VCL primary  center responders spend handling noncore calls decreased by  approximately 30 percent over a 5-month period after responder training  began on these new call-handling procedures. We would expect that as  the average time VCL primary center responders spend handling noncore  calls decreases, these responders should have more time available to  answer additional incoming calls.
section_title: In July and August 2015, Most of Our Test Online Chats Were Answered Within 30 Seconds, but VA Did Not Ensure That Veterans Receive Responses through Its Text Messaging Service
paragraphs: To determine the timeliness of the VCL’s responses to online chats and  text messages, we conducted a covert test in July and August 2015 using  nongeneralizable samples of 15 online chats and 14 text messages. All  15 of our test online chats received responses within 60 seconds, 13 of  which were within 30 seconds. This result is consistent with VA data that  indicate VCL responders sent responses to over 99 percent of online chat  requests within 1 minute in fiscal years 2014 and 2015. VA officials said  that all online chats are expected to be answered immediately. Although  this is an expectation, VA does not yet have formal performance  standards for how quickly responders should answer online chat requests  and expects to develop them before the end of fiscal year 2016.
However, our tests of text messages revealed a potential area of concern.  Four of our 14 test text messages did not receive a response from the  VCL. Of the remaining 10 test text messages, 8 received responses  within 2 minutes, and 2 received responses within 5 minutes.
VA officials stated that text messages are expected to be answered  immediately, but, as with online chats, VA has not yet developed formal  performance standards for how quickly responders should answer text  messages. VA data indicate that VCL responders sent responses to 87  percent of text messages within 2 minutes of initiation of the conversation  in both fiscal years 2014 and 2015. VA officials said that VA plans to  establish performance standards for answering text messages before the  end of fiscal year 2016. VA officials noted and we observed during a site  visit that some incoming texts were abusive in nature or were not related  to a crisis situation. According to VA officials, in these situations, if this  is the only text message waiting for a response, a VCL responder will  send a response immediately. However, if other text messages are  awaiting responses, VA will triage these text messages and reply to those  with indications of crisis first. This triage process may have contributed to  the number of our test texts that did not receive responses within 2  minutes.
The VCL’s text messaging service provider offered several reasons for  the possible nonresponses that we encountered in our test results. These  included: (1) incompatibilities between some devices used to send text  messages to the VCL and the software VA used to process the text  messages, (2) occasional software malfunctions that freeze the text  messaging interface at the VCL primary center, (3) inaudible audio  prompts used to alert VCL primary center responders of incoming text  messages, (4) attempts by people with bad intentions to disrupt the VCL’s  text messaging service by overloading the system with a large number of  texts, and (5) incompatibilities between the web-browsers used by the  VCL primary center and the text messaging software.
VA officials told us that they do not monitor and test the timeliness and  performance of the VCL text messaging system, but rather rely solely on  the VCL’s text messaging service provider for such monitoring and  testing. They said that the provider had not reported any issues with this  system. According to the provider, no routine testing of the VCL’s text  messaging system is conducted. Standards for internal control in the  federal government state that ongoing monitoring should occur in the  course of normal operations, be performed continually, and be ingrained  in the agency’s operations. Without routinely testing its text messaging  system, or ensuring that its provider tests the system, VA cannot ensure  that it is identifying limitations with its text messaging service and  resolving them to provide consistent, reliable service to veterans.
section_title: VA Has Taken Steps to Improve Its Monitoring of VCL Primary Center Performance but Has Not Established Targets and Time Frames for VCL Key Performance Indicators
subsections: section_title: VA Established a Call Center Evaluation Team, Implemented Revised Responder Performance Standards, and Analyzed VCL Caller Complaints
paragraphs: VA has sought to enhance its capabilities for overseeing VCL primary  center operations through a number of activities—including establishing a  call center evaluation team, implementing revised performance standards  for VCL primary center responders, implementing silent monitoring of  VCL primary center responders, and analyzing VCL caller complaints.
Establishment of a call center evaluation team. In October 2013, VA  established a permanent VCL call center evaluation team that is  responsible for monitoring the performance of the VCL primary center.  The call center evaluation team analyzes VCL data, including information  on the number of calls received and the number of calls routed to backup  call centers from the primary center. VA officials told us that they use  these data to inform management decisions about VCL operations. For  example, these data were used as part of its decision to implement  staggered shifts for VCL primary center responders in an attempt to  increase the number of calls answered at the VCL primary center.
Implementation of revised performance standards for VCL primary  center responders. In October 2015, VA implemented new performance  standards for all VCL primary center responders that will be used to  assess their performance in fiscal year 2016. According to VA officials,  these performance standards include several measures of responder  performance—such as demonstrating crisis-intervention skills, identifying  callers’ needs, and addressing those needs in an appropriate manner  using VA approved resources. VA officials told us that by the summer of  2016 VCL primary center supervisors will have access to real-time  information on VCL primary center responders’ performance against  these standards and can track their workload and performance  periodically. These officials explained that they anticipate these  performance standards will be reviewed and revised as needed for the  fiscal year 2017 performance year.
Silent monitoring of VCL primary center responders. In February  2016, VA officials reported that they were beginning silent monitoring of  all VCL responders using recently developed standard operating  procedures, standard data collection forms, and standard feedback  protocols. These officials explained that the VCL primary center silent  monitoring would begin in mid-February 2016 with four VA medical  center–based suicide-prevention coordinators completing silent  monitoring of 15 to 20 calls a week to the VCL primary center through  March 2016. These officials explained that six full-time silent monitors  had been hired as part of the VCL quality assurance staff and would  begin conducting silent monitoring of VCL primary center calls in April  2016 once their training had been completed. During the initial rollout, the  four VA medical center–based suicide-prevention coordinators will  remotely access VCL primary center calls, complete the standard data  collection form, and send the information to the observed VCL primary  center responders’ supervisors for feedback delivery. Once the six full- time silent monitors begin completing these activities, they will complete  all call monitoring and deliver feedback to VCL primary center responders  and will coordinate with VCL primary center supervisors on an as-needed  basis. VA officials explained in February 2016 that they were unsure how  many VCL primary center calls these six full-time silent monitors would be  able to observe and will clarify this expectation once these silent monitors  begin their duties in April 2016.
Analysis of VCL caller complaints. In October 2014, VA created a  mechanism for tracking complaints it receives from VCL callers and  external parties, such as members of Congress and veterans, about the  performance of the VCL primary and backup call centers. Complaints can  be about services provided by either the VCL primary center or one of the  VCL backup call centers. In fiscal year 2015, the VCL received over 200  complaints from veterans and others regarding call center operations.  These complaints included issues with VCL primary center and backup  call center customer service and wait times to reach a responder.  According to VA officials, each complaint is investigated to validate its  legitimacy and determine the cause of any confirmed performance  concerns. This validation process includes speaking with the complainant  and VA staff, as applicable. The results and disposition of each complaint  are documented in VA’s complaint tracking database. For complaints  that include details on specific responders, VA officials told us that they  investigate complaints and use legitimate complaints as part of the  performance evaluation process for VCL primary center responders.  Specifically, these officials explained that when a complaint about a VCL  primary center responder’s customer service is verified as accurate by a  VA psychologist or supervisor after it is investigated, it can affect a VCL  primary center responder’s annual performance appraisal. The  investigation process also includes verifying any associated  documentation of the activities at the source of the complaint.
section_title: VCL Key Performance Indicators Lack Measureable Targets and Time Frames
paragraphs: In 2011, VA established key performance indicators to evaluate VCL  primary center operations; however, we found these indicators did not  have established measureable targets or time frames for their completion.
VCL key performance indicators lack measurable targets. We found  that VA’s list of VCL key performance indicators did not include  information on the targets the department had established to indicate their  successful achievement. For example, VA included a key performance  indicator for the percentage of calls answered by the VCL in this list but  did not include information on what results would indicate success for (1)  the VCL as a whole, (2) the VCL primary center, or (3) the VCL backup  call centers. As another example, VA did not establish targets for the  percentage of calls abandoned by callers prior to speaking with VCL  responders. Measureable targets should include a clearly stated minimum  performance target and a clearly stated ideal performance target. These  targets should be quantifiable or otherwise measurable and indicate how  well or at what level an agency or one of its components aspires to  perform. Such measurable targets are important for ensuring that the  VCL call center evaluation team can effectively measure VCL  performance.
VCL key performance indicators lack time frames for their  completion. We found that VA’s list of VCL key performance indicators  did not include information on when the department expected the VCL to  complete or meet the action covered by each key performance indicator.  For example, for VA’s key performance indicator for the percentage of  calls answered by the VCL, the department did not include a date by  which it would expect the VCL to complete this action. As another  example, VA did not establish dates by which it would meet targets yet to  be established for the percentage of calls abandoned by callers prior to  speaking with VCL responders. Time frames for action are a required  element of performance indicators and are important to ensure that  agencies can track their progress and prioritize goals.
Guidance provided by the Office of Management and Budget states that  performance goals—similar to VA’s key performance indicators for the  VCL—should include three elements: (1) a performance indicator, which  is how the agency will track progress; (2) a target; and (3) a period. VA  officials reported that they are currently implementing a comprehensive  process improvement plan, discussed later in this report, that will help  ensure the right structures and processes are in place, which they believe  are logical precursors to examining VCL outcomes and establishing  targets and time frames for performance indicators. Without establishing  targets and time frames for the successful completion of its key  performance indicators for the VCL, VA cannot effectively track and  publicly report progress or results for its key performance indicators for  accountability purposes.
section_title: VA Is Strengthening Requirements for VCL Backup Call Centers, but VA and SAMHSA Do Not Collect Information to Assess How Often and Why Callers Are Not Reaching the VCL
subsections: section_title: VA Is Enhancing Performance Requirements for Its Backup Call Coverage Contractor
paragraphs: VA’s backup call coverage contract, awarded in October 2012 and in  place at the time of our review, did not include detailed performance  requirements in several key areas for the VCL backup call centers. Clear  performance requirements for VCL backup call centers are important for  defining VA’s expectations of these service partners. However, VA has  taken steps to strengthen the performance requirements of this contract  by modifying it in March 2015 and beginning the process of replacing it  with a new contract.
October 2012 backup call coverage contract. This contract provided a  network of Lifeline local crisis centers that could serve as VCL backup call  centers managed by a contractor. This contractor was responsible for  overseeing and coordinating the services of VCL backup call centers that  answer overflow calls from the VCL primary center. This contract as  initially awarded included few details on the performance requirements for  VCL backup call centers. For example, the contract did not include any  information on several key aspects of VCL backup call center  performance, including: (1) the percentage of VCL calls routed to each  VCL backup call center that should be answered, (2) VA’s expectations  on whether or not VCL backup call centers could use voice answering  systems or caller queues for VCL calls, and (3) VA’s documentation  requirements for VCL calls answered at the VCL backup call centers.  Detailed performance requirements on these key aspects of VCL backup  call center performance are necessary for VA to effectively oversee the  performance of the contractor and the VCL backup call centers. By not  specifying performance requirements for the contractor on these key  performance issues, VA missed the opportunity to validate contractor and  VCL backup call center performance and mitigate weaknesses in VCL  call response. For example, representatives from one VCL backup call  center provided data that showed that the backup call center answered  about 50 percent of the VCL calls it received. However, without a  performance requirement establishing a standard for the percentage of  calls each VCL backup call center should answer, VA could not determine  whether this was acceptable performance for a VCL backup call center.  As of December 2015, this VCL backup call center reported that it had  improved its performance and answered about 66 percent of calls it  received from July 2015 to December 2015.
VA officials told us about several concerns with the performance of the  backup call centers operating under the October 2012 contract based on  their own observations and complaints reported to the VCL. These  concerns included the inconsistency and incompleteness of VCL backup  call centers’ responses to VCL callers, limited or missing documentation  from records of VCL calls answered by VCL backup call center  responders, limited information provided to VA that could be used to track  VCL backup call center performance, and the use of voice answering  systems or extended queues for VCL callers reaching some VCL backup  call centers. For example, VA officials reported that some veterans did  not receive complete suicide assessments when their calls were  answered at VCL backup call centers. In addition, VA officials noted that  they had observed some VCL backup call centers failing to follow VCL  procedures, such as not calling a veteran who may be in crisis when a  third-party caller requested that the responder contact the veteran.  According to VA officials, these issues led to additional work for the VCL  primary center, including staffing one to two responders per shift to review  the call records submitted to the VCL primary center by backup call  centers and to determine whether these calls required additional follow-up  from the VCL primary center. These officials estimated that 25 to 30  percent of backup call center call records warranted additional follow-up  to the caller from a VCL primary center responder, including  approximately 5 percent of backup call center call records that needed to  be completely reworked by a VCL primary center responder.
March 2015 backup call coverage contract modification. Given these  concerns, in March 2015 VA modified the October 2012 backup call  coverage contract to add more explicit performance requirements for its  backup call coverage contractor, which likely took effect more quickly  than if the department had waited for a new contract to be awarded.  These modified requirements included (1) the establishment of a 24- hours-a-day, 7-days-a-week contractor-staffed emergency support line  that VCL backup call centers could use to report problems, (2) a  prohibition on VCL backup call centers’ use of voice answering systems,  (3) a prohibition on VCL backup call centers placing VCL callers on hold  before a responder conducted a risk assessment, (4) documentation of  each VCL caller’s suicide risk assessment results, and (5) transmission of  records for all VCL calls to the VCL primary center within 30 minutes of  the call’s conclusion.
Development of new backup call coverage contract. In July 2015, VA  began the process of replacing its backup call coverage contract by  publishing a notice to solicit information from prospective contractors on  their capability to satisfy the draft contract terms for the new contract; this  new backup call coverage contract was awarded in April 2016. We  found that these new proposed contract terms included the same  performance requirement modifications that were made in March 2015,  as well as additional performance requirements and better data reporting  from the contractor that could be used to improve VA’s oversight of the  VCL backup call centers. Specifically, the proposed contract terms added  performance requirements to address VCL backup call center  performance—including a requirement for 90 percent of VCL calls  received by a VCL backup call center to be answered by a backup call  center responder within 30 seconds and 100 percent to be answered by a  backup call center responder within 2 minutes. In addition, the proposed  contract terms include numerous data reporting requirements that could  allow VA to routinely assess the performance of its VCL backup call  centers and identify patterns of noncompliance with the contract’s  performance requirements more efficiently and effectively than under the  prior contract. The proposed terms for the new contract also state that VA  will initially provide and approve all changes to training documentation  and supporting materials provided to VCL backup call centers in order to  promote the contractor’s ability to provide the same level of service that is  being provided by the VCL primary center.
section_title: VA and SAMHSA Do Not Collect Information Needed to Assess How Often and Why Callers Do Not Reach the VCL
paragraphs: We found that when callers do not press “1” during the initial Lifeline  greeting, their calls may take longer to answer than if the caller had  pressed “1” and been routed to either the VCL primary center or a VCL  backup call center. As previously discussed, VA and SAMHSA  collaborated to link the toll-free numbers for both Lifeline and the VCL  through an interagency agreement. The greeting instructs callers to press  “1” to be connected to the VCL; if callers do not press “1,” they will be  routed to one of SAMHSA’s 164 Lifeline local crisis centers. To mimic the  experience of callers who do not press “1” to reach the VCL when  prompted, we made 34 covert nongeneralizable test calls to the national  toll-free number that connects callers to both Lifeline and the VCL during  August 2015 and we did not press “1” to be directed to the VCL. For 23  of these 34 calls, our call was answered in 30 seconds or less. For 11 of  these calls, we waited more than 30 seconds for a responder to answer— including 3 calls with wait times of 8, 9, and 18 minutes. Additionally, one  of our test calls did not go through, and during another test call we were  asked if we were safe and able to hold. VA’s policy prohibits VCL  responders from placing callers on hold prior to completing a suicide  assessment; Lifeline has its own policies and procedures.
According to officials and representatives from VA, SAMHSA, and the  VCL backup call centers, as well as our experience making test calls  where we did not press “1,” there are several reasons why a veteran may  not press “1” to be routed to the VCL, including an intentional desire to not connect with VA, failure to recognize the prompt to press “1” to be directed to the VCL, waiting too long to respond to the prompt to press “1” to be directed to  the VCL, or  calling from a rotary telephone that does not allow the caller to press  “1” when prompted.
VA officials said they had not estimated the extent to which veterans  intending to reach the VCL did not press “1” during the Lifeline greeting.  These officials explained that their focus has been on ensuring that  veterans who did reach the VCL received appropriate service from the  VCL primary center and backup call centers. In addition, SAMHSA  officials said that they also do not collect this information. These officials  reported that SAMHSA does not require the collection of demographic  information, including veteran status, for a local crisis center to participate  in the Lifeline network. However, they noted that SAMHSA could request  through its grantee that administers the Lifeline network that local crisis  centers conduct a onetime collection of information to help determine how  often and why veterans reach Lifeline local crisis centers. SAMHSA  officials explained that they could work with the Lifeline grantee to explore  optimal ways of collecting this information that would be (1) clinically  appropriate, (2) a minimal burden to callers and Lifeline’s local crisis  centers, and (3) in compliance with the Office of Management and  Budget’s paperwork reduction and information collection policies. The  interagency agreement between VA and SAMHSA assigns SAMHSA  responsibilities for monitoring the use of the national toll-free number—  1-800-273-TALK (8255)—that is used to direct callers to both the VCL  and Lifeline. These responsibilities include monitoring the use of the line,  analyzing trends, and providing recommendations about projected needs  and technical modifications needed to meet these projected needs. Using  the information collected from the Lifeline local crisis centers on how often  and why veterans reach Lifeline, as opposed to the VCL, VA and  SAMHSA officials could then assess whether the extent to which this  occurs merits further review and action.
Although the results of our test are not generalizable, substantial wait  times for a few of our covert calls suggest that some callers may  experience longer wait times to speak with a responder in the Lifeline  network than they would in the VCL’s network. Without collecting  information to examine how often and why veterans do not press “1”  when prompted to reach the VCL, VA and SAMHSA cannot determine the  extent veterans reach the Lifeline network when intending to reach the  VCL and may experience longer wait times as a result. In addition,  limitations in information on how often and why this occurs do not allow  VA and SAMHSA to determine whether or not they should collaborate on  plans to address the underlying causes of veterans not reaching the VCL.  Standards for internal control in the federal government state that  information should be communicated both internally and externally to  enable the agency to carry out its responsibilities. For external  communications, management should ensure there are adequate means  of communicating with, and obtaining information from, external  stakeholders that may have a significant impact on the agency achieving  its goals.
section_title: VA Has Developed and Begun Implementation of Plans to Improve VCL Operations
paragraphs: In June 2014, VA assessed the operational state of the VCL and, based  on its findings, designed a performance-improvement plan that outlined  actions to address problems VA identified regarding the VCL’s workforce,  processes, technology, and infrastructure. To implement this plan, in  March 2015 VA began a series of rapid process-improvement events,  such as improvements to VCL primary center responder training,  designed to solve problems identified by VCL staff and stakeholders with  actions that could be implemented within 60 to 90 days.
According to VA officials and documentation provided by these officials,  these rapid process-improvement events led to several changes at the  VCL primary center in 2015 and 2016. As we previously noted, these  changes include implementation of staggered shifts; development of  silent monitoring procedures, and the hiring of dedicated staff to complete  this monitoring; and new call-handling procedures previously discussed.  They also include some remaining follow-up activities, such as completing  the implementation of remaining planned quality-assurance activities in  fiscal year 2016. These measures—if fully implemented—represent  positive steps to improve VCL operations.
VA has developed additional plans to address other concerns with VCL  operations. These plans address issues at the VCL primary center related  to renovation of new space, upgrades to telecommunications, and the  introduction of a caller queue.
Renovation of new space for VCL primary center operations. We  found that the VCL primary center responders are housed in two different  buildings originally designed for patient care delivery. According to VA  officials, these buildings do not reflect call center leading practices that  recommend large, open rooms that provide supervisors greater access to  the responders they oversee. However, in February 2016, VA officials  reported that the department committed funding to relocate the VCL  primary center operations to a renovated space on the VA medical center  campus. The relocation is to be implemented in two phases. VA officials  expect that the first phase, which includes moving administrative and  monitoring staff, will be completed in June 2016; the second phase will  relocate the rest of the VCL staff, including all responders. VA officials  said they anticipate that the second phase will be completed in fiscal year  2017. VA officials told us that they plan on using the National Call Center- Health Resource Center’s large open-space layout as a model in  designing the VCL primary center’s new space. According to VA officials,  the National Call Center-Health Resource Center follows leading  practices for call center operations as set by the International Call  Management Institute.
Upgrade of VCL primary center telecommunication infrastructure.  VA officials told us that the VCL primary center uses the telephone  infrastructure of the VA medical center rather than a separate telephone  system that would be more conducive to operating a call center.  According to a telephone infrastructure change justification that VA  information-technology officials prepared, the VCL primary center’s  existing telephone system does not meet the requirements for operating a  call center of its size. This documentation indicates that improvements  are needed in several features of the VCL’s existing telephone system— including call routing, call recording, data capture, and automatic callback.  In February 2016, VA officials reported that planned improvements to the  VCL primary center’s telephone system would be implemented by June  2016; however, the VCL primary center will continue to operate using part  of the VA medical center’s telephone system.
Introduction of VCL primary center caller queue. VA’s evaluation of  the VCL conducted in 2014 noted that a possible option for improving  VCL call response would be to implement a queue at the VCL primary  center that would allow callers to wait a longer period for a VCL primary  center responder before being sent to a VCL backup call center.  Currently, VA allows VCL primary center responders 30 seconds to  answer calls before routing them to VCL backup call centers for a  response. In February 2016, VA officials told us that they are considering  implementing this type of queue. According to these officials, they are  considering allowing VCL calls to remain at the VCL primary center for up  to 5 minutes and they explained that this 5-minute period was determined  based on feedback they received from veterans on how long they would  be willing to wait for a responder. These officials further explained that  voice prompts would offer callers options as they waited in the queue to  reach the next available VCL primary center responder or to be  transferred to other VA call centers for concerns unrelated to crisis  situations.
section_title: Conclusions
paragraphs: The VCL plays an important role in providing a means by which veterans  and those concerned about them can discuss unique challenges and  crises they face, and provides a way to access VA’s mental health care  services. However, the rapid growth of the VCL in recent years has  coincided with operational and planning challenges that constrain its  ability to serve veterans in crisis in a timely and effective manner. To its  credit, VA has taken some interim but noteworthy steps to address these  challenges. Building on these steps, VA and SAMHSA need to take  additional actions to provide reasonable assurance that the VCL’s  mission to serve veterans and others in crisis situations is met.
As our testing demonstrates, VA has not yet achieved its goal of  answering 90 percent of all VCL calls within 30 seconds at the VCL  primary center, but its planned and recently implemented changes, such  as staggered shifts and enhanced call-handling procedures, are intended  to gain VA system efficiencies that will help the department meet its goal  once additional responders are hired. However, VA has not applied the  same level of attention to its text messaging service and does not  regularly test the VCL’s text messaging system. Without doing so, VA  cannot ensure that veterans are receiving timely responses from VCL  responders to their text messages.
In addition, while VA has taken a number of steps to improve its  monitoring of the VCL, VA continues to experience challenges related to  weaknesses in VCL key performance indicators—including a lack of  measurable targets and time frames. If left unresolved, these weaknesses  will likely have negative effects on VA’s ability to ensure the VCL is  providing the best service possible to veterans.
Despite efforts to coordinate the operations of the VCL and Lifeline  through an interagency agreement, VA and SAMHSA have not collected  information necessary to determine how often and why veterans intending  to reach the VCL reach Lifeline instead. As a result, neither VA nor  SAMHSA can assess the extent this occurs and the underlying causes  that may need to be addressed.
section_title: Recommendations for Executive Action
paragraphs: To improve the timeliness and quality of VCL responses to veterans and  others, we recommend that the Secretary of Veterans Affairs direct the  Under Secretary for Health to take the following two actions:  regularly test the VCL’s text messaging system to identify issues and  correct them promptly; and document clearly stated and measurable targets and time frames for  key performance indicators needed to assess VCL performance.
We further recommend that under the applicable terms of their  interagency agreement, the Secretary of Veterans Affairs and the  Secretary of Health and Human Services direct the Under Secretary for  Health and the Administrator of the Substance Abuse and Mental Health  Services Administration (SAMHSA), respectively, to collaborate and take  the following two actions:  collect information on how often and why callers intending to reach  the VCL instead reach Lifeline local crisis centers; and  review the information collected and, if necessary, develop plans to  address the underlying causes.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to VA and HHS for review and  comment. In their written comments, summarized below and reprinted in  appendixes II and III, both agencies concurred with our  recommendations. VA and HHS described ongoing or planned actions  and provided a timeline for addressing our recommendations. HHS also  provided technical comments, which we incorporated as appropriate.
In response to our first recommendation, to regularly test the VCL’s text  messaging system to identify issues and correct them promptly, VA said  that the VCL’s analytics department will develop and implement a more  robust and proactive system to test daily the VCL’s text messaging  service by July 2016. In the interim, VA stated that it has a process for  identifying, addressing, and troubleshooting problems that utilizes e-mail  templates to notify its contract text service provider of issues or errors that  require a response to troubleshoot the error.
In response to our second recommendation, to document clearly stated  and measurable targets and time frames for key performance indicators  needed to assess VCL performance, VA said that it is in the process of  developing a monthly scorecard with elements assessing call center,  staffing, quality-assurance, and crisis-response metrics with specific  performance targets. VA estimates that by October 2016 it would  establish targets and time frames for its performance indicators.
In response to our third recommendation, to collaborate with SAMHSA to  collect information on how often and why callers intending to reach the  VCL instead reach Lifeline local crisis centers, VA said that the VCL’s  newly formed Clinical Advisory Board would foster collaboration amongst  capable experts and leverage their collective expertise in facilitating an  improved experience for callers, greater operational efficiencies, and  increased access to the VCL for veterans in crisis. VA noted that the  Clinical Advisory Board included members of SAMHSA, the VA Suicide  Prevention Office, and other VA clinical offices. VA estimates that it would  collect sufficient data, conduct a collaborative analysis with SAMHSA,  and complete reporting to both agencies on this issue by October 2016.  HHS said that in response to this recommendation it would review ways  to collect data on callers intending to reach the VCL but instead reaching  Lifeline local crisis centers.
In response to our fourth recommendation, to collaborate with SAMHSA  to review the information collected and, if necessary, develop plans to  address the underlying causes for callers intending to reach the VCL  instead reaching Lifeline local crisis centers, VA said that the Clinical  Advisory Board referenced above would evaluate this issue as a standing  agenda item in its monthly meetings. VA said that the Clinical Advisory  Board would establish a baseline regarding the frequency of this issue’s  occurrence, monitor reported complaints about the press “1” functionality,  and provide us with data from Clinical Advisory Board meetings to  demonstrate action taken toward implementing our recommendation. VA  expects to complete these actions by January 2017. HHS said that in  response to this recommendation it would review the data collected as  described above and, if necessary, address the underlying causes as  appropriate.
These VA and HHS actions, if implemented effectively, would address the  intent of our recommendations.
In its technical comments, HHS emphasized the distinction between the  Lifeline network and the VCL, noting that the two programs operate with  different policies, procedures, and resources. We revised the draft to  more clearly reflect this distinction.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time we will send copies of this report to the  appropriate congressional committees, the Secretary of Veterans Affairs,  the Secretary of Health and Human Services, and other interested  parties. In addition, the report is available at no charge on the GAO  website at http://www.gao.gov.
If you or your staffs have any questions about this report, please contact  Seto J. Bagdoyan at (202) 512-6722 or bagdoyans@gao.gov, or Randall  B. Williamson at (202) 512-7114 or williamsonr@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be  found on the last page of this report. GAO staff who made key  contributions to this report are listed in appendix IV.
section_title: Appendix I: Scope and Methodology
subsections: section_title: Testing VCL Response Times
paragraphs: To determine the extent to which the Department of Veterans Affairs (VA)  meets response-time goals for calls, online chats, and text messages  received through the Veterans Crisis Line (VCL), we conducted several  tests of VCL services during July and August 2015. These tests were  designed to measure the timeliness of the VCL’s response to calls, online  chats, and text messages.
subsections: section_title: Tests of VCL Call Response Timeliness
paragraphs: We conducted a covert test of the VCL’s call response time using a  generalizable sample of 119 test calls placed in July and August 2015. To  develop this generalizable sample, we interviewed VA officials with  knowledge about VCL primary and backup call center operations;  obtained the VCL primary center’s historical call volume data in hourly  increments for fiscal year 2013 through the end of the second quarter of  fiscal year 2015; and generated a schedule of days and times during  which our test calls would be made. This test call schedule was created  by dividing the 62-day sample period into 496 primary sampling units,  which we defined as 3-hour blocks of time. We then defined secondary  sample units as 10-minute increments within each 3-hour block of time  and selected a stratified two-stage random cluster sample of 144 10- minute increments during which our test calls would be made. We  selected the 144 10-minute increments by: (1) stratifying the primary  sampling units into four strata—overnight, morning, afternoon, and  evening—based on time of day; (2) identifying a stratified sample of 36  primary sampling units that were allocated across the four strata based  on call volume and our available resources; and (3) randomly selecting  four 10-minute increments from each selected primary sampling unit.  The results of this test can be used to estimate all VCL callers’ wait times  for July and August 2015. Because we followed a probability procedure  based on random selections, our sample is only one of a large number of  samples that we might have drawn. Since each sample could have  provided different estimates, we express our confidence in the precision  of our particular sample’s results as a 95 percent confidence interval (i.e.,  a margin of error of within plus or minus a certain number of percentage  points). This is the interval that would contain the actual population value  for 95 percent of the samples we could have drawn. Percentage  estimates from our analysis included in this report have a margin of error  of within plus or minus 9 percentage points at the 95 percent confidence  level. Estimates of the median wait time have a margin of error of within  plus or minus 10 percent at the 95 percent confidence level.
When placing test calls, we used 20 telephone numbers with randomly  selected area codes to mask the origin of the calls. Two analysts then  independently measured and documented wait times by reviewing audio  recordings of each test call. Wait times were measured from the time that  the caller pressed “1” to reach the VCL and the time that a responder  answered the test call. The final wait time for each test call was the  lowest of the two wait times recorded by these analysts.
We successfully completed and measured the wait times for 119 test calls  in 30 of the 36 selected primary sampling units. We did not complete calls  in 25 of our selected 10-minute increments due to technical or scheduling  issues. The resulting completion rate for our test calls was 83 percent  (119 out of 144). The omitted test calls were distributed across all four  strata and were unrelated to the time of day.
section_title: Tests of VCL Online Chat and Text Message Response
paragraphs: To test VA’s online chat and text message response timeliness, we  reviewed VA’s procedures and training materials for operating both  services. We then interviewed and observed VCL responders at the VCL  primary center who responded to online chat and text messages. We also  spoke with the VCL text messaging service provider to learn about the  text messaging operations. To test the VCL’s online chat and text  message response, we scheduled one covert test online chat or text  message during each of the 30 primary sampling units used for the  generalizable sample described above and recorded our wait times for a  response. We measured the wait time for online chats and text  messages as the elapsed time between when we sent the online chat or  text message to the VCL and when we received a response from a  responder. We initiated our test online chats through a link provided on  the VCL’s website. We sent test text messages to the VCL through an  Internet text messaging service provider in order to record our test data  electronically. We removed one test text message attempt from the  sample because of technical issues we experienced that may have  prevented the message from reaching the VCL. As a result, our final  samples consisted of 15 test online chats and 14 test text messages.
section_title: Analysis of the VCL’s 2013 through 2015 Call Data
subsections: section_title: Data-Reliability Testing
paragraphs: We verified the reliability of VA’s reported VCL call data by interviewing  officials responsible for managing them and reviewing reports that VA’s  backup call coverage contractor provided to VA that documented the  time, duration, and routing of every VCL call. The routing information  included details on the call centers where each call was routed and  identified the call center that ultimately answered each call. We were able  to identify our test calls in these reports and confirmed that the data  matched records we maintained for our test calls. This exercise also  allowed us to confirm whether our test calls were answered at the VCL  primary center or a VCL backup call center. On the basis of these actions,  we found these data to be sufficiently reliable for the purposes of  describing the quantity of requests for services reaching the VCL. We  used these data to evaluate the timeliness of the VCL’s call response and  compared the data to the department’s own goals.
section_title: Analysis of VCL Call Data Related to the Implementation of Staggered Shifts at the VCL Primary Center
paragraphs: To assess the effectiveness of the implementation of staggered shifts for  responders at the VCL primary center, we compared VCL call data from  September 6, 2015, through December 31, 2015, to that of September 1,  2014, through December 31, 2014. We selected September 6, 2015, as  the start date for our 2015 period of analysis because it was the first day  that VA fully implemented staggered shifts at the VCL primary center. We  chose the cutoff of December 31, 2015, because it corresponded to the  most recent complete month of data available at the time of our analysis.  We used call data from September 1, 2014, through December 31, 2014,  because they reflected a comparable period from the year prior. We used  these 2014 data as a comparison group to account for any seasonality  patterns, variations, or fluctuations that might affect the demand for VCL  services within a particular season, day of the week, or other periods. Our  evaluation compared the average hourly call response percentages of the  periods we examined and included analysis for the time of day using  hourly intervals, day of the week, and holidays within each period. The  average hourly response percentages are likely affected by several  factors—such as call volume, staffing levels, and complexity of calls, for  which we did not control. Our analysis examined differences by day of the  week, time of day, and holidays, but did not control for the above- mentioned or other factors that may affect the percentage of calls  answered at the VCL primary center.
section_title: Test of Lifeline Local Crisis Center Response Times
paragraphs: To determine whether callers attempting to reach the VCL who did not  press “1” experienced longer wait times than those that did, we conducted  a nongeneralizable test. The VCL is accessed by calling a single national  toll-free number—1-800-273-TALK (8255)—shared by both the VCL and  the National Suicide Prevention Lifeline (Lifeline). This toll-free number is  managed by the Substance Abuse and Mental Health Services  Administration (SAMHSA). To conduct our nongeneralizable test, we  used a random sample of 34 covert test calls to conduct these tests  where we mimicked the experience of VCL callers who do not follow the  instructions of the voice prompt on this single national toll-free number to  press “1” in order to reach the VCL. To do this, we placed two test calls  where we did not press “1” as prompted to reach the VCL during each of  the scheduled primary sampling units in August 2015. We recorded the  wait times for each of the 34 test calls by calculating the amount of time  that elapsed between the moment that an automated message informed  us that the call was being transferred to a Lifeline local crisis center and  when a responder answered our call. We masked the origin of these calls  in a manner similar to that described for our generalizable sample of 119  test calls placed to the VCL. Although the 34 test calls were randomly  made, the results of these test calls are not generalizable due to the small  number of calls included in our sample.
section_title: Appendix II: Comments from the Department of Veterans Affairs
section_title: Appendix III: Comments from the Department of Health and Human Services
section_title: Appendix IV: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to the contacts named above, Gabrielle M. Fagan (Assistant  Director), Marcia A. Mann (Assistant Director), James D. Ashley, Dean  Campbell, Shaunessye D. Curry, Amber D. Gray, Katherine Nicole  Laubacher, Olivia Lopez, Maria McMullen, Brynn P. Rovito, Amber H.  Sinclair, and Shana B. Wallace made key contributions to this report.  Members of our investigative staff also contributed to the report.