text: Background VBA Disability Benefits Process VA pays monthly disability compensation to veterans with service- connected disabilities according to the severity of the disability. VA’s disability compensation claims process starts when a veteran submits a claim to VA (see fig. 1). A claims processor then reviews the claim and helps the veteran gather the relevant evidence needed to evaluate the claim. Such evidence includes the veteran’s military service records, medical exams, and treatment records from VHA medical facilities and private medical service providers. If necessary to provide support to substantiate a claim, VA will also provide a medical exam for the veteran, either through a provider at a VHA medical facility or through a VBA contractor. According to VBA officials, VBA monitors a VHA facility’s capacity to conduct exams and in instances when the facility may not have capacity to conduct a timely exam, VBA will send an exam request to one of its contractors instead. For exams assigned to a VBA contractor, VBA sends an exam request to the contractor, who then rejects or accepts the exam request. Once the contractor accepts the exam, it assigns a contracted examiner to conduct the exam and complete an exam report designed to capture essential medical information for purposes of determining entitlement to disability benefits. The contractors send the completed report to VBA, which uses the information as part of the evidence to evaluate the claim and determine whether the veteran is eligible for benefits. According to contractor officials, if they need clarification on an exam request, they might reject the request and send it back to VBA who, in turn, will revise the request before sending it back to the contractor. Use of Contracts to Complete Disability Compensation Exams VA has used contracted examiners—through VBA and VHA contracts—to supplement VHA-provided exams for at least two decades. VBA began using contractors to conduct disability compensation exams at 10 VBA regional offices in the late 1990s through a pilot program authorized under federal law. In 2014, federal law authorized VBA to expand the pilot to all its regional offices starting in fiscal year 2017. Before fiscal year 2017, VHA and VBA both administered disability exam contracts. However, since fiscal year 2017, all such contracts have been administered by VBA and none have been administered by VHA. VBA awarded 12 contracts to five contractors to begin providing exams in 2016. According to VA officials, performance under 10 of these contracts was delayed until late September 2017 due, in part, to multiple contract bid protests. During this delay, VA officials told us that the agency awarded short-term contracts to allow existing contractors to perform exams until the bid protests were resolved. VBA’s current contracts cover exams for veterans in five U.S. geographic districts, one district for overseas exams, and one district for servicemembers participating in special programs, such as the Benefits Delivery at Discharge and Integrated Disability Evaluation System programs (see fig. 2). VBA awarded two contracts in each of its five U.S. geographic districts and one contract each in districts 6 and 7, which include special programs and overseas exams, respectively. VBA also awarded two additional short- term contracts in December 2017 to help address workload issues in districts 1-5. With the addition of these two contracts, VBA has a total of 14 contracts currently in place. According to agency officials, because VBA wanted to update performance measures for its contractors, VA issued a Request for Proposals in May 2018 with plans to award new contracts in fall 2018 for its U.S. geographic districts. Until it awards the new contracts, VBA will continue to use the current contracts. According to VBA officials, VA plans to continue using VBA contractors in the long term to conduct exams that exceed VHA’s capacity. In recent years, VBA contractors have completed an increasing number of exams, from roughly 178,000 in fiscal year 2012 to almost 600,000 in fiscal year 2017, according to VBA- provided data. VA estimates that in fiscal year 2019, contractors will complete over 1.8 million exam reports for almost 800,000 veterans. However, VBA officials noted that future projections for contracted exams might change based on the need to supplement VHA capacity to ensure timely exams. VBA Contract Exam Office and Requirements for Contractors In 2016, VBA established an exam program office to manage and oversee contractors, monitor their performance, and ensure that they meet contract requirements. For example, the contracts require that contractors develop plans outlining how they will ensure examiners are adequately trained. Contractors are also required to provide VBA with monthly exam status reports, which include the number of canceled, rescheduled, and completed exams, among other things. VBA also has an office dedicated to completing quality reviews of contractors’ exam reports, which are used to assess contractor performance. The contracts require that VBA conduct quality reviews of a sample of contractors’ exam reports. According to VA documents and officials, the results of these quality reviews, and contractor timeliness scores in completing exams, are included in quarterly performance reports. The contracts require that VBA provide these performance reports to the contractors. VBA holds quarterly meetings with the contractors to discuss their quarterly performance based on these reports. VBA Licensing and Training Requirements for Contracted Examiners VBA contracts require that contracted examiners have full, current, valid, and unrestricted licenses, and current and valid State Medical Board certifications, before conducting any exams—the same requirements that apply to VHA medical providers. According to agency officials, VBA also requires that contracted examiners complete the same training that VHA providers must take before they can conduct any disability medical exams. The required training consists of a set of online courses developed by VHA’s Disability Medical Assessment Office, such as courses on VA’s disability claims process and one on completing exam reports. In addition, examiners who provide some specialized exams, such as posttraumatic stress disorder exams and traumatic brain injury exams, are required to take additional courses. In addition to VHA- developed training, VBA contracts require that contractors provide examiners with a basic overview of VA programs. VBA Quarterly Contractor Performance Targets for Quality and Timeliness The contracts also outline quality and timeliness performance targets that VBA uses to assess contractor performance (see table 1). VBA can use contractors’ performance in meeting these targets to determine financial incentives. VBA’s performance measures are as follows: Contractor quality: VBA calculates quality scores for each contractor based on a sample of exam reports that VBA’s quality office selects for review on a quarterly basis for each contract. According to VBA documents, the quality score represents the percentage of exam reports reviewed that had no errors as measured against specific criteria. Errors identified in quality reviews could range from incomplete information (e.g., an examiner’s medical specialty information is not listed on exam report) to completing the wrong exam report for a given condition. Contractor timeliness: VBA calculates timeliness scores for each contractor based on the average timeliness of all exams completed in a given quarter for each contract. VBA measures timeliness as the number of calendar days between the date the contractor accepts an exam request and the date the contractor initially sends the completed exam report to VBA. VBA Reported Contractors Missed Exam Quality Targets, and VBA Could Not Accurately Measure Performance On Timeliness Targets Contractors Missed Quality Targets in First Half of 2017; More Recent Data Are Not Yet Available for Most Districts VBA reported that almost all contractors missed VBA’s quality target of 92 percent in the first half of calendar year 2017, and more recent data are not yet available for most districts. More specifically, VBA-determined quarterly quality scores—the percentage of disability compensation exam reports with no errors as measured against VBA criteria—for the seven contracts used by VBA in calendar year 2017 showed that contractors were frequently well below the quality target. Quarterly quality scores ranged from 62 percent to 92 percent (see fig. 3). According to VBA data, only one contractor’s quality score in one quarter met VBA’s target of 92 percent while the vast majority of contractors’ scores were classified by VBA as “unsatisfactory” performance. VBA has not yet completed all of the quality reviews used to calculate contractor quality scores, particularly for exams that were completed in the second half of 2017. VBA is hiring and training additional quality review staff to complete these reviews and help manage the workload moving forward. According to VBA officials, staff will complete the remaining quality reviews and finalize the quality scores for 2017 by December 2018. VBA Could Not Accurately Measure Contractor Timeliness Against Targets, but Our Aggregate Analysis Shows About Half of Exams Were Completed Within 20 Days According to agency officials, VBA has not calculated contractor timeliness as it is outlined in the contracts. VBA measures timeliness as the number of days between the date the contractor accepts an exam request and the date the contractor initially sends the completed exam report to VBA. According to officials, this measure does not include any time contractors may spend correcting an exam report returned to them by VBA. Returned exam reports are few in number, VBA officials said. However, once a contractor submitted a corrected or clarified exam report, VBA officials said the exam management system did not preserve the date the exam was initially completed. At that point, the system only tracked the date VBA received the corrected or clarified report. As a result, the number of days in VBA’s system could include time contractors took to correct any issues identified by VBA after submitting the initial report. While VBA’s data does not allow it to reliably assess contractor performance against the targets in the contracts, VBA’s data can be used to measure timeliness in other ways. For example, we were able to use the data to calculate the entire amount of time it took to complete exams, which includes time contractors took to correct any issues identified by VBA. As such, the results of our analysis should not be interpreted as reflecting contractor compliance with timeliness targets under the contracts. However, to provide timeframes that are similar to VBA’s targets, we chose 20 days for districts 1-5 and 30 days for districts 6-7 as timeframes for our analysis. Moreover, we analyzed timeliness across all contractors rather than for individual contractors. In particular, we analyzed VBA data on 646,005 contracted exams completed from February 2017 to January 2018, which included 575,739 exams in districts 1-5 and 70,266 exams in districts 6-7. Our analysis of VBA data shows that 53 percent of exams were completed within 20 days for districts 1-5, and 56 percent were completed within 30 days for districts 6-7. However, some exams took at least twice as long to complete. For example, 12 percent of exams in districts 1-5 took more than 40 days to complete (see fig. 4). Contractor officials described a number of reasons why exams might take longer in some cases. For example, they said that scheduling delays might occur due to a veteran’s availability or severe weather, and that it can be challenging to find specialists for certain exam types in rural locations. Our analysis of timeliness focused on exams that were completed, and it did not include exams that have been requested and not yet completed by a contractor. For example, a contractor may have accepted an exam request from VBA, but not yet scheduled an appointment with the veteran. Alternatively, a contractor may have conducted an exam with the veteran, but not yet sent the exam report to VBA. As of late June 2018, VBA-calculated data showed that 87,768 requested exams had not yet been completed, including 37,077 exams that had already exceeded VBA’s timeliness targets. Tracking these exams is important because a large volume of such exams could ultimately increase the amount of time veterans have to wait for their claims to be processed. VBA officials stated that the agency closely monitors contractors’ workloads and helps expedite requested exams that have exceeded VBA’s targets for completing exams. In addition, VBA included a performance measure in its May 2018 Request for Proposals to track the percentage of requested exams that have been with a contractor for more than seven days. Such a measure could help VBA identify whether contractors have a backlog of exams and better assess whether veterans are receiving timely exams. Delayed Quality Reviews and Performance Reports, and Data Limitations, Hinder VBA’s Monitoring of Contractors VBA Identified Some Contractor Performance Problems but Was Delayed in Completing Quality Reviews and Performance Reports VBA Identification of Performance Problems VBA’s contract exam program office, primarily through its Contracting Officer’s Representatives (COR), has identified some contractor performance problems, such as delays in completing specific exams, through its oversight of contractor performance. This oversight includes day-to-day monitoring of contractor workloads and frequent contact with contractor officials. Through such contact and reviews of contractors’ daily and weekly exam status updates, the CORs work with contractor officials to identify ways to expedite disability compensation exams for veterans who have been waiting longer than VBA’s 20-day or 30-day targets. In addition, VBA contract quality staff who review samples of contractor exam reports hold teleconferences with the CORs and contractor officials to provide feedback and discuss issues arising from their reviews, such as specific types of errors. The VBA contract exam program office also oversees and manages contractors through supplemental guidance memos, contractor site visits, and reviews of veteran customer satisfaction surveys. For example, in November 2017, VBA sent a supplemental guidance memo to all contractors to clarify guidance on conducting and documenting hearing loss exams. Further, VBA has conducted site visits to all five contractors’ headquarters or clinic sites since September 2017. Headquarters visits include reviews of contractors’ procedures, such as those for assigning exam requests, and contractors’ information systems, such as those for tracking the status of exams. VBA visits to contractor clinics focus on facility issues, such as accessibility and safety. According to VBA officials, the CORs also review reports on satisfaction surveys completed by veterans after their exam appointments to identify veterans’ concerns regarding contractors and to follow up with contractors, when needed. For example, in response to one veteran’s survey comment regarding a contracted examiner who did not show up to conduct a scheduled exam, VBA officials told us they followed up with the contractor and learned that the examiner’s car broke down. According to VBA, it reimbursed the veteran for round-trip transportation costs to the clinic. Additionally, VBA’s contract quality review staff have conducted special focused reviews to investigate concerns raised by veterans and by staff in VBA regional offices and VHA medical facilities. For example, VBA conducted a review of one contracted examiner who had high rates of diagnosing severe posttraumatic stress disorder. After reviewing this examiner’s reports, VBA found their overall quality to be poor. As a result, VBA requested that the contractor no longer use this examiner. In addition to identifying and addressing problems with individual exams and examiners, VBA has identified broader challenges faced by contractors in meeting VBA’s demand for exams and providing timely reports. For example, VBA identified two contractors who were not prepared to perform all of their assigned exams because they did not have enough examiners, particularly in rural locations, which led to delays and a backlog of exam requests, according to VBA officials. VBA officials described how they worked with these contractors over several months to adjust and closely monitor the volume of exams sent to the contractors to address the backlog. However, according to VBA officials, by December 2017, VBA determined that one of the contractors was not able to meet the demand for exams, and the agency stopped sending new exam requests to this contractor. According to VBA, by late June 2018, it had discontinued all work with this contractor. VA officials said that to obtain additional exam capacity to make up for the two contractors’ shortages, they awarded short-term contracts in December 2017 to two other contractors who were providing exams in other VBA districts. VBA Delays in Assessing Quality and Completing Reports VBA has not completed all required quarterly quality reviews and accompanying quarterly performance reports on contractors, according to VBA officials. These reviews and reports are key components to effectively assessing contractor performance in a timely manner. Specifically, in late June 2018, VBA officials said that they had conducted almost all their quality reviews for contracted exams completed in districts 1-5 during the second half of 2017, but that they needed to finalize the quality scores. They also said that they were beginning their quality reviews for contracted exams completed in 2018. At the time of our review, VBA had released one quarterly performance report for the fourth quarter of calendar year 2017, and officials said they were drafting others. VBA officials attributed delays in completing quality reviews and quarterly performance reports primarily to a lack of VBA quality review staff. The quarterly performance reports provide contractors with information on their performance against VBA quality and timeliness targets. For example, prior reports included detailed breakouts of quality errors by type and suggestions for performance improvements. As officials of one contractor said, delays in receiving quarterly performance reports limit VBA’s ability to provide contractors with timely and valuable feedback they can use to improve the quality of their exams. The delay in completing the quarterly reviews and reports also has implications for VBA’s ability to allocate exam requests across contractors and administer potential financial incentives across contractors. More specifically, VBA can use performance data to help determine how to allocate exams in each district that has two contractors, as outlined in the contracts. For example, VBA can decide to allocate more exams to the contractor with higher performance results. Further, the contracts outline how VBA can use performance data to administer financial incentives linked to performance targets. For example, VA is to provide a bonus to a contractor who meets or exceeds the 92 percent quality standard for a quarter, and meets or exceeds the 20- or 30-day timeliness standard. However, because of its delays in completing quality reviews and the lack of reliable data on contractor timeliness, VA has not yet administered these incentives. VA officials told us that the agency will determine if it will administer the 2017 incentives after it completes its performance assessments of contractors. VBA officials said they are currently hiring more staff to address the lag in quality reviews and subsequent reports to contractors, as well as to provide more oversight of contractors. At the time of our review, VBA did not have its authorized level of 15 quality analysts and 2 senior quality reviewers, but VBA officials said that they expected to complete hiring to bring the quality reviewer staff up to 17 full-time positions by the end of fiscal year 2018. In addition, VBA officials acknowledged that they did not have enough CORs in VBA’s exam program office to oversee the 14 exam contracts (including the two short-term contracts). As of April 2018, VBA officials said the office had 3 CORs, but hiring was expected to bring the number up to 14 by the end of fiscal year 2018. VBA officials said that they determined staffing levels for VBA’s contract exam program office—including CORs and exam quality reviewers—based on an assessment of the resources needed to expand the program, among other factors. Although VBA did not provide documentation outlining how it determined its workforce needs, the agency provided us with updated organizational charts in June 2018 demonstrating increased staff levels for the exam program office. VBA’s Data Limitations Hinder Its Ability to Oversee Certain Contract Provisions, and VBA Has Not Conducted Comprehensive Performance Analysis VBA’s lack of reliable data on the status of exams, including insufficient exams—exam reports that VBA returns to contractors to be corrected or clarified—limits its ability to effectively oversee certain contract provisions. VBA officials acknowledged that they could not calculate the number of completed exams that were once marked as insufficient or how long they had remained in that status due to the data limitations of the exam management system the agency used until spring 2018. The contracts require that contractors correct insufficient exams within a certain number of days and bill VBA for these exams at half price. However, VBA’s lack of complete and reliable information on insufficient exams hinders its ability to ensure that either of these requirements is met. VBA officials also indicated that they were unable to fully assess individual contractor timeliness against VBA’s performance targets because the exam management system did not include the date the initial exam report was submitted to VBA, which is needed to calculate timeliness as outlined in the contracts. In March 2018, VBA began implementing a new exam management system designed to collect more comprehensive and accurate information on the status of exams. VA documentation on the new system shows that it will include detailed data on insufficient exams, which, according to VBA officials, should allow VBA to track whether contractors are properly discounting their invoices for those exams. However, in June 2018, VBA stated that three of its five contractors did not have complete functionality with VBA’s new exam management system. As a result, VBA officials said the agency still did not have complete data in the new system that would allow it to track insufficient exams. Officials said they were working to address these issues. More broadly, as described in VA system documents, the new system is designed to allow VBA to track more detailed data on exam completion dates and on other points throughout the exam process, such as dates for initial requests for clarification from contractors, and dates when appointments are scheduled. However, VBA is in the early stage of this transition, and agency officials stated that unexpected technical issues have affected communication between the new exam management system and other VBA systems. While they work to resolve the issues, VBA officials said that they have been manually moving some exam requests through the system each day. Further, VBA has not documented how it plans to ensure the additional data is accurate and use it to oversee contractor performance as outlined in the contracts, particularly for insufficient exams. Federal internal control standards state that management should use quality information to achieve key objectives. In addition, management should formulate plans to achieve those objectives. For example, agencies should assess collected data and ensure it is accurate so that it can be used to provide quality information to evaluate performance. In the absence of a plan for how it will capture and use data in its new exam management system to assess performance, VBA risks overpaying contractors for insufficient exams and continuing to inaccurately measure contractor timeliness. Further, according to agency officials, VBA has not conducted comprehensive analyses of performance data that would allow it to identify and address higher-level trends and program-wide challenges across contractors, geographic districts, exam types, or other relevant factors. Agency officials told us they have no plans to conduct such analyses. Federal internal control standards state that management should establish and operate monitoring activities and evaluate the results of those activities. In addition, management should evaluate deficiencies both at the individual and aggregate level. While VBA officials acknowledged that higher-level analyses could improve program oversight, they explained that analyzing performance data has been challenging due to the limitations of the exam management system. Thus, VBA has prioritized addressing contractor-specific problems and resolving long-standing pending exams over in-depth analysis of the performance data. However, with the expected improvements provided by VBA’s new exam management system and increased staff to manage the program and conduct quality reviews, VBA should be better positioned to conduct analyses of performance data in the future. By conducting higher-level analyses across contractors, geographic districts, exam types, or other relevant factors, VBA could make a more informed assessment of the challenges contractors and examiners face and where additional workload capacity and training may be needed. In addition, better analyses would allow VBA to determine if the contract exam program is achieving its quality and timeliness goals in a cost effective manner. Auditor Verifies Contracted Examiner Licenses, but VBA Does Not Verify Training Completion or Collect Information on Training Effectiveness VBA Uses a Third-Party Auditor to Verify Contracted Examiner Licenses VBA has a third-party auditor who verifies that all active contracted examiners have a current, valid, and unrestricted medical license in the state where they examined a veteran. The auditor provides regular reports of its audits to VBA. Specifically, the auditor verifies the license numbers of all active contracted examiners in the states where they perform VA disability compensation exams; National Provider Identifiers; and any prior or current sanctions or restrictions resulting in a revoked or suspended license at the time of a VA exam. In addition, contractors send VBA monthly reports of examiners’ medical license, specialty, and accreditation based on the contractors’ verification of this information. Every 2 months, VBA sends the auditor a consolidated report of this information covering all five contractors. The auditor verifies examiners’ information in that report before sending a final audit report to VBA, noting if the auditor was or was not able to verify examiners’ licenses. After reviewing the report, VBA contacts the contractors to gather additional information to resolve any issues, and in cases in which licensing requirements are not met, VBA stops using the examiner and offers new exams to veterans who have been seen by the examiner. VBA and auditing firm officials noted that audit results show that almost all examiners have current and valid licenses, and contractors are required to stop using those who do not meet licensing requirements. VBA and auditing firm officials said that issues identified in the audits are usually due to typos or differences in how information is captured across different licensing databases. However, based on an audit, VBA provided an example of an examiner with a restricted medical license who had completed exams for one contractor. In this case, VBA notified the contractor, who then stopped using the examiner and said it was taking action to prevent errors in its license verification process from occurring again. In addition, the contractor reimbursed VBA for the cost of exams conducted by the examiner and also offered new exams to veterans who had been seen by the examiner. VBA Relies on Exam Contractors to Verify Training is Completed and Does Not Review Training Records for Accuracy VBA relies on contractors to verify that their examiners complete required training, and agency and contractor officials told us that VBA does not review contractors’ self-reported training reports for accuracy or request supporting documentation, such as training certificates, from contractors. As required by the contracts, contractors must track and maintain records demonstrating each examiner has completed required training. Each of VBA’s five contractors has its own process for ensuring that required training is provided to and completed by their examiners, but generally, contractors export the courses from VA’s online training system into their own online training systems for their examiners to access. The contractors, rather than VBA, access the contractor training systems to verify that examiners have completed the required training before they are approved to conduct exams. When requested by VBA, contractors are required to send VBA reports demonstrating that their examiners have met training requirements. As stated in the latest version of the contracts, contractors must immediately stop using any examiner found to have not completed required training, notify VBA, and re-examine the involved veterans at no cost to VBA, if requested by the agency. Although VBA currently does not verify the accuracy of training self- reported by contractors to the agency, VBA officials said that they plan to enhance monitoring through spot checks of training records and a new training system. Specifically, in fiscal year 2019, VBA officials said they plan to start conducting spot checks of some examiners’ training records for accuracy and compliance during site visits to contractor headquarters and clinics. However, VBA has not provided details or documentation on these planned checks, such as how it will determine which records to review or the steps it will take to verify the accuracy of training records. VBA officials also said they are planning to develop an online system that would allow VBA to certify that examiners have completed required training, rather than relying on contractors for this information. However, as of July 2018, VBA had yet to determine when this system would be developed and had not documented plans to do so in order to use such information for monitoring training. VBA also said it would hire staff to manage contractor training, but has yet to do so. GAO’s prior work has emphasized tracking and other control mechanisms to ensure that all employees receive appropriate training. While VBA said it would enhance its monitoring of training records, documenting and implementing a plan and processes to verify training could help ensure examiners have met training requirements. Without such a plan, VBA risks using contracted examiners who are unaware of the agency’s process for conducting exams and reporting the results, which could lead to delays for veterans as a result of poor-quality exams that need to be redone and insufficient exam reports that need to be corrected. VBA Does Not Collect Information to Determine if Training Effectively Prepares Examiners VBA does not collect information from contractors or examiners to help determine if required training effectively prepares examiners to conduct high quality exams and complete exam reports. VBA has provided additional guidance to contractors for some specialty exams. However, VBA identified these issues after some contractors requested guidance in monthly meetings, rather than through VBA efforts to proactively or regularly collect information from contractors or examiners to inform potential changes to training. VBA is considering including a component in the online training system that would collect information on the effectiveness of required training. However, VBA has not outlined additional details on collecting such information. VBA officials said that VBA did not collect such information in the past, in part, because staff were focused on program oversight. To assess progress toward achieving results and to make changes to training if needed, GAO has found that evaluation is a key component of any training program. Given that VBA officials told us that the agency plans to issue new contracts in fall 2018, the number of contracted examiners who are new to VA processes may increase. Thus, collecting and assessing regular feedback on training from contractors and examiners, such as through surveys, discussion groups, or interviews, could help VBA determine if training effectively prepares examiners to conduct exams and complete exam reports. Further, information on the effectiveness of training could supplement data on contractor performance and results from VBA’s quality reviews to help assess if additional training courses are needed across contractors or for specific exam types. Conclusions As VBA increasingly relies on contractors to perform veterans’ disability compensation exams, it is important that the agency ensures proper oversight of these contractors. VBA’s lack of accurate and up-to-date data and reports on contractor performance hampers its ability to oversee the quality and timeliness of exams provided through contractors. VBA’s new exam management system provides opportunities to improve oversight through more comprehensive and accurate data. These improvements might be limited, however, without a plan to use the data to produce the quality information needed by VBA to monitor insufficient exams, ensure it pays contractors the correct amount for those exams, and help it accurately calculate contractor timeliness. Further, the new system provides an opportunity for VBA to conduct analyses that could identify high-level trends and challenges facing the program across contractors and districts, such as delays in completing exams in specific parts of the country or contractor performance issues related to specific exam types. Despite these capabilities, VBA has not outlined plans for using improved information in this manner. Without doing so, the agency may miss opportunities to improve the program and, ultimately, its service to veterans. VBA could better prepare contracted examiners for their role by taking actions to ensure required training has been completed and by collecting information to assess and improve training. Such actions could help improve the quality of exams and exam reports, which could mitigate the need for exam rework and, ultimately, delays in determining veterans’ benefits. With VBA planning to award new contracts and potentially more new contracted examiners coming on board, verifying that required training is completed and collecting information on the effectiveness of training are critical. As VA continues to rely on contracted examiners, it is important that the agency is well positioned to carry out effective oversight of contractors to help ensure that veterans receive high-quality and timely exams. Recommendations for Executive Action We are making the following four recommendations regarding contracted disability compensation exams to VA. The Under Secretary for Benefits should develop and implement a plan for how VBA will use data from the new exam management system to oversee contractors, including how it will capture accurate data on the status of exams and use it to (1) assess contractor timeliness, (2) monitor time spent correcting inadequate and insufficient exams, and (3) verify proper exam invoicing. (Recommendation 1) The Under Secretary for Benefits should regularly monitor and assess aggregate performance data and trends over time to identify higher-level trends and program-wide challenges. (Recommendation 2) The Under Secretary for Benefits should document and implement a plan and processes to verify that contracted examiners have completed required training. (Recommendation 3) The Under Secretary for Benefits should collect information from contractors or examiners on training and use this information to assess training and make improvements as needed. (Recommendation 4) Agency Comments and Our Evaluation We provided a draft of our report to the Department of Veterans Affairs (VA) for its review and comment. VA provided written comments, which are reproduced in appendix II. VA concurred with all our recommendations and described the Veterans Benefits Administration’s (VBA) plans for taking action to address them. Regarding our first recommendation, VA outlined improvements in the information collected through VBA’s new exam management system, and said that VBA is currently testing a mechanism to validate exam invoices submitted by contractors. We noted these improvements to the system in our draft report sent to the agency for comment. We maintain that it will be important for VBA to take the next step of developing and implementing a plan for how it will use information from the new system to ensure both accurate timeliness data and proper exam invoicing. Regarding our second recommendation, VA stated that VBA will use improved data in the new exam management system to regularly monitor and assess aggregate performance data, identify error trends, and monitor contractor performance and program-wide challenges. Regarding our third and fourth recommendations, VA stated that VBA plans to develop and implement a training plan for contractors that will include a mechanism to validate that required training has been completed and to assess the effectiveness of this training through feedback from trainees, contractors, and quality review staff in VBA’s contract exam program office. VA stated that VBA will use this data to improve the implementation and content of training. VA requested that GAO combine these two recommendations into one. However, we believe they are two distinct recommendations and have kept them as such. VBA could meet the intent of each recommendation with the development and implementation of one plan that covers both training verification and assessment, as outlined in its comments. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Veterans Affairs, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions, please contact me at (202) 512- 7215 or curdae@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made significant contributions to this report are listed in Appendix III. Appendix I: Additional Information on Selected Methodologies Review of VBA Contracts To evaluate VBA monitoring of contractor performance and VBA oversight of contracted examiners’ qualifications and training, we reviewed relevant federal laws, regulations, and VA guidance on the use of contracted examiners for disability compensation exams. To identify relevant contract provisions and requirements related to contractor performance, monitoring of such performance, licensing, and training, among other areas, we reviewed selected provisions of selected versions of the 12 current VA Medical Disability Examination contracts originally awarded in 2016, of 5 short-term contracts VA awarded in early 2017, and of 2 short-term contracts VA awarded in December 2017. With regard to the 12 current contracts, we reviewed the selected provisions in the originally awarded contract from 2016 and in the most recently amended version of the contract (as provided to us by VBA officials). Based on our review of these two versions of the contract, the selected provisions appeared to remain in place, unless noted otherwise in this report. However, we did not review the various contract modifications that, according to VBA, occurred in the interim period to confirm whether the selected provisions we focused on in our review actually remained in place during the period between the original contract and the most recent amendment. With regard to the 2 short-term contracts awarded in December 2017, we reviewed the selected provisions in the original December contract. According to VBA officials, there have been no subsequent modifications to these short-term contracts. With regard to the 5 short-term contracts awarded in early 2017, we only reviewed selected provisions relating to contractor quality and timeliness performance. Thus, any statements in this report relating to other aspects of the contracts are not based on these short-term contracts. Further, we only reviewed such provisions in the originally awarded short-term contract, and we did not review the various contract modifications that, according to VBA, occurred subsequently, to confirm that those provisions remained in place over time. However, we found that those selected provisions were generally in place in all of the various contracts we reviewed. Analysis of VBA Data on the Timeliness of Contracted Exams To answer what is known about the timeliness of VBA contracted exams, we analyzed VBA data on disability compensation exams completed by five contractors between February 2017 and January 2018. VBA’s Office of Performance Analysis and Integrity provided exam-level data that it maintains in the agency’s Enterprise Data Warehouse, including data on the exam request date, the date the contractor accepted the request, the date the contractor completed the exam, and the VBA district where the exam was conducted, among other information. These data were created from data originally collected in VBA’s Centralized Administrative Accounting Transaction System (CAATS), which is the system that VBA used to request exams from contractors until spring 2018. According to VBA officials, the status of exam requests (e.g., pending, completed, cancelled) was not always accurate in CAATS. To create more reliable data and identify the most current information on the status of exams, the Office of Performance Analysis and Integrity identified and replaced missing or incorrect data in CAATS by running checks against other VBA systems, including the Veterans Benefits Management System, which maintains veterans’ benefits claims records. We assessed the reliability of the data we received from VBA by conducting electronic testing for missing data and errors, and by interviewing VBA officials about their data collection and quality control procedures. We determined that the data were sufficiently reliable for our purposes of reporting the time it took to complete exams within districts. Our analysis included 646,005 contracted exams completed between February 2017 and January 2018. We selected February 2017 as our starting point because it was the first full month of data available that covered most of VBA’s current contractors. To allow for 12 full months of data, we selected January 2018 as our ending point. In addition, we limited our population to include exams that were requested on or after January 13, 2017 in districts 1-5 or on or after April 1, 2016 in districts 6- 7, based on the periods of performance in the contracts for those districts. We calculated timeliness at the level of the exam request. We calculated the number of days between the date an exam request was accepted by the contractor and the date the exam report was completed by the contractor. The timeliness values we calculated may include additional time needed to request and receive contractors’ corrections or clarifications on previously submitted exam reports. In our report, we refer to these exams as “insufficient exams.” VBA officials acknowledged that due to data limitations the new exam management system is intended to resolve, VBA’s CAATS system did not retain data on the number of exams that were once marked as insufficient or how long they remained in that status. While VBA officials acknowledged that this data limitation affects the agency’s ability to assess individual contractor timeliness on VBA’s performance targets outlined in the contracts, the limitation did not prevent us from analyzing the timeliness of contracted exams overall. The overall timeliness values we calculated represent the total time taken to complete exams regardless of whether additional time was needed for corrections. To put the timeliness values we calculated in context, we calculated the percentage of exams that were completed within VBA’s timeliness targets of 20 days for districts 1-5 and 30 days for districts 6-7 for the entire 12- month period of our analysis. We also calculated the percentage of exams that were completed within other timeframes (e.g., 21-40 days, more than 40 days). According to the contracts, contractors are not expected to complete all exams within the timeliness target, but rather should meet the timeliness target on average in a given quarter, so our analysis was different from one that VBA might conduct in order to determine contract compliance. Because VBA does not retain detailed data on exam completion dates necessary to assess contractor performance against VBA’s timeliness targets, and because we calculated timeliness across contractors, the percentages we calculated do not represent an assessment of whether contractors met VBA’s timeliness targets. GAO did not conduct a legal analysis of the various contractors’ compliance with the contract requirements. Alternate Timeliness Values Given that the start of VBA’s timeliness measure is the date the contractor accepts the exam request (rather than the date VBA requests the exam), we calculated alternate timeliness values to account for potential delays in accepting exam requests. VBA officials stated that VBA requests contractors accept or reject exam requests within 3 days. For all exam requests that contractors took more than 3 days to accept, we calculated alternate totals that included the additional days. For example, if a contractor took 5 days to accept the exam request and completed the exam 20 days later, we calculated an alternate total of 22 days to complete the exam. We used these alternate values to calculate adjusted percentages for each category presented in Figure 4 of our report. For example, using the alternate timeliness values, about 50 percent of exams in districts 1-5 would have been completed in 20 days and 53 percent in districts 6-7 would have been completed within 30 days, rather than the respective 53 percent and 56 percent shown in Figure 4. Moreover, we found that about 82 percent of exam requests during our period of analysis were accepted within 3 days. Pending Exams To report more recent data on exams that were accepted but not yet completed by contractors—pending contracted exams—VBA provided aggregate data on the number of pending exams as of June 25, 2018. For example, for districts 1-5, it provided data on the number of exams that had been pending for 20 days or fewer, 21-40 days, 41-60 days, 61- 100 days, and more than 100 days. We calculated percentages based on the VBA-provided totals. Appendix II: Comments from the Department of Veterans Affairs Appendix III: GAO Contact and Staff Acknowledgments GAO Contact Elizabeth Curda, (202) 512-7215 or curdae@gao.gov. Staff Acknowledgments In addition to the contact named above, Nyree Ryder Tee (Assistant Director); Teresa Heger (Analyst-in-Charge); Alex Galuten; Justin Gordinas; and Greg Whitney made key contributions to this report. Also contributing to this report were James Bennett, Matthew T. Crosby, Teague Lyons, Sheila R. McCoy, Jessica Orr, Claudine Pauselli, Samuel Portnow, Monica Savoy, Almeta Spencer, and April Van Cleef.
target: In 2016, VBA awarded 12 contracts to five private firms for up to $6.8 billion lasting up to 5 years to conduct veterans' disability medical exams. Both VBA contracted medical examiners and medical providers from the Veterans Health Administration perform these exams, with a growing number of exams being completed by contractors. Starting in 2017, VBA contracted examiners conducted about half of all exams. GAO was asked to review the performance and oversight of VBA's disability medical exam contractors. This report examines (1) what is known about the quality and timeliness of VBA contracted exams; (2) the extent to which VBA monitors contractors' performance; and (3) how VBA ensures that its contractors provide qualified and well-trained examiners. GAO analyzed the most recent reliable data available on the quality and timeliness of exams (January 2017 to February 2018), reviewed VBA and selected contract documents and relevant federal laws and regulations, and interviewed agency officials, exam contractors, an audit firm that checks examiners' licenses, and selected veterans service organizations. The Veterans Benefits Administration (VBA) has limited information on whether contractors who conduct disability compensation medical exams are meeting the agency's quality and timeliness targets. VBA contracted examiners have completed a growing number of exams in recent years (see figure). VBA uses completed exam reports to help determine if a veteran should receive disability benefits. VBA reported that the vast majority of contractors' quality scores fell well below VBA's target—92 percent of exam reports with no errors—for the first half of 2017. Since then, VBA has not completed all its quality reviews, but has hired more staff to do them. VBA officials acknowledged that VBA also does not have accurate information on contractor timeliness. VBA officials said the exam management system used until spring 2018 did not always retain the initial exam report completion date, which is used to calculate timeliness. In spring 2018, VBA implemented a new system designed to capture this information. VBA monitoring has addressed some problems with contractors, such as reassigning exams from contractors that did not have enough examiners to those that did. However, the issues GAO identified with VBA's quality and timeliness information limit VBA's ability to effectively oversee contractors. For example, VBA officials said they were unable to track the timeliness of exam reports sent back to contractors for corrections, which is needed to determine if VBA should reduce payment to a contractor. The new system implemented in spring 2018 tracks more detailed data on exam timeliness. However, VBA has not documented how it will ensure the data are accurate or how it will use the data to track the timeliness and billing of corrected exam reports. VBA also has no plans to use the new system to analyze performance data to identify trends or other program-wide issues. Without such plans, VBA may miss opportunities to improve contractor oversight and the program overall. A third-party auditor verifies that contracted examiners have valid medical licenses, but VBA does not verify if examiners have completed training nor does it collect information to assess training effectiveness in preparing examiners. While VBA plans to improve monitoring of training, it has not documented plans for tracking or collecting information to assess training. These actions could help ensure that VBA contractors provide veterans with high-quality exams and help VBA determine if additional training is needed.