Veteran-Owned Small Businesses: VA Improved Its Verification Program but Lacks an Effective Operational Plan for Ongoing Efforts
Published: 2016-03-21T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In fiscal year 2014, the VA made contract awards totaling $4.0 billion to veteran-owned small businesses, including $3.6 billion to service-disabled veteran-owned small businesses. VA must verify the ownership, control, and status of firms seeking such preferences. GAO found in January 2013 ( GAO-13-95 ) that VA faced challenges verifying firms on a timely and consistent basis, communicating with veterans, enhancing information technology systems, and developing and implementing long-term strategic plans. GAO assessed (1) VA's progress in establishing a timely and consistent verification program and improving communication with veterans, and (2) the steps VA has taken to identify and address program challenges and longer-term goals. GAO reviewed VA's verification procedures and strategic plan, reviewed a generalizable random sample of 96 verification applications, and interviewed VA officials and representatives from two veterans' organizations selected from prior work and four verification assistance counselors selected to obtain geographic representation.
section_title: What GAO Found
paragraphs: Since GAO's 2013 report, the Department of Veterans Affairs (VA) took significant steps to improve how it verifies and communicates with veteran-owned small businesses, consistent with several of GAO's previous recommendations. VA reported that due to process improvements, it reduced average application processing times by more than 50 percent—from 85 days in 2012 to 41 in 2015. VA reported that it generally met its regulatory goals for application processing, and GAO's review of randomly selected application files generally corroborated this statement. VA refined the program's quality controls and implemented an internal audit process. Veterans' organizations and verification counselors with whom GAO spoke noted improvements in VA's communications and interactions with veterans, although three of the four verification counselors with whom GAO spoke suggested the program's website could be clearer and all four said the agency's letters to veterans could be clearer. In response, VA officials said they have been redesigning the website to make documents easier to locate. Officials also said the regulatory language in the letters was necessary and they encourage veterans to obtain free assistance with their applications from VA-certified counselors.
VA has been undertaking multiple efforts to address continuing verification program challenges (such as an outdated case-management system) and long-term goals (making processes more veteran-friendly). However, the agency has not had a comprehensive operational plan for managing these efforts to completion. GAO previously recommended that VA's Office of Small and Disadvantaged Business Utilization (OSDBU), which oversees the verification program, establish a strategic plan for the program and integrate efforts to replace an outdated case-management system with agency strategic planning. VA developed a strategic plan for 2014–2018 that included longer-term goals for the program, such as making the verification process more veteran-friendly. In August 2015, VA began piloting a restructured process that allows veterans to communicate directly with the individual processing their application. VA plans to fully transition to the new process by September 2016. VA recently hired a new director for the program, which has had four directors since 2011, including two acting directors in 2015. VA also has continued its efforts to replace the outdated case-management system for the program, but has faced delays due to contractor performance and funding issues. As a result, VA officials do not anticipate the replacement system will be in place until early 2017. While VA has developed a high-level operating plan for OSDBU, the plan does not integrate schedules or specify actions and milestone dates for achieving the multiple changes under way or discuss how to integrate the efforts. VA officials told GAO they were working on developing a detailed operating plan but were waiting to evaluate preliminary results of the verification pilot. GAO's work on organizational transformations states that organizations should set implementation goals and develop timelines to show progress. A detailed plan to guide multiple ongoing efforts is critical given repeated delays in VA's efforts to acquire a new case-management system and the lack of continuity in the program's leadership. Once such an operating plan is developed, it also will be important to update it on a timely basis. Otherwise, VA would continue to be at risk for delays in implementing its initiatives and achieving its long-term goals.
section_title: What GAO Recommends
paragraphs: GAO recommends that VA: (1) complete its fiscal year 2016 operating plan and include an integrated schedule addressing key actions for the verification program and milestone dates for achieving them, and (2) establish a process to review and update the operating plan to address changing conditions. VA agreed with these recommendations.

section_title: Letter
paragraphs: Since the passage of the Veterans Benefits, Health Care, and Information  Technology Act of 2006, total contracting awards from the Department of  Veterans Affairs (VA) to veteran-owned small businesses (VOSB) and  service-disabled veteran-owned small businesses (SDVOSB) increased  significantly, from $616 million ($356 million of which were to SDVOSBs)  in fiscal year 2006 to $4.0 billion ($3.6 billion to SDVOSBs) in fiscal year  2014. The act requires VA to give priority to SDVOSBs and VOSBs when  awarding contracts and provides for limited competition contract awards  to achieve contracting goals that VA must establish. Additionally, VA  must verify the ownership, control, and status of firms seeking such  preferences and maintain a database of verified SDVOSBs and VOSBs.  The program has grown from 2,900 verified firms in 2010 to more than  7,400 verified firms in 2015.
Since VA launched its verification program in 2008, we and VA’s Office of  Inspector General (OIG) have reported on weaknesses, such as limited  progress in defining an organizational structure to manage the program,  establishing and monitoring verification operations, and instituting  effective controls to prevent and detect fraud by ineligible firms. At the  same time, some veteran small business owners and their advocates  have complained of a slow and burdensome verification process and  questioned VA’s rationale for some application denials. We made several  recommendations to VA aimed at developing and implementing a plan to  help ensure a more thorough and effective verification program (in 2010)  and refining and implementing a strategic plan that combined outcome- oriented, longer-term goals with performance measures (in 2013). In  September 2014, VA announced a major organizational transformation to  respond to criticism that the agency was not succeeding in providing high- quality service to veterans. To improve service and rebuild confidence in  VA, the agency launched the MyVA initiative to provide a unified veteran  experience across the organization.
You asked that we update our January 2013 report on VA’s verification  program. This report assesses VA’s progress in (1) establishing a timely  and consistent verification program and improving communication with  veterans, and (2) the steps VA has taken to identify and address program  challenges and longer-term goals.
To assess VA’s progress in establishing a timely and consistent  verification program, we reviewed relevant statutes, regulations, and  procedures for the verification program. We reviewed case files for a  random sample of 96 applications (of 1,306 submitted to VA between  June and September 2014 that resulted in approval or denial). We  collected information on processing time, review completeness, and  documentation of key decisions and rationales. We analyzed the results  to make generalizable estimates about applications submitted from June  through September 2014 and to determine if VA consistently followed its  policies and procedures for processing applications. We assessed the  reliability of the data by interviewing VA officials, reviewing  documentation, and checking for illogical values or obvious errors and  found the data sufficiently reliable for estimating population values. We  reviewed administrative program data obtained from VA on application  processing times for fiscal years 2012 through 2015 and compared those  numbers to our findings from the case file review. We also reviewed  administrative data from VA on the number and type of site visits  conducted in fiscal years 2014 and 2015. We assessed the reliability of  CVE’s administrative data by interviewing VA officials and reviewing  documentation and found the data to be sufficiently reliable for describing  reported processing statistics. We also reviewed a program quality  manual and internal audit reports from December 2014 to March 2015.  We reviewed more than 100 allegations of noncompliance with program  regulations sent to VA between June 2014 and May 2015 by a veteran- owned small business advocate. We selected 10 for a more in-depth  review and reviewed VA’s files on verified businesses associated with the  allegations to identify the steps VA took to investigate the allegations. We  interviewed VA officials to discuss changes to the verification program  since our last report in January 2013, quality control procedures, and  steps taken to investigate allegations of noncompliance with program  regulations.
To evaluate VA’s progress in addressing communication challenges with  firms seeking verification, we reviewed VA’s work instructions for  processes that involve communicating with veterans. We analyzed data  from the case file review to determine the extent to which VA complied  with procedures for communicating with applicants and verified  businesses. We reviewed documents relating to VA’s help desk customer  satisfaction surveys and survey instruments approved by the Office of  Management and Budget. We interviewed VA officials about  communication procedures. We also interviewed representatives from  two veterans’ service organizations and four Procurement Technical  Assistance Centers (PTAC)—which provide verification counseling  services to veterans—about VA’s verification process and communication  efforts. We selected the veterans groups based on our prior work in the  area and the PTACs based on recommendations from the Defense  Logistics Agency and the Association for Procurement Technical  Assistance Centers and to obtain geographic diversity.
To assess the steps VA has taken to identify and address verification  program challenges and longer-term goals, we reviewed the Office of  Small and Disadvantaged Business Utilization (OSDBU) Strategic Plan  for 2014–2018 and its Operating Plan for 2016 and compared them with  useful practices and lessons learned on organizational transformations,  as identified in previous GAO work. We reviewed our prior work on the  verification program and that of the VA OIG. We interviewed VA officials  to determine what steps, if any, they took to address issues we and VA’s  OIG identified and identify and address other challenges. We reviewed  VA’s efforts to replace its case-management data system, including  project planning documents, a list of technical requirements, and other  contract documents. We interviewed VA officials about their plans for  developing the new system. Finally, we discussed VA’s plans to refocus  the verification process with VA officials and reviewed associated  planning documents. See appendix I for more information on our  objectives, scope, and methodology.
We conducted this performance audit from December 2014 to March  2016 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Within VA, its OSDBU has overall responsibility for the SDVOSB and  VOSB verification program. OSDBU’s Center for Verification and  Evaluation (CVE) maintains the mandated database of verified SDVOSBs  and VOSBs and is responsible for verification operations, such as  application processing. CVE is led by a director and deputy director, and  staff are organized into seven teams that either assist with a verification  phase or supporting function. A federal employee leads each team and  CVE contracts with several SDVOSBs to provide contractors that conduct  verifications and supporting activities. As of January 2016, CVE had 16  federal employees and 156 contract staff (employed by five different  SDVOSB contractors) verifying applications or filling supporting roles.  CVE and its information technology are funded by VA’s Supply Fund, a  fund that recovers its operating expenses through fees and markups on  different products or services. CVE’s final obligations for fiscal year  2014 were $17.9 million and its approved budget for fiscal year 2015 was  $16.1 million, representing a decrease of about 10 percent ($1.8 million).
VA developed eligibility requirements and a process to verify the  ownership and control of firms seeking contracting preferences as  SDVOSBs and VOSBs and to confirm the status of any owner who  indicates a service-connected disability. To be eligible for verification  under VA’s rules  the small business concern (hereafter, firm) must be unconditionally  owned and controlled by one or more eligible parties (veterans,  service-disabled veterans, or surviving spouses); the owners of the firm must have good character (any small business  owner or concern that has been debarred or suspended is  ineligible);  the applicant cannot knowingly make false statements in the  application process;  the firm and its eligible owners must not have significant financial  obligations owed to the federal government; and  the firm must not have been found ineligible due to a Small Business  Administration protest decision.
VA’s verification process consists of reviewing and analyzing a  standardized set of documents submitted with each verification  application. VA’s current verification process has four phases—initiation,  examination, evaluation, and determination (see fig. 1). Denied applicant  firms can request a reconsideration of the denial decision, but if the denial  is upheld, must wait 6 months before submitting another application.
VA maintains a database on its Vendor Information Pages (VIP) website  of verified SDVOSBs and VOSBs. Once VA verifies a firm, the firm  name appears with a verified logo in VIP. Verification is valid for 2 years,  with the option of renewing for 2 additional years. To apply for renewal,  verified firms must submit an application before expiration of their  verification, answer questions about any key changes to their business  structure, submit supporting business documents such as updated tax  returns and any amended operating documents, and receive a full  evaluation by CVE. Firms that receive a renewal have to undergo a full  verification (reverification) after the 2-year renewal period expires (that is,  firms must go through the full process every 4 years).
In 2014, VA launched the MyVA Reorganization Plan in an effort to  improve the efficiency and effectiveness of VA’s services to veterans. The  plan’s strategy emphasizes improved service delivery, a veteran-centric  culture, and an environment in which veteran perceptions are the  indicator of VA’s success. MyVA extends to all aspects of the agency’s  operations, including the verification program. In response to this  organizational change, OSDBU is required to align its own strategy with  MyVA and take steps to make its operations more customer service- oriented and veteran-centric. OSDBU has established in its strategic plan  that its longer-term goals for the verification program are to transform the  verification process to make it more veteran-friendly and to expand the  program’s capacity to serve more veterans.
section_title: VA Improved Processing Times, Quality Controls, and Communication with Veterans
paragraphs: In our May 2010 and January 2013 reports, we found that VA faced  challenges in establishing a program to verify firms on a timely and  consistent basis. We made a number of recommendations to address  these issues. Since that time, CVE has made significant improvements to  the verification program—consistent with our recommendations—such as  improving application processing times, quality controls, and  communication with veterans.
subsections: section_title: VA Improved Processing Times
paragraphs: Based on CVE’s administrative data, application processing times have  decreased by more than 50 percent since October 2012—from an  average of approximately 85 days—to 42 days in fiscal year 2015 (see  fig. 2). VA officials attributed the decreased processing time to a number  of process improvements, such as moving from paper to electronic  applications, developing detailed written work instructions for staff and  contractors conducting verification activities, and enhancing resources  available to applicants to make them more aware of program  requirements. Officials told us that they have been generally meeting their  regulatory processing goal of 60 days (from receipt of a complete  application) and had 5 applications (out of 3,129) in fiscal year 2014 and  11 applications (out of 4,651) in fiscal year 2015 that did not meet this  goal.
Our review of randomly selected application files corroborated that CVE  generally met its processing goals, but the verification process can take  longer from a veteran’s perspective. In calculating processing times, CVE  excluded any time spent waiting for additional information it asked firms to  supply, so the number of days it took an applicant to become verified  typically was longer than CVE reported. To illustrate how long the process  could take from the veteran’s perspective, we used the results of our case  file review to estimate the average number of days it took veterans to  receive an initial determination—including any time a firm took to prepare  and submit additional requested documentation (time that VA excluded  from its estimated processing times). Based on our analysis, it took  firms that applied for verification from June through September 2014 an  average of 68 days to receive an initial determination on verification  eligibility.
Additionally, firms can submit and withdraw their application multiple  times if they need to correct issues or wish to apply at a later date, an  option that can lengthen the verification process for some firms. Each  time a firm resubmits an application, CVE resets the processing clock.  Based on our case file review, we estimated that for 15 percent of  applications submitted from June 2014 through September 2014, it took  more than 4 months from the initial application date for firms to receive a  determination from CVE. In 5 of the 96 applications we reviewed, the  verification process took more than 6 months to complete. For 2 of the 5  applications, the process took 1 year or more to complete. In each of the  5 cases, the firm had submitted and withdrawn an application at least one  time before submitting an application that received a final determination.  VA officials said that weeks or months could pass between a firm’s  withdrawal of its application and resubmission of a new application.  Additionally, VA officials said that allowing applicants to withdraw and  resubmit multiple applications was advantageous to the veteran (without  the option, more veterans would receive denials and have to wait 6  months before submitting another application).
section_title: VA Improved Quality Controls, and Implemented New Procedures to Document and Investigate Allegations of Noncompliance
paragraphs: CVE has continued to refine its quality management system since our  May 2010 and January 2013 reports, including developing  comprehensive work instructions, conducting site visits, and revising  policies for investigating allegations of noncompliance. In particular, VA  has made progress since our 2013 report. For example, at that time, VA  was introducing significant changes to its procedures and operations, and  we determined that our original focus on evaluating VA’s compliance with  policies and procedures would be of limited value.
Since our 2013 report CVE has put into place detailed written work  instructions—which are used by CVE staff and contractors conducting  verification activities—for each part of the verification process, and a  quality manual that documents the requirements of its quality  management system. According to CVE officials, the work instructions  are updated on a regular basis. CVE officials said the agency received  certification in 2015 that its revised quality management system was  compliant with the International Organization for Standardization  9001:2008 quality management standards. CVE also has implemented  an internal audit process and a continuous improvement process. As of  September 2015, CVE had taken action on and closed 332 of 350 (95  percent) internal audit recommendations made since February 2013.
Based on our case file review, we estimate that VA followed its policies  and procedures for verifying SDVOSBs and VOSBs in 100 percent of  applications that were approved or denied from June through September  2014. This included checking the veteran and disability status of the  applicant, conducting research on the firm from publicly available  information, and reviewing business documents to determine compliance  with eligibility requirements (such as direct majority ownership by the  veteran and experience of the veteran manager).
In our May 2010 report, we found that VA had a large backlog of firms  awaiting site visits, including some high-risk firms. As a result, we  recommended that VA develop and implement a plan to, among other  things, conduct timely site visits to high-risk firms. VA reported in  October 2013 that it had conducted more than 1,000 site visits in fiscal  year 2013 and there was no longer a backlog of firms awaiting site visits,  which was consistent with our recommendation. In fiscal years 2014  and 2015, CVE conducted 1,750 site visits to gather additional  information about firms during its application review, or check the  accuracy of verification teams’ decisions and help ensure that verified  firms continue to comply with program regulations. Specifically, CVE  conducted 1,144 site visits in fiscal year 2014 and 606 site visits in fiscal  year 2015 on verified firms and firms applying for verification. Of the fiscal  year 2015 site visits, CVE used risk-based determinations to select the  vast majority of firms for visits (93.7 percent or 568 firms). The remainder  were randomly selected or chosen because they were in the application  process. CVE officials said that firms identified through risk-based  selection were chosen based on their risk to VA (for instance, if the firm  had a VA contract). Officials reported low error and noncompliance  rates as identified through site visits, described specifically below.
CVE officials said the site visits identified two instances in fiscal year  2015 and nine instances in fiscal year 2014 in which CVE evaluators  mistakenly verified a firm (a less than 1 percent error rate).
CVE issued 25 cancellations to firms found out of compliance with  program regulations (a 4 percent noncompliance rate) in fiscal year  2015 and 57 cancellations in fiscal year 2014 (a 5 percent  noncompliance rate), an outcome that can result from changing  characteristics of the firm after verification.
These statistics, particularly the identification of a small number of  instances in which CVE evaluators mistakenly verified a firm in the past 2  fiscal years, are consistent with the findings from our case file review that  VA has been following its policies and procedures for verifying firms.
CVE officials said they have been working with VA’s Office of Enterprise  Risk Management to determine how many site visits should be conducted  annually and how firms should be selected. VA spent about $3 million in  fiscal year 2014 to conduct 1,144 site visits (about 16 percent of all  verified firms). Officials stated that because of the cost of conducting the  site visits (about $2,600 per visit)—and the low rate of noncompliance  identified by site visit examiners (about 4.7 percent in fiscal year 2014)— the agency has been looking to reduce the number of site visits while  maintaining the effectiveness of the site visit program. CVE also randomly  selected a sample of 104 of 2,312 firms that received VA contracts from  March 2014 through April 2015 for site visits, in order to obtain a  statistical estimate of the noncompliance rate for the program. CVE  officials said they have compiled data from visits performed in fiscal years  2014 through 2016, which they will use to identify risk factors that could  affect compliance (such as a firm’s business type, industry type, and size  of VA contract). VA officials said they plan to use these data to monitor  emerging risk factors and adjust their selection of firms to receive site  visits accordingly.
CVE also monitors program compliance through investigations of  suspicious firms identified through tips from external sources. CVE  officials told us they received about 400 tips in 2014 about noncompliance  with program regulations. CVE is responsible for investigating  noncompliance with program requirements, and the VA OIG is  responsible for investigating fraud. Officials said that they investigate  credible allegations they receive by conducting public research to  substantiate or disprove the tip, reviewing eligibility requirements related  to the tip, and making a recommendation for corrective action, if  necessary. We reviewed case files associated with 10 firms for which  CVE received such allegations (between June 2014 and May 2015).  These allegations were made by a veteran-owned small business  advocate. We found that CVE investigated 6 of the 10 allegations. For the  4 cases that it did not review, CVE officials said the allegations were not  specific enough to conduct an investigation. Additionally, officials said the  allegations were not sent to the e-mail address that VA had established  for this purpose, and thus, may not have been routed to the correct  individuals within VA charged with investigating allegations. According  to a policy memorandum issued in October 2015, CVE previously only  accepted referrals on its non-compliance referral form. However, the  policy memorandum changed CVE’s policy regarding reviews of non- compliance referrals, and CVE began reviewing and responding to all  allegations of noncompliance, whether received on the referral form or  not. Additionally, CVE will notify the sender if the agency has enough  information to investigate the allegation and request additional  information, if necessary.
Based on our review of the files for these 10 allegations, we found that  CVE had not always documented that a noncompliance allegation had  been received or that it was conducting a review of the firm’s eligibility  based on the allegation. CVE officials said they adopted a policy in July  2015 to upload all findings from investigations that result from  noncompliance allegations to the case-management system so that CVE  staff and contractors working on a firm’s verification have access to that  information.
section_title: VA Has Taken Steps to Improve Communication with Veterans, Although Some External Groups Suggested Improvements
paragraphs: VA has taken steps to improve communication with veterans since our  January 2013 report, in which we discussed concerns of some veterans’  organizations about the verification program. VA implemented additional  procedures to improve communication with verified firms about their  verification status. Specifically, according to agency officials, VA sends  e-mail reminders 120, 90, and 30 days before the expiration of a firm’s  verification status; contacts verified firms by telephone 90 days before  expiration of verification status; and notifies firms in writing 30 days before  cancelling a firm’s verification. Additionally, VA communicates with  applicants at several points in the verification process, such as to indicate  that an application is complete, additional documents are needed, and a  determination has been made. VA also e-mails applicants about issues  that would result in a denial to offer them the ability to make changes to  the application or to withdraw it prior to receiving the denial. Our case file  review found VA generally followed its procedures to send reminder  notices to applicants who needed to submit additional documentation,  and to send notices that an application was complete.
In our January 2013 report, we noted that VA had recognized that some  applicants needed additional support and launched a Verification  Counseling program in June 2012 to assist firms interested in becoming  verified. Since that time, VA has continued to work with PTACs across the  United States to provide verification assistance to veterans free of  charge. VA officials said they have trained more than 300 procurement  experts on the verification process so they can assist veterans applying  for verification. In addition, we determined that VA provides other  resources such as fact sheets, verification assistance briefs, and standard  operating procedures for the verification program on its website. VA also  provides a tool on its website that allows firms to obtain a list of  documents required for an application depending on the type of company  they own (such as a limited liability corporation or sole proprietorship).  Moreover, CVE officials said that they have increased interaction with  veterans seeking verification through web-based activities and outreach  at conferences and other events. For instance, since our 2013 report,  CVE began conducting monthly pre-application, reverification, and town  hall webinars to provide information and assistance to verified firms and  others interested in the verification process. VA officials also told us that  they attend veteran small business conferences and other meetings in  which they can conduct outreach for the verification program.
CVE has taken steps to collect veteran feedback about the program. The  agency held two focus groups in July 2015 and began surveying firms in  August 2015. Several veterans who participated in the focus groups  commented on the lack of clarity of VA’s communications. For example,  one veteran said that there was a lack of clarity about the documents that  should be submitted with an application. Two veterans noted issues with  the timing and redundancy of document requests during the application  process. And two veterans said there was lack of clarity around certain  program rules. Although survey results cannot be generalized to all  veterans going through the verification process, the contractor  administering the surveys reported in September 2015 that feedback from  firms that had been through the verification process appeared positive,  particularly with respect to improvements in the verification process.  CVE officials stated they hope the surveys will allow them to more  systematically collect feedback from veterans on different aspects of the  program, including the pre-application experience, the verification process  from submission to determination, and site visit examinations.
All the counselors and representatives of veterans’ service organizations  with whom we spoke noted that VA has improved its verification process,  but most had suggestions for continued improvement. One PTAC  representative noted that VA could better leverage organizations such as  PTACs, veteran small business outreach centers, and state-level  Departments of Veterans Affairs to disseminate information. Another  PTAC representative noted that PTACs and other veterans’ groups could  host events for CVE to interact with veterans to increase awareness of  the verification program and the services PTACs provide. Additionally,  this PTAC representative said CVE could do a better job referring  veterans to PTACs that could assist them with the verification process.  He added that having PTACs assist veterans before they started the  application process would reduce applicant error and frustration and  processing times. VA officials stated that they make PTAC referrals  through the help desk and monthly preverification webinars and  participate in outreach events when possible. Officials also noted that  travel dollars to attend and conduct outreach at external events are  limited and they try to use the monthly webinars to interact with and  inform veterans.
In addition, VA has taken steps to address external stakeholders’  concerns about the information available on VA’s website and clarity of  communications to applicants. Three of the four counselors noted that  resources on VA’s website for the verification program can be difficult to  locate. Representatives from one of two service organizations said VA  does not provide adequate documentation of program standards for  applicants. Officials said that they have been working with OSDBU to  redesign the website to make documents, such as verification assistance  briefs and the tool to identify required documents, easier to locate.  Additionally, the counselors we interviewed noted that VA’s  communications to applicants were at times unclear. VA’s procedures  require that determination letters to applicants include specific reasons for  denial or potential denial. All four of the counselors we interviewed also  stated that VA’s determination letters to applicants could be clearer and  that they include regulatory compliance language that could be difficult for  some applicants to understand.
We used an automated readability test on five determination letters from  our case file review (written from August through December 2014).  According to the test, the reading level required to understand the letters  ranged from a college sophomore to a college senior level, so readability  might present a challenge for some applicants. VA officials maintained  that the inclusion of regulatory language in the determination letters was  necessary, but acknowledged the language could present readability  challenges. Moreover, the officials noted that they encourage veterans to  obtain free assistance with their applications from VA certified verification  counselors. We also observed several instances in our review in which a  letter initially stated that documents were due on one date, and then later  stated the applicant should disregard the initial statement and that  documents were due on a different, earlier date. VA officials said this was  due to a glitch in the system that generated the letters and a software  update issued in May 2015 resolved the issue. Officials also said  document requests are now generated using a new template, instead of  the older template that previously caused issues. VA has efforts under  way (discussed below) to replace the program’s case management  system, which generates the templates for document requests.
section_title: VA Lacks a Comprehensive Operational Plan for Completing Multiple, Ongoing Efforts
subsections: section_title: VA Intends to Restructure Part of the Verification Process
paragraphs: Although VA improved application processing times and communication  with veterans since our 2013 report, VA has recognized that the  verification process can be more cost-effective and veteran-friendly.  OSDBU developed a strategic plan for 2014–2018 that included longer- term goals for the program, such as making the verification process more  veteran-friendly. Additionally, OSDBU officials also told us that in 2015  the Supply Fund Board asked OSDBU to design a veteran-centered  process that highlights customer service and maximizes cost efficiency.  OSDBU and CVE have various efforts under way—such as restructuring  the verification and reverification processes and revising program  regulations—intended to improve veterans’ experience with the program  and provide cost efficiencies. These changes are also intended to better  align the verification program with MyVA—VA’s organization-wide  transformation initiative.
subsections: section_title: Restructuring the Verification Process
paragraphs: In August 2015, VA began to test (VA officials refer to it as a pilot) a  restructured verification process that gives veterans a case manager who  serves as a point of contact throughout the process and allows veterans  to communicate directly with the individual processing their application.  According to OSDBU officials, the new process is expected to provide  cost savings to the agency by reducing the amount of time spent  reviewing applications and addressing veterans’ questions. For  example, VA officials told us that under the current four-phase process,  eligibility issues are not identified and communicated to the veteran until  the later stages of the process, which could be 35–40 days after  application submission. Under the new process, veterans would be  interviewed shortly after submitting their application, which would allow  VA to identify issues up front and avoid multiple reviews of applications  for firms not meeting program requirements. CVE officials also stated that  changes to the process are intended to help improve communication with  applicants. For example, under the current process, applicants may  correspond with several different customer service representatives, each  of whom would have to read through case notes before addressing the  veteran’s question. Under the new process, a case manager, who serves  as the point of contact for the veteran and coordinates staff evaluating the  application, would be familiar with the application status and any issues  that arose throughout the process. Additionally, by interviewing the  applicant at the beginning of the application process, CVE may be able to  reduce some of the problems caused by applicants not understanding the  written communications sent by CVE. Key differences between the pilot  and current processes as described by CVE officials are shown in table 1.  CVE officials stated that as of November 2015, 369 applications had been  reviewed using the new process and that CVE had processed about 15  percent to 20 percent of applications under the new process. CVE  officials said they plan to process about half of all applications using the  new process by April 2016.
CVE officials stated that they plan to fully transition to a new process by  September 2016. The new process will be based on the approach used  during the pilot with some adjustments, as determined by CVE’s  evaluation of the pilot procedures. The officials stated that they developed  a number of metrics to inform adjustments to the pilot, which include  length of application processing times, number of approvals, denials, and  withdrawals, and number of applications processed by case-management  teams each month. Additionally, to help evaluate the verification  procedures used during the pilot, CVE officials stated that VA held one  focus group in October 2015 and since September 2015 has been  surveying firms that participated in the new process to obtain feedback.  CVE has made adjustments to the process in response to metrics and  feedback. For example, CVE decided to have case analysts process the  majority of applications, which allowed assessors more time to process  complex applications and provide guidance to the case analysts and  other assessors, when needed. CVE’s Acting Director and Deputy  Director were responsible for evaluating data obtained from these metrics  according to CVE officials. In addition, CVE officials stated that  information obtained from the metrics was intended to provide them with  a better sense of how many teams will be needed to conduct verification  and what types of skills verification staff and contractors needed for the  work. CVE officials stated that VA has used current staffing resources for  the pilot and transitioned more staff from the current verification process  to the pilot, as the pilot expanded. Officials said that VA has developed a  range of preliminary cost estimates based on different staffing levels used  during the pilot. The agency plans to develop a cost estimate for average  and total processing costs after they have transitioned staff to the new  process. As of March 2016, CVE officials stated they completed their  evaluation of the pilot and have selected a process they determined was  cost effective and efficient and did not compromise quality.
section_title: Reverification Policy
paragraphs: VA also recently revised its reverification process to improve efficiency  and customer service. According to CVE officials, reverification used to  require nearly the same effort of CVE staff, contractors, and veterans as  the full initial verification process. Under a new process CVE implemented  in October 2015, CVE contractors review documentation from the  veteran’s previous application, determine what additional documentation  is needed to reverify the firm, and conduct an initial interview with the  veteran to identify and provide information on what documents need to be  updated for reverification.
CVE officials said these changes are intended to improve veterans’  understanding of the requirements for reverification, enhance the veteran- centric nature of the program, and further reduce application processing  times—by reducing the number of documents the veteran uploads at the  beginning of the process and therefore the time CVE contractors and staff  spend reverifying applications. According to CVE, it is too soon to  determine if these changes have achieved the desired effect, but they  intend to evaluate the new procedures by developing survey questions on  customer satisfaction and reviewing processing times for reverification  applications.
section_title: Revisions to Ownership and Control Requirements
paragraphs: In addition to changes to the verification and reverification processes, VA  has continued to make revisions to its program regulations to streamline  the process and provide clarity for veterans. In our 2013 report, we found  that VA had begun modifying program regulations to extend the  verification period from 1 to 2 years and published an interim final rule to  this effect in late June 2012. More recently, in November 2015 VA  published a proposed rule for clarifying and simplifying eligibility criteria in  the Federal Register, on which it had been working since 2013. VA  obtained input and recommendations from veteran stakeholder groups to  inform the new regulations.
According to officials, the proposed revisions are intended to simplify the  criteria used to determine veteran ownership and control, and account for  common business practices that might otherwise lead to a denial decision  under the current regulation. For example, in addressing the challenges  associated with one current regulatory provision, CVE officials said that  the proposed rule simplifies the ownership criteria by modifying the term  “unconditional” to allow businesses to include rights of first refusal and  “tag-along” rights in their operating documents and still participate in the  program. Similarly, officials said that VA plans to allow minority owners  to vote on extraordinary business decisions such as closing or selling the  business. Officials stated that the revisions to the regulation were not  expected to provide cost and resource efficiencies, or affect the new  verification process being developed through the pilot.
Comments on the proposed rule were due January 5, 2016, and officials  expect to finalize the proposed rule in mid-2016. Officials said they  received about 100 comments and that they still were evaluating and  categorizing the comments as of January 21, 2016. When the rule  changes are finalized, VA plans to train staff on the revised regulation and  update its review sheet used during application review.
section_title: VA Plans to Hire Permanent Leadership for the Program and Realign Organizational Structure
paragraphs: We previously found that leadership and staff vacancies contributed to  the slow pace of implementation of the verification program. In our May  2010 report, we found that leadership in OSDBU was lacking because the  position of Executive Director remained vacant from January 2009 until  January 2010. Furthermore, one of two leadership positions directly  below the Executive Director had been vacant since October 2008, and  the other positions had been filled by an Acting Director. In 2010, we  recommended that VA develop and implement a plan that ensures a  more thorough and effective verification program and addresses actions  and milestone dates for filling vacant positions within OSDBU, including  the leadership positions. By July 2011, VA had filled the vacant  leadership positions but has since experienced turnover in the leadership  positions for the verification program. Specifically, CVE has had four  different directors since 2011, including two acting directors in 2015. In  addition, the position of Deputy Director was vacant from March 2014 to  September 2015. VA posted a job announcement for the CVE director  position in October 2015 that closed in November 2015. OSDBU’s  Executive Director told us in December 2015 that VA planned to complete  the recruiting process and hire a permanent director by January 2016. VA  hired a permanent director in February 2016.
In addition to taking steps to acquire permanent leadership for the  program, VA has made changes to CVE’s organizational structure to align  staffing resources with agency needs and reflect the new pilot verification  process. CVE officials said that under the new organizational structure, a  federal employee heads each of three teams—critical path, risk and  compliance, and verification support—and oversees the contractors who  conduct the majority of the work for the program. The critical path team is  responsible for verifying applications. All federal staff responsible for  overseeing the verification process and the contractors reviewing  applications have been moved to this team, according to CVE officials.  The risk and compliance team oversees CVE’s site visit program,  investigates allegations of noncompliance, and makes referrals to VA’s  Debarment and Suspension Committee and OIG. The verification support  team staffs the help desk and provides support processing  communications to applicants. CVE officials also said that the quality  assurance team has been moved from within CVE to the larger OSDBU  organization and will continue to support CVE internal audits as well as  provide support to OSDBU. OSDBU officials indicated that VA made this  change to share more responsibilities between CVE and OSDBU and  reduce program operating costs.
According to CVE officials, VA has developed position descriptions for the  pilot verification process and used data from the pilot to determine optimal  staffing levels needed to process applications under the new procedures.  As discussed earlier, CVE relies heavily on contractor support to conduct  its verification activities and currently has 16 federal employees and 156  contractors working on the verification program. CVE officials stated that  VA plans to continue using contractor staff to conduct verification  activities because the use of such staff gives VA the flexibility to adjust  staffing levels as needed to respond to changes in the number of  verification applications received. According to VA officials, and generally  consistent with findings from our case file review, federal employees  make the final determination for verification decisions. Officials said that  based on information obtained through the pilot, VA has determined that it  needs a total of 10 federal reviewers on the critical path team; however, it  still was determining its needs for contractor staff as of November 2015.  According to OSDBU officials, VA has contracts in place for the  verification program staff through April 2016 and plans to start the  process for securing new contracts in January 2016.
section_title: VA Continues to Face Delays in Replacing the Program’s Outdated Case- Management System
paragraphs: We reported in January 2013 that VA moved from using a paper-based  verification application to an electronic application when it implemented a  new case-management system in 2011, consistent with our prior  recommendation. However, we identified significant shortcomings in VA’s  data system, including that the system did not collect important data and  had limited reporting and workflow management capabilities. We  recommended that VA integrate its efforts to modify or replace the  program’s data system with a broader strategic planning effort to ensure  that the system addressed the program’s short- and long-term needs. VA  concurred with the recommendation and included a strategic performance  goal to improve information technology capabilities in its strategic plan,  which was consistent with our recommendation. VA awarded a contract  for an enhanced data system; however, VA has since faced delays in  developing the system. VA’s Office of Information Technology hired a  contractor in September 2013 to develop the new system, but VA  cancelled the contract in October 2014 due to poor contractor  performance. VA paid the contractor about $871,000 for work that had  been performed before the contract’s termination, and received several  planning documents from the contractor that helped inform its current  acquisition effort, according to CVE officials.
VA officials told us that CVE and VA’s Office of Information and  Technology then began working with a contractor in September 2014 to  identify data systems in existing federal programs that VA could use to  build its own system. In February 2015, officials told us that they planned  to award a contract for development of the new system in the first quarter  of fiscal year 2016 based on the contractor’s research to identify existing  federal systems. VA was unable to award a contract based on this effort.  Specifically, the contractor identified two programs that had the potential  to be used in developing a new case-management system for VA.  However, VA officials conducted additional research on each program’s  system and found technical or administrative reasons that prevented VA  from partnering with any of the identified programs to use their existing  systems to develop a new case-management system for CVE.
VA’s more recent efforts to develop a new case-management system also  have faced setbacks. In May 2015, VA established an internal working  group consisting of staff from OSDBU and VA’s Office of Information and  Technology and Office of Acquisition and Logistics to plan and manage  the development of the new system, based on the planning documents  provided by the contractor and in accordance with internal guidelines for  managing new information technology projects. In July 2015, VA  officials told us they decided to develop a pilot system through another  existing contract. Officials said they intended to use the pilot system to  provide VA with the opportunity to evaluate the capabilities of a new  system without the time and expense of putting an entire new system in  place. VA developed specifications and other planning documents for  the pilot system, and planned to develop and evaluate the system from  November 2015 through January 2016. If the pilot was successful, VA  had planned to issue a solicitation and award a contract for development  of a full system by April 2016 and fully transition to the new system by  September 2016.
However, in November 2015, OSDBU officials told us that the Supply  Fund board had requested that OSDBU develop and provide a business  case for the new system. VA officials stated that as a result, they have  revised the timeline for developing the new system, and expect to begin  the pilot in January 2016. If the pilot is successful, VA plans to fully  transition to a new system in early 2017. But as a result, VA continues to  use a data system that does not collect important data and has limited  reporting and workflow management capabilities.
section_title: VA Has Not Developed a Plan with Specific Actions and Milestones for Implementing the Multiple Changes to the Verification Program
paragraphs: In both our May 2010 and January 2013 reports, we found that VA faced  challenges in developing and implementing plans for establishing an  effective verification program. Specifically, in our May 2010 report, we  found that VA did not have a plan or specific time frames for  implementing a thorough and effective verification program, including  filling vacant staff positions; improving verification procedures to ensure  greater completeness, accuracy, and consistency of verification reviews;  and conducting timely site visits at high-risk firms. We recommended that  VA develop and implement such a plan. VA took actions that addressed  the specific actions referenced in our recommendation, obviating the  need for a plan to accomplish these actions. In our January 2013 report,  we found that VA faced challenges in its strategic planning efforts and  recommended that VA refine and implement a strategic plan with  outcome-oriented longer-term goals and performance measures.  Subsequently, VA developed a strategic plan for fiscal years 2014–2018  that, consistent with our recommendation, described OSDBU’s vision,  mission, and performance goals for its programs, including the verification  program. Additionally, VA has developed a high-level operating plan for  fiscal year 2016 that identifies key actions needed to meet OSDBU’s  objectives, such as transitioning to a new verification process, completing  revisions to verification regulations, and developing a new case- management system.
But, VA’s operating plan is not comprehensive and does not include an  integrated schedule with specific actions and milestone dates for  achieving program changes or discuss how the efforts described in the  previous sections might be coordinated. For example, the operational  plan states that VA needs to have in place an information technology  system that allows both case-management and client-relationship  management. However, it does not describe the specific actions that VA  must take to acquire such a system or how system development will be  integrated with other ongoing efforts, such as adoption of a new  verification process. Instead, the operating plan states that a main  element to achieving the goal is to develop a case-management system.  In another example, the operating plan states that VA must have a  verification process that provides a positive veteran experience and that a  main element to achieving that goal is to transition to the new process.  But the plan does not explain the specific actions necessary to fully  transition to the new process or the timetable for the transition. In  addition, VA does not have a process in place to update the verification  program operating plan on a timely basis to ensure that it reflects current  initiatives, conditions, and long-term goals. We previously reported that  useful practices and lessons learned from organizational transformation  show that a transformation, such as CVE’s efforts to make the verification  process more efficient and veteran-friendly, is a substantial commitment  that could take years before it is completed, and therefore must be  carefully and closely managed. As a result, setting implementation goals  and a timeline to build momentum and show progress from day one is  essential for organizations. Tracking implementation goals and  establishing a timeline can help pinpoint performance shortfalls and gaps  and identify the need for midcourse corrections.
According to OSDBU officials, each OSDBU program team (such as  CVE) is to develop an action plan for its specific program that includes  resource needs and expected timelines. The Executive Director stated  that OSDBU intends to review and incorporate each action plan into the  operating plan for all of OSDBU, so that OSDBU has a detailed plan for  the verification and other OSDBU programs. CVE officials told us that  they had initially delayed finalizing CVE’s plan because they were waiting  to evaluate the results of the verification pilot, and intended to finalize the  plan with actions and milestone dates in December 2015.
As of January 29, 2016, CVE had yet to complete an operating plan for  the verification program. Without a plan that contains actions and  milestone dates for the multiple efforts CVE has been undertaking, VA  may face difficulties in managing these efforts to completion.  Furthermore, without engaging in effective operational planning moving  forward, VA lacks assurance that it can achieve its longer-term objectives  for the verification program.
section_title: Conclusions
paragraphs: VA has made significant improvements in its verification program since  our 2013 report, including application processing time, quality control, and  communication with veteran applicants. Nonetheless, the agency  continues to face challenges in making the program less resource- intensive and more efficient and veteran-friendly. VA has acknowledged  these issues and has begun to transform the program to address them.
VA’s efforts to restructure the verification process, realign organizational  structure, and acquire a new case-management system represent  significant efforts for CVE’s team of 16 federal employees. But these  efforts began and have continued in the absence of a detailed operational  plan to guide and integrate them with VA’s strategic objectives. And the  agency has faced challenges with planning—both strategic and  operational—as we found in previous reviews of the program dating to  2010. By putting such a plan in place to guide the program’s  transformation, VA could obtain reasonable assurance that these efforts  will be properly sequenced, managed to completion, and help VA  accomplish its longer-term goals. Moreover, having a detailed plan to  accomplish multiple ongoing efforts is critical given the repeated delays in  VA’s efforts to acquire a new case-management system and the lack of  continuity in CVE leadership. Such a plan is also critical in the context of  VA’s efforts to carry out an organizational transformation and its long-term  goals to expand the program’s capacity to serve more veterans. Without a  policy to review and update the operating plan to reflect current conditions  and priorities, VA would continue to be at risk for delays in implementing  its initiatives and achieving its long-term goals.
section_title: Recommendations for Executive Action
paragraphs: To improve the management and oversight of VA’s SDVOSB and VOSB  verification program, we recommend that the Secretary of Veterans  Affairs direct OSDBU to complete its fiscal year 2016 operating plan and  include an integrated schedule that addresses key implementation goals  and the actions and milestone dates for achieving them, such as the  coordination of the redesign of the verification process and the design,  acquisition, and deployment of a new case-management system; and  establish a process to review and update the operating plan for the  verification program on a timely basis to address new VA initiatives, other  changing conditions, and long-term goals.
section_title: Agency Comments
paragraphs: We provided a draft of this report to the Department of Veterans Affairs  for comment. In its written comments, VA agreed with our  recommendations. Specifically, VA said that it completed a draft of the  fiscal year 2016 operating plan, which received input from the major  OSDBU program areas (including the verification program). A final  version of the plan, which will incorporate key implementation goals and  milestones, will be released by March 31, 2016. VA also stated that  OSDBU has implemented a process to review and update the operating  plan for the verification program and all other program areas on a timely  basis. According to VA, the process will allow OSDBU to address VA  initiatives and programmatic contributions linked to realizing those  initiatives and articulating how other changing conditions and long-term  goals will be managed. VA also provided timeframes for completing its  planned actions. VA provided technical comments and updates on the  status of some of its ongoing initiatives, which we incorporated as  appropriate.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Veterans Affairs, and other interested  parties. In addition, the report will be available at no charge on the GAO  website at http://www.gao.gov. If you or your staff have any questions  about this report, please contact me at (202) 512-8678 or  shearw@gao.gov. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  GAO staff who made key contributions to this report are listed in appendix  III.
section_title: Appendix I: Scope and Methodology
paragraphs: This report assesses progress by the Department of Veterans Affairs (VA)  in (1) establishing a timely and consistent verification program and  improving communication with veterans, and (2) the steps VA has taken  to identify and address program challenges and longer-term goals. We  and others previously identified verification program challenges, including  application processing timelines and quality controls, communication with  veterans, case-management system, and strategic and operational  planning.
To assess VA’s progress in establishing a timely and consistent  verification program, we reviewed relevant statutes, regulations, and  procedures for the verification program. We also reviewed the verification  program quality manual to understand what the Center for Verification  and Evaluation’s (CVE) quality management standards were and how  CVE ensured quality of the program, and reviewed internal audit reports  from December 2014 to March 2015. We interviewed officials in VA’s  Office of Small and Disadvantaged Business Utilization (OSDBU) and  CVE about their policies and procedures for processing applications, for  quality control, for investigating allegations of noncompliance with  program regulations, and changes to the verification program since our  last report (2013).
We conducted a case file review to determine the extent to which VA had  followed its policies, procedures, and quality controls for processing  applications, as well as the extent to which VA had processed  applications within regulatory time limits. We selected a stratified  probability sample of all verification applications (initial and renewal  applications) submitted to VA between June and September 2014. We  chose this time period so we could obtain a sample of applications for  which CVE had completed its application review and so that the  applications in our sample would have been processed under a recent  and similar set of verification procedures. The applications were stratified  by two groups of decision outcomes: (1) withdrawals and (2) approvals  and denials. The sample was designed to make generalizeable estimates  for approvals and denials only, with the sample of withdrawals providing  nongeneralizeable examples. We used simple random sampling methods  to select 96 of the 1,306 applications submitted to VA during this time  frame that resulted in an approval or denial. We developed and pre-tested  an instrument to collect data from the case files on application processing  time frames, completeness of VA’s review, and documentation of key  decisions and rationales. We assessed the reliability of these data by  interviewing VA officials knowledgeable about the data, reviewing  documentation related to the data systems, and checking the data for  illogical values or obvious errors and found them to be sufficiently reliable  for estimating population values. The sample allowed us to estimate the  proportion of cases for which VA consistently followed its policies and  procedures and met regulatory time frames for reviewing applications for  the verification program for applications submitted from June through  September 2014. Because we used simple random sampling methods to  select approvals and denials, our estimated proportions did not require  weighting. We used hypergeometric methods to estimate 95 percent  confidence intervals, which account for the small size of the sample and  population, estimated proportions near 0 or 100, and a nonignorable  sampling fraction.
We reviewed administrative program data obtained from VA on  application processing times for fiscal years 2012 through 2015 and  compared those numbers to our findings from the case file review. We  also reviewed administrative data from VA on the number and type of site  visits conducted in fiscal years 2014 and 2015. We assessed the  reliability of CVE’s administrative program data by interviewing VA  officials and reviewing documentation related to VA’s data system, and  we found the data to be sufficiently reliable for describing VA’s reported  processing statistics.
We also reviewed the files of 10 verified businesses for which CVE  received allegations of noncompliance with program regulations to  identify the steps CVE took to investigate the allegations. To select these  files, we reviewed more than 100 allegations of noncompliance with  program regulations sent to VA between June 2014 and May 2015 by a  veteran-owned small business advocate. We catalogued tips that were  relevant to the verification program (tips that dealt with potential problems  in the verification process versus tips that dealt with VA contracting  issues) and selected 10 firms for which to conduct a more in-depth review  of how VA reviewed or addressed the alleged fraud. We purposefully  selected these firms to obtain variation in the type of allegation (e.g., that  the firm was not owned by a service-disabled veteran but instead was a  “pass-through” or that the firm did not meet the criteria for a small  business), whether an official protest was filed, and results (whether the  firm remained in the verification database). We reviewed the files  associated with each of these firms and collected information about the  steps VA took to respond to allegations of noncompliance and status  protests.
To assess VA’s progress in addressing communication challenges with  veterans, we reviewed VA’s work instructions for processes that involve  communicating with veterans. We also reviewed VA’s website to identify  guidance available for applicants, such as an applicant guide, frequently  asked questions, verification assistance briefs, and an online self- assessment tool for prospective applicants. We interviewed VA officials to  determine what procedures they have in place to communicate with  applicants and verified businesses and obtain feedback from these  entities on VA’s verification process and communication efforts. We  interviewed representatives of two veteran service organizations and four  Procurement Technical Assistance Centers (PTACs)—which provide  verification assistance to veterans—to obtain information about their  opinions of VA’s procedures to verify applications and communicate with  veterans. We selected the veterans groups based on our prior work in the  area and the PTACs based on recommendations from the Defense  Logistics Agency and the Association for Procurement Technical  Assistance Centers and to obtain geographic diversity. There are 98  PTACs in the United States with more than 300 local offices. We  interviewed counselors at the Florida, Missouri, Nevada, and Washington  PTACs. We also analyzed data collected through the case file review to  determine the extent to which VA complied with its procedures for  communicating with applicants and verified businesses. We reviewed  methods VA used to collect feedback from program participants, such as  documents relating to VA’s help desk customer satisfaction surveys,  survey instruments approved by the Office of Management and Budget,  results from surveys that had been deployed as of October 2015, and  results from focus groups conducted to identify areas for improvement in  the verification process. We also assessed the readability of five  determination letters that VA sent to veteran applicants from August  through December 2014 to corroborate testimonial evidence from  veterans groups indicating that these letters can be difficult to understand.  We selected these letters by taking the first five cases from our case file  review sample that had been issued either a predetermination or denial  decision. To determine the reading level at which determination notices  were written, we used an automated readability tool, the Flesch-Kincaid  Grade-Level test, which rates text on a U.S. school grade level.
To assess the steps VA has taken to identify and address verification  program challenges and longer-term goals, we reviewed prior work on the  verification program that we and VA’s Office of Inspector General (OIG)  conducted. We also reviewed VA’s planning, organizational, and budget  documents, such as OSDBU’s 2014-2018 Strategic Plan, OSDBU’s 2016  Operating Plan, CVE organizational charts, and CVE’s budget for fiscal  years 2014 and 2015. We compared these planning documents with  useful practices and lessons learned on organizational transformations,  as identified in previous GAO work. We interviewed VA officials to  determine what steps, if any, they have taken to address issues we or the  OIG identified, and identify and address other challenges associated with  the verification program. We also discussed VA’s plans to restructure the  verification process with officials from VA’s OSDBU and CVE.
We also reviewed documents pertaining to the new verification process,  such as process maps and work instructions for the pilot verification  process, the reverification policy issued in October 2015, and the revised  program regulations posted for public comment in November 2015. We  reviewed VA’s human capital and staff management practices, including  CVE’s organizational structure, leadership, and reliance on contractors to  conduct verification activities. We analyzed VA data on the number of  contractors and federal staff working on the verification program and  compared these numbers to those found in our 2010 and 2013 reports.  We used testimonial evidence obtained during interviews with contractors  and agency officials to describe the responsibilities of contractor and  federal staff. We also reviewed VA’s policy and planning documents and  position descriptions to describe the changes it plans to make to its  organizational structure.
To assess the progress VA has made in modifying or replacing its  information technology system for case management, we reviewed  project planning documents that included a list of technical requirements,  as well other contract documents. We interviewed VA officials about their  plans for developing the new case-management system, including plans  for issuing a solicitation, fully transitioning to the new system, and  ensuring the system supports the pilot verification process.
We conducted this performance audit from December 2014 to March  2016 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Veterans Affairs
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Harry Medina (Assistant  Director); Katie Boggs (Analyst-in-Charge), Meghana Acharya, Mark Bird,  Charlene Calhoon, Pamela Davidson, Kathleen Donovan, Beth Faraguna,  John McGrail, Barbara Roesmann, and Jeff Tessin made key  contributions to this report.