VA Health Care: Goals and Related Measures Needed to Better Assess the Impact of Same-Day Services
Released: 2019-08-07T00:00:00

Letter

Background

In response to the 2014 access crisis, VA launched the MyVA initiative, which was designed to transform the health care experience of veterans. In concert with the MyVA initiative, VA introduced the MyVA Access Declarations in April 2016 with the goal of improving access by providing veterans more control as to how they receive their health care. The MyVA Access Declarations was a list of nine “access declarations” that were intended to serve as the foundational principles for improving and ensuring access to care. Two of these “access declarations” required providing timely primary and mental health care and included same-day services.

VHA Policies on Same- Day Services

VHA had policies in place for same-day services in primary and mental health care clinics for several years prior to the same-day-services initiative. In primary care, the 2014 Patient-Aligned Care Team (PACT) handbook required all primary care providers and registered nurses to ensure they provide same-day access (unless it is too late in the day as determined by the individual facility) for face-to-face encounters, telephone encounters and, when required by VHA guidance or policy, other types of encounters. The PACT handbook was supplemented by a 2015 VHA memo on unscheduled patient walk-ins. The memo states that if an unscheduled patient presents at a PACT clinic with a clinical concern, the patient cannot be turned away without evaluation by a clinical member of the team, regardless of clinic hours, resource availability, or eligibility/enrollment status.

VHA also had previously developed policies stating that veterans are entitled to timely access to mental health care. Specifically, a 2007 VHA memo required that all veterans requesting or referred for mental health care or substance abuse treatment receive an initial evaluation within 24 hours. VHA’s 2015 Uniform Mental Health Services handbook also noted that all new patients requesting or referred for mental health care services must receive an initial evaluation within 24 hours and a more comprehensive diagnostic and treatment planning evaluation within 30 days.

Additionally, since 2008, VHA has required the integration of primary care and certain mental health care services at VA medical centers serving a veteran population greater than 5,000. This care model, known as Primary Care–Mental Health Integration (PC-MHI), integrates mental health staff into each primary care PACT clinic, allowing veterans to receive services for depression, anxiety, post-traumatic stress disorder, and substance use without needing to obtain a separate referral to providers in the mental health care clinic. According to VHA guidance, PC-MHI has been shown to improve access to same-day mental health care and reduce no-show rates to appointments.

Oversight of VHA Access to Care Efforts

VHA’s veterans access to care office was created in 2016 as the national oversight office for VHA access-to-care issues. Additionally, each VISN is responsible for overseeing the VA medical centers within their designated regions. This oversight includes oversight of access issues and the implementation of initiatives such as the same-day service initiative. VA medical center directors are responsible for ensuring local policies are in place for the effective operation of their primary and mental health care clinics, including affiliated CBOCs.

VHA Used a Five- Pronged Approach to Design Its Same-Day Service Initiative; Selected VA Medical Centers Relied on Previous Approaches to Implement It

VHA Used a Five-Pronged Approach to Design and Set Up the Same-Day- Services Initiative

VHA used a five-pronged approach to design its same-day services initiative: VHA (1) defined same-day services, (2) developed guidance, (3) updated its mental health policies, (4) offered training, and (5) assessed VA medical center readiness to implement the initiative.

VHA defined same-day services. As an initial step, VHA leadership developed the following definitions of same-day services in primary and mental health care:

Same-day services in primary care: “When a veteran requires primary care services right away, during regular business hours, he or she will receive services the same day at a VA medical center. If a veteran calls after normal business hours, he or she will receive care the next business day.”

Same-day services in mental health: “If a veteran is in crisis or has another need for mental health care right away, he or she will receive immediate attention from a health care professional at a VA medical center.”

VHA also identified a variety of ways in which veterans can receive same- day services, including: (1) providing a face-to-face visit; (2) returning a phone call; (3) arranging a telehealth or video care visit; (4) responding to a secure email; or (5) scheduling a future appointment.

VHA developed guidance for the same-day service initiative. To help VA medical centers implement its definition of same-day services, in April 2016, VHA developed written guidance—the MyVA Access Implementation Guidebook. The guidebook provides a variety of solutions to help VA medical centers meet the intent of the same-day service initiative. The guidebook includes specific solutions for VA medical centers struggling to provide same-day services in primary or mental health care for veterans with urgent care needs: Implementing open access in primary and mental health care: Open access aims to balance the supply of (for example, available appointments) and demand for (for example, the number of patients assigned to a provider and annual visits per patient) services to increase patient access. Achieving open access requires implementing specific strategies including achieving full staffing, planning for contingencies such as clinical staff absences or vacancies and managing the number of times patients see a provider each year, among other strategies.

Implementing primary care-mental health integration: In order to complete the implementation of PC-MHI across the VA system, the guidebook suggests facilities address staffing vacancies, develop a PC-MHI implementation plan, and choose an open access scheduling model (for example, full open access where there are no scheduled appointments and patients are seen on a first come, first served basis), among other things.

Utilizing same-day referrals to mental health for suicide prevention: This solution reiterates many of the mental health policy changes that VHA introduced in conjunction with the same-day service initiative such as implementing an initial screening evaluation, developing a process for same-day care for established patients with an urgent need, and deploying open access scheduling, among other things.

The guidebook states that all of the solutions were chosen because they were used successfully at other VA medical centers; can be quickly implemented; and have a high impact on veterans’ access to care. The guidebook also notes that flexibility is a key element when choosing solutions and explains that VA medical centers should select and modify solutions as needed. The guidebook does not make any of the solutions mandatory; however, several of the mental health solutions were introduced to facilities through separate VHA memos and are required.

VHA updated mental health policies. VHA updated certain mental health policies to facilitate the implementation of the same-day services initiative. Specifically, in April 2016 VA issued a memo updating its mental health policy to require that any veteran new to mental health services requesting or referred for care in person be seen the same day by a licensed independent provider to screen for and address immediate care needs. This was a change from the previous timeframe of 24 hours for an initial evaluation. The memo also created new processes for VA medical centers to assess same-day services in mental health care clinics, including a medical chart review and a one-time review of standard operating procedures to ensure that the new guidelines are being followed. VHA also distributed other memos that either sought to clarify existing guidance or expand same-day services into other areas of mental health care, such as substance abuse. Additionally, VHA provided a memo to VA network directors and mental health leads about scheduling models for mental health care that all VA medical centers needed to implement for the same-day service initiative.

VHA provided training on the same-day-services initiative. VHA provided voluntary training for same-day services some of which discussed the solutions from the guidebook and the updated mental health policy. The trainings began in February 2016 for primary care and in May 2016 for mental health. The trainings consisted of national telephone calls (often with slide presentations) that any VA medical center staff member could join, and the presentation materials were posted to VHA’s internal website. The telephone trainings generally occurred twice a month in primary care and every week in mental health care.

VHA assessed VA medical center same-day service readiness. Beginning in January 2017, VHA provided technical assistance around same-day services to VA medical centers. VHA reviewed several aspects of same-day services, including how VA medical centers were able to provide same-day services and identified any approaches that may have needed improvement. Generally, low-performing VA medical centers received continuous on-site support; moderate performing VA medical centers received a combination of virtual and on-site support; and, high performing VA medical centers primarily received virtual support.

To determine the progress that VA medical centers were making in providing same-day services, VHA conducted surveys that required medical center directors to self-certify—and, in some cases, VISN directors to validate—that their VA medical centers (including affiliated CBOCs) were able to provide same-day services. In the event that a VISN director could not validate medical center survey information, VHA followed up with the medical center and VISN director to create an action plan to mitigate any issues that were delaying validation. These surveys were conducted in 2016 and 2017; focused on either primary care, mental health care or both; and varied in the information collected to determine how VA medical centers were providing same-day services (See Table 2 for information on the same-day-services readiness assessment surveys used by VHA).

According to VHA, all VA medical centers were offering same-day services in primary and mental health care by December 2016. In January 2018, VHA announced that same-day services in primary and mental health care had been achieved in all VA medical centers and CBOCs (more than 1,000 facilities).

Selected VA Medical Centers Generally Relied on Previous Approaches to Implement the Same- Day-Services Initiative

Officials we spoke with from all six VA medical centers in our review told us they were providing same-day services in primary and mental health care prior to the same-day service initiative, an assertion supported by VHA survey data. For example, in a VHA survey conducted in May 2016, around the same time as the launch of the same-day service initiative, 142 out of 165 officials (86 percent) that responded to the survey said that their medical centers offered same-day appointments “always” or “very frequently” in primary care for urgent concerns.

We found that the VA medical centers in our review used a variety of approaches in providing same-day services in primary and mental health care, most of which were in existence before the initiative. As noted earlier, VHA did not require the implementation of any specific solutions in the guidebook and afforded VA medical centers the flexibility to choose appropriate local solutions for the same-day service initiative. Many VA medical centers used this flexibility to continue providing same-day services as they had prior to the initiative often because that is what their resources allowed them to do or, in the case of mental health, because it was built into the foundation of their service line. VHA officials noted that mental health services—particularly PC-MHI—were built around same- day services so VHA’s guidance was familiar to them. The approaches used by the selected VA medical centers included using “float providers” who had not already been assigned specific patients to assist those who requested same-day services; carving out specific appointment times in the schedule for walk-ins; overbooking appointments in providers’ schedules, and offering walk-in clinics.

VHA suggested that certain solutions should be prioritized if VA medical centers were struggling to provide same-day services and, in particular for mental health, created new requirements around same-day services. However, officials at selected VA medical centers noted that some of the suggested solutions in the guidebook—particularly open access—and requirements in updated mental health policies were difficult to implement because of longstanding challenges with staffing, space, or competing VHA policies. For example, VHA’s guidebook suggests the implementation of open access in primary and mental health care in such situations. However, officials at four of the six VA medical centers we visited noted that open access was difficult to implement because of the long-standing challenges mentioned above. In addition, VHA updated its mental health policy to include that any veteran new to mental health services requesting or referred for care in person be seen the same day by a licensed independent provider to screen for and address immediate care needs. However, one medical center official noted that they had designed their mental health clinic processes around registered nurses, who are responsible for completing the initial assessments of new patients. The official added that the medical center did not have licensed independent providers readily available at certain facilities to help complete the assessments in a timely manner.

Officials at all six medical centers we visited noted that implementation was also sometimes challenging as veterans’ expectations shifted with the same-day-services initiative, with veterans’ expecting more immediate access to care from physicians for a variety of conditions. For example, one medical center official noted that veterans are presenting for care and wanting to see a provider because it is these veterans’ understanding that they could get care immediately for any condition including chronic, less urgent issues. Additional officials at the same facility echoed this concern and noted that they are not certain that this was the policy’s intent. Another medical center official noted that several medical center officials asked VHA to change the name “same-day service” because it gives the impression that veterans would always be able to see their provider immediately. This official added that there is some confusion for both staff and veterans about what are same-day services. Additionally, according to one veterans service organization official that we spoke with, a small number of veterans reported that the availability of same-day services varied by facility (VA medical center versus CBOC) and location (urban versus rural). Another medical center official noted that same-day services are not sustainable if the definition is immediate care by a provider for any condition, especially non-urgent issues.

VHA officials told us that the same-day service initiative was a response to the 2014 access crisis and they wanted facilities to use the resources available to them rather than waiting on new policies and strategies. They stated that their main concern was that veterans’ needs were met, not necessarily how they were met. As such, VHA officials told us that they found VA medical centers’ implementation of same-day services acceptable. The VHA officials added that the guidebook is still the foundational document for same-day services. VHA officials told us that it is important for VA medical centers to educate patients on the appropriate use of same-day services. They added that in fiscal year 2019 they are (1) developing a more precise definition of same-day services; (2) developing a website to better explain the purpose of the initiative; and (3) requiring on-demand trainings to provide a clearer explanation about what same day services are available and what staff roles and responsibilities are, among other things. The training is expected to be completed no later than the first quarter of fiscal year 2020.

VHA Has Not Documented Objectives or Developed Performance Goals and Related Measures to Assess the Impact of Same- Day Services on Veterans’ Access to Care

VHA is limited in its efforts to assess the impact of same-day services due to its lack of documented objectives, developed performance goals and related performance measures. Our previous work has shown the benefit of fully connected objectives and performance goals with measurable targets. Objectives state the longer term desired impact or outcome to be achieved, while performance goals communicate the target the agency seeks to achieve within a certain timeframe. Performance measures are indicators of the progress the agency is making towards a goal or target within a particular time frame.

VHA officials told us that the overall objectives of same-day services are to improve veterans’ access to care and customer service while having minimal impact on medical centers’ existing workflows. However, VHA has not documented these objectives—for example, in a directive. In addition, VHA has not developed and documented performance goals that, with associated performance measures, would facilitate monitoring of progress towards the desired outcome of the same-day services initiative. VHA officials stated that the same-day-services initiative was developed quickly in response to the 2014 access crisis, and noted that at the time, their focus was “to get something out quickly” instead of taking time to standardize the initiative around specific policies and procedures, which could include documenting objectives and developing performance goals. VHA officials acknowledged that their decision to focus on quickly implementing the initiative without documenting objectives and developing performance goals and associated performance measures makes assessing the impact of the same-day services initiative more challenging.

VHA has taken some steps to collect data on same-day services. For example, VHA officials stated that they primarily rely on two measures to assess the impact of the same-day services initiative: patient experience scores and the number of same-day appointments. However, without performance goals these measures do not provide VHA with a means to monitor progress and provide limited information on same-day services’ impact.

Patient experience score: VHA uses the Survey of Healthcare Experiences for Patients (SHEP) to measure veterans’ perceptions of their experience at VA medical centers. For same-day services, VHA monitors responses to two questions. According to VHA officials, the key measure is based on the survey question that asks “in the last 6 months, when you contacted this provider’s office to get an appointment for care you needed right away, how often did you get an appointment as soon as you needed?” While SHEP scores provide some data related to customer service and access to care, VHA has not developed performance goals that sets targets for these or other aspects of the same-day services initiative that would benefit from monitoring. Such goals would better enable VHA to identify gaps in performance and plan any needed improvements; ensure balance between agency priorities, such as customer service and access; and identify unintended effects, such as disruption to clinic workflows. For example, officials at one medical center told us that focusing on customer service creates issues with respect to routine care in that veterans’ definition of customer service is based on what makes them happy, while providers are focused on providing the best treatment. Officials added that these two definitions do not always align. In addition, officials at another medical center stated that implementing same-day services impacted their providers’ schedules and the resulting changes to their processes created chaos.

Number of same-day appointments: VHA measures the number of same-day appointments, which, according to a VHA official, are identified in VHA data as appointments completed on the same day they are created in VHA’s scheduling system. According to a VHA training document, VA completed 12 million same-day appointments in fiscal year 2018. However, without performance goals with clear targets for same-day appointments, an official from one VISN said she was unclear how many same-day appointments medical centers should be scheduling. Additionally, same-day services performance goals may afford VHA the opportunity to monitor other key measures—such as those that capture services that do not require an appointment—which could provide VHA with important information on the impact of same-day services on access to care. Moreover, performance goals and additional performance measures may help prevent unintended consequences, such as an over-emphasis on same-day appointments as the way to provide same-day services, which VHA officials stated they are working to curb. For example, officials at two selected medical centers also noted that measuring the number or proportion of same-day appointments does not capture all the ways medical centers provide same-day services. Officials at two other selected medical centers noted they can meet veterans’ same-day needs through multiple avenues, such as a registered nurse providing patient education or by renewing a prescription, that do not require an appointment and therefore, would not be counted in the number of same-day appointments. VHA officials stated that the impact of the same-day services on access to care is difficult to measure and additional measures would help properly measure the impact.

VHA’s lack of documented objectives and developed performance goals and related measures is inconsistent with our prior work on effective management practices and federal internal control standards. Specifically, we have previously reported that performance measures benefit from certain key practices, such as breaking down of broad long- term objectives into specific near-term performance goals with measurable targets and time frames, and key attributes, such as balance to prevent skewed incentives over-emphasizing certain goals. Additionally, Standards for Internal Control in the Federal Government states that documentation provides a means to retain organizational knowledge and mitigate the risk of having that knowledge limited to a few personnel. Without clearly documented objectives, performance goals, and related performance measures, VHA is hindered in its efforts to define success for its same-day service initiative and measure progress achieving it.

VHA officials stated they rely on VISN and VA medical center officials to oversee same-day services; however, we found that without performance goals and related performance measures, VISNs and VA medical centers found it challenging to oversee the same-day services initiative. Specifically, officials at five of the six medical centers and two of the four VISNs we visited stated that it is difficult to measure same-day services; which in turn makes assessing the initiative’s impact on veterans’ access to care difficult. Officials at one medical center explained that the challenge stems from the fact that that VHA has not defined what outcome it wants to achieve. In addition, officials at another VA medical center stated that they have a number of access measures available to them, but it was unclear to them which measures they should be prioritizing as part of their oversight of the same-day services initiative.

Further, absent performance goals, we found that VISNs and medical centers, which operate in a decentralized environment, varied in their oversight strategies. For example, one VISN required all medical centers to complete a self-assessment of their access capacity and sustainability, and collected information on a number of key open access elements, including Patient-Aligned Care Team staffing levels and provider panel sizes, among others. However, oversight by other VISNs was reportedly less robust. For example, at one VISN, officials stated it is difficult to audit access broadly and described their oversight of same-day services as “fairly minimal.”

At the medical center level, oversight also varied as officials tried to develop their own oversight solutions. Officials at one medical center we visited used a feature within the outpatient appointment scheduling system that allowed them to count the specific services, such as pharmacy refills, that veterans seeking same-day mental health care had requested. According to these officials, the tool provided additional data not found in existing VHA access-related reports and allowed them to better understand veterans’ demand for specific same-day services and utilize resources more efficiently. These officials added that they developed this solution because they had not received guidance from VHA on how they should measure demand, and they had skilled staff with the ability to develop their own measures. However, not all VA medical centers we visited had the skilled staff to develop similar solutions. Developing performance goals and related performance measures would better position VHA to obtain useful, comparable information on the impact of same-day services on access to care across VISNs and medical centers.

Moving forward, VHA is planning to conduct a “mystery shopper” evaluation of same-day services to assess the impact of same-day services. The mystery shopper evaluation will consist of various scenarios in which veterans, engaged through a contractor, will attempt to access same-day care at a variety of clinics in VA medical centers. As described in a VHA planning document, the evaluation is intended to provide VHA with information on veterans’ experience in obtaining same-day services and will attempt to understand variations in how same-day services are provided. However, VHA officials have not determined if the evaluation will be ongoing. VHA officials stated that in addition to the mystery shopper evaluation, they are considering additional measures to better assess the impact of same-day services beyond their current measures, such as the number of pharmacy refills completed the same day they were requested. However, as of May 2019, VHA had not developed specific performance goals to align these measures to, or set timeframes for their creation. Without overall performance measures that are tied to documented performance goals, VHA will continue to be limited in its ability to assess the impact of same-day services on veterans’ access to care.

Conclusions

VHA’s same-day services initiative for primary and mental health care is one of several efforts by VHA to help improve veterans’ access to care in the 5 years since access issues garnered national attention. VHA’s stated objectives for the same-day-services initiative are to improve veterans’ access to care and customer service while having minimal impact on medical centers’ existing workflows. However, VHA has not documented these objectives or developed performance goals and related measures that provide for monitoring towards the desired outcomes. VHA primarily relies on veteran satisfaction scores and the number of same-day appointments to monitor the same-day-services initiative, but these measures alone do not enable an assessment of the impact of same-day services on access to care. Without documented objectives, and performance goals and related measures tied to these goals, VHA will continue to be limited in its ability to determine, how, if at all, the same- day-services initiative has improved veterans’ access to care.

Recommendation for Executive Action

The Under Secretary for Health should document same-day services objectives and develop performance goals and related performance measures to facilitate the periodic assessment of the impact of same-day services on veterans’ access to care. (Recommendation 1)

Agency Comments

We provided a draft of this report to VA for review and comment. In its written comments, which are reproduced in appendix I, VA concurred in principle with our recommendation. VA stated that its Office of Veterans Access to Care will clarify objectives, develop performance goals, and explore the options for reliable performance measures. VA noted that identifying options for performance measures will take approximately 9 months and that additional time may be needed for development, testing and refinement. VA provided a target completion date of April 2020.

We are sending copies of this report to the appropriate congressional committee and the Secretary of Veterans Affairs. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at DraperD@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II.

Appendix I: Comments from the Department of Veterans Affairs

Appendix II: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Ann Tynan (Assistant Director), Dan Klabunde (Analyst-in-Charge), Jennie F. Apter, and Q. Akbar Husain made key contributions to this report. Also contributing were Muriel Brown, Jacquelyn Hamilton, Ethiene Salgado-Rodriguez, and Merrile Sing.