VA Primary Care: Improved Oversight Needed to Better Ensure Timely Access and Efficient Delivery of Care
Released: 2015-10-22T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: VA's 150 medical facilities manage primary care services provided to veterans. VA requires facilities to record and report data on primary care panel sizes to help facilities manage their workload and ensure that veterans receive timely and efficient care. VA also requires facilities to record and report data on primary care encounters and expenditures.

GAO was asked to examine these data and VA's oversight of primary care. This report examines (1) VA's panel size data across facilities and how VA uses these data to oversee primary care, and (2) VA's encounter and expenditure data across facilities and how VA uses these data to oversee primary care. GAO analyzed fiscal year 2014 data on primary care panel size, encounters, and expenditures for all VA facilities. GAO also conducted a more in-depth, nongeneralizable analysis of data and interviewed officials from seven facilities, selected based on geographic diversity and differences in facility complexity. GAO also interviewed VA Central Office and network officials to examine their oversight of primary care, including the extent to which they verify the data and use it to monitor the management of primary care.

section_title: What GAO Found
paragraphs: GAO found that the Department of Veterans Affairs' (VA) data on primary care panel sizes—that is, the number of patients VA providers and support staff are assigned as part of their patient portfolio—are unreliable across VA's 150 medical facilities and cannot be used to monitor facilities' management of primary care. Specifically, as part of its review, GAO found missing values and other inaccuracies in VA's data. Officials from VA's Primary Care Operations Office confirmed that facilities sometimes record and self-report these data inaccurately or in a manner that does not follow VA's policy and noted that this could result in the data reliability concerns GAO identified. GAO obtained updated data from six of seven selected facilities, corrected these data for inaccuracies, and then calculated the actual panel sizes for the six facilities. GAO found that for these six facilities the actual panel size varied from 23 percent below to 11 percent above the modeled panel size, which is the number of patients for whom a provider and support staff can reasonably deliver primary care as projected by VA. Such wide variation raises questions about whether veterans are receiving access to timely care and the appropriateness of the size of provider workload at these facilities.

Moreover, GAO found that while VA's primary care panel management policy requires facilities to ensure the reliability of their panel size data, it does not assign responsibility to VA Central Office or networks for verifying the reliability of facilities' data or require them to use the data for monitoring purposes. Federal internal control standards call for agencies to clearly define key areas of authority and responsibility, ensure that reliable information is available, and use this information to assess the quality of performance over time. Because VA's panel management policy is inconsistent with federal internal control standards, VA lacks assurance that its facilities' data are reliable and that the facilities are managing primary care panels in a manner that meets VA's goals of providing efficient, timely, and quality care to veterans.

In contrast to VA's panel data, GAO found that primary care encounter and expenditure data reported by all VA medical facilities are reliable, although the data show wide variations across facilities. For example, in fiscal year 2014, expenditures per primary care encounter—that is, a professional contact between a patient and a primary care provider—ranged from a low of $150 to a high of $396 after adjusting to account for geographic differences in labor costs across facilities. Such wide variations may indicate that services are being delivered inefficiently at some facilities with relatively higher per encounter costs compared to other facilities. However, while VA verifies and uses these data for financial purposes, VA's policies governing primary care do not require the use of the data to monitor facilities' management of primary care. Federal internal control standards state that agencies need both operational and financial data to determine whether they are meeting strategic goals and should use such data to assess the quality of performance over time. Using panel size data in conjunction with encounter and expenditure data would allow VA to assess facilities' capacity to provide primary care services and the efficiency of their care delivery. By not using available encounter and expenditure data in this manner, VA is missing an opportunity to potentially improve the efficiency of primary care service delivery.

section_title: What GAO Recommends
paragraphs: GAO recommends that VA verify facilities' panel size data, monitor and address panel sizes that are too high or too low, and review and document how to use encounter and expenditure data to help monitor facilities' management of primary care. VA agreed with GAO's recommendations and described its plans to implement them.

section_title: Letter
paragraphs: The Department of Veterans Affairs’ (VA) Veterans Health Administration  (VHA) provided care to about 6.6 million veterans in fiscal year 2014 and  spent about $58 billion for their care in that year. Primary care services  are often the entry point to the VA health care system for veterans,  including an aging veteran population and a growing number of younger  veterans returning from military operations in Afghanistan and Iraq.  Veterans may obtain primary care services at VA’s 150 medical facilities,  which include medical centers and more than 800 community based  outpatient clinics (CBOCs). Primary care services are delivered through  patient aligned care teams consisting of a primary care provider and  support staff—a nurse care manager, clinical associate, and  administrative clerk. When other services are needed to meet patient  goals and needs, these teams oversee and coordinate that care. VA’s  medical facilities are overseen by 21 Veterans Integrated Service  Networks—called networks—and by VA’s Central Office, which oversees  the entire VA health care system.

In recent years, VA has emphasized providing primary care as the way to  enhance health care delivery to meet veterans’ needs. In support of this  emphasis, VA has detailed guidelines regarding how primary care is to be  provided and staffed, including a requirement that medical facilities record  and report data on primary care panel size—that is, the number of  patients for whom a patient aligned care team delivers primary care  services. According to VA, panel size is an important factor in helping VA  determine the total number of patients that can be cared for in the VA  health care system; moreover, maintaining appropriate panel sizes helps  ensure that providers will be able to offer quality care in a timely manner  to a reasonable number of patients. Panel sizes that are too high may  lead to veterans experiencing delays in obtaining care among other  negative effects, and panel sizes that are too low may be associated with  inefficiency and wasted resources. Data on panel size can also help VA  facilities manage and VA Central Office and networks oversee VA’s  primary care program. In addition to data on panel sizes, VA policy also  requires facilities to record and report data on primary care encounters— that is, the number and types of visits to a primary care provider—and  expenditures for delivering primary care. According to VA officials, data  on encounters and expenditures may help VA oversee its primary care  program by identifying potential inefficiencies.

GAO and others have recently reported that VA has not provided  veterans timely access to care. To address such issues, Congress  enacted the Veterans Access, Choice and Accountability Act of 2014,  which, among other things, requires VA to submit biennial reports on  various factors related to staffing, including primary care staffing models.  Members of Congress have raised questions about the extent to which  VA has sufficient resources to provide more timely access to quality  primary care and whether these resources are being used efficiently in  the service of this goal. Members have also raised questions about how  data on primary care panel sizes, encounters, and expenditures are being  used to oversee the primary care provided across VA. We were asked to  examine what these data show and how VA uses available data to  oversee its primary care program. In this report, we examine  1.  what VA’s panel size data show across facilities and how VA Central  Office and networks use these data to oversee primary care, and  2.  what VA’s encounter and expenditure data show across facilities and  how VA Central Office and networks use these data to oversee  primary care.

To examine what VA’s panel size data show across facilities and how VA  Central Office and networks use these data to oversee primary care, we  reviewed VA data and policy documents and interviewed officials from  VHA’s Primary Care Operations Office, within VA Central Office. We  analyzed fiscal year 2014 data from the Primary Care Operations Office  for all of VA’s facilities on 1) modeled panel size, which represents the  number of patients on a primary care panel for whom a patient aligned  care team is expected to deliver primary care as projected by VA for each  facility, and 2) reported panel size, which represents the average number  of patients on a primary care panel that each facility reports to the  Primary Care Operations Office. We did not evaluate VA’s model for  developing panel sizes. We also reviewed the number of patients, full- time equivalent (FTE) providers, support staff, and exam rooms that each  facility records and reports to the Primary Care Operations Office. In  addition to examining the panel size data for all VA facilities, we  conducted a more detailed examination of these data at seven facilities,  which were selected based on geographic diversity and differences in  facility complexity. The seven facilities we selected were located in  Detroit, Michigan; Dublin, Georgia; Los Angeles, California; Pittsburgh,  Pennsylvania; St. Cloud, Minnesota; Temple, Texas; and Washington,  D.C.

We conducted a data reliability assessment of the data for all of VA’s  facilities. This assessment included checks for missing values and  outliers as well as interviews with VA officials knowledgeable about the  data, including officials from selected facilities, the networks responsible  for the oversight of the selected facilities, and the Primary Care  Operations Office. Our review of the data revealed concerns about  facilities’ reported panel sizes and other reported data elements that were  attributable to inaccuracies in how facilities recorded the data, which  precluded us from using these data to report panel sizes across all of  VA’s facilities. As a result of these data inaccuracies, we only calculated  actual panel sizes for six of the seven selected facilities where we were  able to use updated data provided by each facility and correct for  inaccuracies. We then compared the actual panel size to each facility’s  modeled panel size for fiscal year 2014 and examined potential reasons  for any variations we observed. We also examined the extent to which  reported panel size and related data were being accurately recorded and  reported by the seven facilities. Our analysis is not generalizable beyond  the facilities we reviewed. In addition, we assessed oversight activities  across the networks we reviewed and for VA Central Office in the context  of federal internal control standards for control environment, information  and communication, and monitoring. Federal internal control standards  provide an overall framework for establishing and maintaining internal  control and for identifying and addressing major performance and  management challenges and areas at greatest risk of fraud, waste,  abuse, and mismanagement. Under the standards for internal control,  control environment requires that the agency’s organizational structure  clearly defines key areas of authority and responsibility; information and  communication requires reliable information to determine whether an  agency is meeting its goals for accountability for effective and efficient  use of resources; and monitoring should assess the quality of  performance over time.

To examine what VA’s encounter and expenditure data show across  facilities and how VA Central Office and networks use these data to  oversee primary care, we reviewed VA data and policy documents and  interviewed officials from the seven selected facilities, the networks  responsible for the oversight of the selected facilities, and VHA’s Primary  Care Operations Office and the Office of Finance. We analyzed fiscal  year 2014 primary care encounter data—including the number of  encounters for both face-to-face and telephone primary care  appointments—expenditure data, and data on the number of unique  primary care patients for all VA facilities. Using these data, we calculated  expenditures per encounter and expenditures per unique patient across  all of VA’s medical facilities and identified cases where facilities had  expenditures per encounter that were significantly above or below the VA  average. We conducted a data reliability assessment of the data that we  used, which included checks for missing values and interviews with  officials from the Office of Finance, who were knowledgeable about the  data. Our review revealed some inconsistencies and errors in the  encounter and expenditure data reported by facilities, which we discussed  with officials from the Office of Finance. Overall, however, we found that  these inconsistencies and errors did not compromise the overall  completeness and accuracy of the encounter and expenditure data. As a  result, we determined that these data were sufficiently reliable for our  purposes. In addition, we assessed the oversight activities performed by  the networks that oversee the seven facilities we reviewed and by VA  Central Office in the context of the federal internal control standards for  information and communication and monitoring.

We conducted this performance audit from October 2014 to October 2015  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Within VA Central Office, VHA’s Primary Care Services Office develops  policies related to the management of primary care—including the  recording and reporting of primary care panel size data—and VHA’s  Primary Care Operations Office is responsible for executing policies  related to primary care delivery and monitoring primary care. VHA’s Office  of Finance develops policies related to the recording and reporting of  primary care encounter and expenditure data. Each of VA’s 21 networks  is responsible for overseeing the facilities within their network, and this  responsibility includes overseeing facilities’ management of primary care.  (See fig. 1.)
subsections: section_title: Panel Size
paragraphs: Based on a review of studies, VA established a baseline panel size of  1,200 patients at any given time for a full-time primary care physician  provider. The Primary Care Services Office adjusts the baseline panel  size for each facility based on a model VA officials said they developed in  2003 that uses data reported by facilities—including data on the number  of FTE providers, support staff, and exam rooms—and projections on the  average number of primary care visits. These projections are based on  patient characteristics, such as the proportion of patients with chronic  conditions. VA refers to the adjusted baseline for each facility as the  “modeled panel size,” which in fiscal year 2014 ranged from 1,140 to  1,338 across VA’s facilities. VA generally updates the modeled panel size  annually for each facility.

VA’s handbook on primary care management requires that facilities  record and report primary care data using the Primary Care Management  Module (PCMM) software. These data include the number of patients,  FTE providers, support staff, and exam rooms, and the reported and  modeled panel size. Each facility maintains its own PCMM software and  is required to update its panel size data on an ongoing basis in PCMM,  which electronically reports facilities’ data to a separate national database  maintained by the Veterans Support Service Center. This national  database allows the Primary Care Operations Office and VA’s networks  to review the data.

section_title: Encounters and Expenditures
paragraphs: An encounter is a professional contact between a patient and a provider  who has the primary responsibility for diagnosing, evaluating, and treating  the patient’s condition. In addition to individual office visits, there are  other types of encounters, such as telephone visits and group visits.  Each facility identifies and tracks all of its expenditures associated with  primary care encounters. Facilities transmit their encounter and  expenditure data using the Decision Support System, which is maintained  by the Office of Finance. This office is responsible for collecting and  maintaining financial information for VA’s cost accounting—which  identifies and assesses the costs of programs at the national, network,  and facility levels—and for budgetary purposes.

section_title: VA Lacks Reliable Data on Medical Facilities’ Primary Care Panel Sizes and Does Not Have Effective Oversight Processes to Verify and Use Facilities’ Data to Monitor Primary Care Data Reported by VA Facilities on Primary Care Panel Sizes Are Unreliable
paragraphs: We found that VA lacks reliable data on primary care panel sizes across  its facilities because the data that facilities record and report to VA  Central Office and networks are sometimes inaccurate. Because reliable  reported panel sizes were not available for all facilities, we calculated  actual panel sizes at six of seven selected facilities and compared them  to each facility’s modeled panel size for fiscal year 2014. We found that  actual panel sizes across the six facilities varied from 23 percent below to  11 percent above their respective modeled panel size. Moreover, we  found that VA Central Office and networks do not have effective oversight  processes for verifying and using facilities’ panel size data to monitor  facilities’ management of primary care.

We found that VA lacks reliable data on primary care panel sizes across  its 150 facilities because the data that facilities record in the PCMM  software and report to the Primary Care Operations Office and to  networks are sometimes inaccurate. Federal internal control standards  state that reliable information is needed to determine whether an agency  is meeting its goals for accountability for effective and efficient use of  resources.

However, our review of the reported panel size data for all of VA’s  facilities for fiscal year 2014 revealed missing values as well as values  that appeared to be unreasonably high or low, which raised concerns  about these data. Officials from the Primary Care Operations Office,  whom we interviewed about the reliability of these data, agreed that  inaccuracies exist in the way facilities report data elements in PCMM,  such as the number of patients assigned to primary care panels and the  number of FTE providers, support staff, and exam rooms. Primary Care  Operations Office officials pointed out that because the data are self- reported, facilities can and sometimes do record the data inaccurately or  in a manner that does not follow VA’s policy on panel management. For  example, the officials stated that some facilities may not count support  staff and exam rooms as outlined in VA’s policy. These officials also  stated that PCMM has limitations that may affect the reliability of facilities’  reported panel size data. For example, officials explained that the  software makes it difficult for facilities to ensure that inactive patients (i.e.,  those who have not seen their primary care provider within the preceding  two years or have died) are removed from providers’ panels.

We identified similar inaccuracies in our more in depth review of panel  size data reported by the seven selected facilities. Specifically, at three  facilities we found inaccuracies in the reported number of FTE primary  care providers and the reported number of patients, which impacted the  facilities’ reported or modeled panel sizes. For example, the number of  FTE primary care providers reported by one of these facilities was too   low because the facility incorrectly recorded each FTE provider as only   90 percent of a FTE. We did not identify inaccuracies in the data  reported by the remaining four facilities. (See table 1.)

Because some medical facilities’ reported panel size data are unreliable,  VA Central Office and network officials cannot readily determine each  facility’s average primary care panel size nor compare these panel sizes  to each facility’s modeled panel size to help ensure that care is being  delivered in a timely manner to a reasonable number of patients.  Moreover, having unreliable data can misinform VA in other aspects as  well. For example, because VA’s model is based on historical data  reported by facilities, unreliable data may result in VA’s modeled panel  size being too high or too low for certain facilities. Also, if facilities are  using unreliable data to manage their primary care panels—for example,  using the data to assign patients to primary care providers—the facilities  may be misinformed about the available capacity on primary care  providers’ panels—information that is key to determining facilities’ staffing  and other resource needs.

Primary Care Operations Office officials told us that they intend to  address data reliability issues over time. Specifically, the Primary Care  Operations Office is in the process of implementing new software, called  web-PCMM, which officials believe will address some concerns about the  reliability of the data because the software features controls to help  ensure that facilities record and report the data accurately and  consistently. For example, web-PCMM will automatically remove inactive  patients from providers’ panels. In preparation for the implementation of  web-PCMM, Primary Care Operations Office officials said they have been  training network and facility staff on the features and capabilities of the  new software and instructing facility staff to review and correct their panel  size data to help improve data accuracy. It is not yet known the extent to  which the new software will actually address the data reliability issues  because facilities will continue to self-report data. The Primary Care  Operations Office started piloting the new software at selected facilities in  2014 and had planned to implement it agency-wide after resolving  software interoperability issues identified during the pilot. However,  officials said that implementation is currently on hold because of a lack of  funding, and the officials could not provide an updated timeframe for its  system-wide implementation. According to these officials, VA has spent  about $8.8 million through July 2015 on the development and  implementation of web-PCMM and requires an additional $1.5 million to  implement it agency-wide.
subsections: section_title: Wide Variations Exist between Actual Panel Sizes and VA’s Modeled Panel Size at Medical Facilities We Reviewed
paragraphs: Because reliable data on reported panel sizes were not available for all of  VA’s facilities at the time of our review, we calculated actual panel sizes  at six of the seven selected facilities using updated data from these  facilities and correcting for the inaccuracies we found at two facilities.  We compared the actual panel size to each facility’s modeled panel size  for fiscal year 2014. Although Primary Care Operations Office officials  recommend that facilities keep panel sizes 10 to 15 percent below  modeled panel sizes to accommodate growth and provider attrition, we  found that actual panel sizes ranged from 23 percent below to 11 percent  above their respective modeled panel size. This wide variation may  indicate that actual panel sizes at some facilities are too low—potentially  leading to inefficiency and wasted resources—or too high—potentially  leading to veterans experiencing delays in obtaining care, among other  negative effects. It may also indicate that VA’s modeled panel sizes are  determined incorrectly based on unreliable facility data or do not  sufficiently account for patient acuity levels and demand for primary care  services. Actual average panel sizes across the six facilities ranged from  a low of 1,000 patients per provider to a high of 1,338 patients per  provider. (See fig. 2.)

At the three facilities where actual panel sizes were the highest of the six  for which we calculated the actual panel sizes, officials cited three key  factors that contributed to the higher panel sizes.

Growing patient demand: Officials at all three facilities stated that the  growing number of patients seeking primary care services at their  facilities has required them to assign a larger number of patients to  each provider. Officials at one of these facilities stated that not  assigning new patients to a panel would result in a greater number of  walk-in patients seeking emergency care and a loss of continuity of  care.

Staffing shortages: Officials at all three facilities described difficulty  recruiting primary care providers, which resulted in a shortage of  providers. At one of these facilities, about 40 percent of primary care  provider positions were vacant at the time of our review. Officials at all  three facilities attributed recruiting difficulties to the rural location of  these facilities, lack of academic affiliation of the facilities, and the  lower pay that VA offers primary care providers compared to nearby  private sector medical facilities. In addition, at one of these facilities,  officials stated that non-compete clauses limited the facility’s ability to  hire providers currently working in the private sector who might  otherwise seek employment with VA.

Exam room shortages: Officials at two of the three facilities stated that  a lack of available exam room space has limited their ability to hire  additional primary care providers—and thereby reduce panel sizes.  They stated that the process for acquiring additional space—whether  through building additional space or leasing it—is cumbersome and  requires extensive preplanning. For example, at one of these facilities,  officials stated that expanding the facility’s existing exam room space  or opening another CBOC to accommodate growing demand for  primary care typically takes 5 to 6 years. The officials told us that  while the Veterans Access, Choice, and Accountability Act of 2014  provided facilities with funds to acquire additional space, it did not  simplify the process for acquiring space.

Officials at two of the three facilities stated that the higher actual panel  sizes have contributed to provider burnout and attrition. At one facility— where actual panel sizes were 11 percent above the modeled panel  size—officials stated that the facility has been unable to hire enough  providers to make up for attrition. The officials added that providers have  expressed concerns to facility leadership that high panel sizes were  impeding their ability to provide safe and effective patient care. All three  facilities have taken measures to address higher actual panel sizes. For  example, in order to ease staffing shortages the facilities have contracted  with non-VA providers to provide care at VA facilities and have offered  evening and weekend clinic hours to fully utilize available exam room  space. However, while these measures have helped address capacity  shortages at these facilities, they do not fully address the longstanding  concerns resulting from higher panel sizes.

In contrast, at the facility where actual panel size was the lowest of the six  we reviewed—23 percent below its modeled panel size—officials said  they have made a concerted effort to establish lower panel sizes while  increasing the number of primary care providers. Officials stated that they  had recently lowered providers’ panel sizes because they believed that  the modeled panel size did not sufficiently account for factors affecting  patients’ demand for primary care services, such as high acuity levels.  These officials noted that they previously followed the modeled panel size  but found that it was too high and resulted in primary care provider  burnout and poor patient access to primary care providers. Since VA  Central Office and network staff generally do not examine differences  across medical facilities VA-wide, it is unclear whether the facility with  lower panel sizes for providers was providing primary care services in an  inefficient manner or whether VA’s modeled panel size for this facility was  too high.

section_title: VA Central Office and Networks Do Not Have Effective Oversight Processes for Verifying and Using Facilities’ Panel Size Data to Monitor Primary Care
paragraphs: VA Central Office and networks do not have effective oversight processes  for verifying and using facilities’ panel size data to monitor facilities’  management of primary care. VA’s panel management policy requires  facilities to ensure the reliability of their reported panel size data, but the  policy does not assign oversight responsibility to VA Central Office or the  networks for verifying the reliability of these data or for using the data for  monitoring purposes. Federal internal control standards state that  agencies should clearly define key areas of authority and responsibility,  ensure that reliable information is available, and assess the quality of  performance over time.

However, officials from the Primary Care Operations Office told us that— except for a few isolated situations—they do not verify the panel size data  recorded in PCMM to systematically identify unreliable data or to monitor  panel sizes across all VA medical facilities. For example, these officials  told us that in 2014, they conducted reviews of three facilities that were  struggling with recording and reporting reliable data in PCMM to identify  ways to improve the reliability of the facilities reported data. The officials  said they have not validated facilities’ reported panel size data or used  the data to monitor primary care because the office has a limited number  of staff and mainly relies on the networks and facilities to ensure that the  data are recorded and reported correctly and that monitoring is  conducted.

Across the seven networks that oversee the seven selected facilities for  which we conducted a more in-depth analysis, we also identified  variations in the extent to which the networks verified facilities’ panel size  data and used the data to monitor and address panel sizes that were too  high or too low. Specifically,

Data verification: Officials from four of the seven networks told us that  they took some steps to verify that facilities’ panel size data were  reliable, such as reviewing the data for errors and large variations. For  example, officials from one of these networks stated that if they  identified large variability in the number of exam rooms—a relatively  stable data element over time—it could indicate problems with data  reliability, which the network officials would discuss with officials from  the facility reporting the data. Officials from another network stated  that they compared data reported by facilities to data previously  reported by the facilities to identify large variations. Officials from the  remaining three networks told us that they did not any take steps to  verify that facilities’ reported panel size data were reliable. According  to Primary Care Operations Office officials, VA networks can request  access to facilities’ PCMM software, which would enable them to  verify the data; however, the officials acknowledged that many of VA’s  21 networks are unaware of this capability.

Use of data for monitoring primary care: Officials from six of the seven  networks said they discussed reported panel size data during monthly  calls with facility officials, at primary care committee meetings, or  during facility site visits. However, officials from only four of these six  networks stated that they took steps to address panel sizes that are  too high or too low compared to a facility’s respective modeled panel  size. For example, officials at one network told us that they helped a  facility recruit additional primary care providers to address high panel  sizes. In another network, officials said that they were helping a facility  secure additional exam room space to address high panel sizes.  Officials at a third network told us that they recently had to curtail  monitoring activities to address facilities’ panel sizes due to staffing  shortages. In contrast, officials from the one network that does not  use panel size data to monitor facilities’ management of primary care  told us that they rely on the facilities to manage their own primary care  panels and do not believe that the network should take an active role  in this process. As a result, officials from this network were unaware  that a facility within their network had made a concerted effort to  establish panel sizes that were well below its modeled panel size.

Absent a robust oversight process that assigns responsibility, as  appropriate, to VA Central Office and networks for verifying facilities’  panel size data and using the data to monitor facilities’ management of  primary care—such as, examining wide variations from modeled panel  sizes—VA lacks assurance that facilities’ data are reliable and that they  are managing primary care panels in a manner that meets VA’s goals of  providing efficient, timely, and quality care to veterans. Primary Care  Operations Office officials stated that VA Central Office is in the process  of revising its policy on primary care panel management and is  developing additional guidance to require VA Central Office and VA  networks to verify reported panel size data in addition to other monitoring  responsibilities. However, as the revised policy and guidance are still  under development, it is unknown when they will be implemented and  whether they will fully address the issues we identified.

section_title: VA’s Encounter and Expenditure Data Show Wide Variation across Medical Facilities, and VA Central Office and Networks Do Not Use These Data to Monitor Primary Care
subsections: section_title: Expenditures per Primary Care Encounter Varied Widely across VA Facilities
paragraphs: Based on our review of fiscal year 2014 VA-wide primary care  expenditure and encounter data, we found that expenditures per primary  care encounter varied widely across VA facilities, from a low of $150 to a  high of $396, after adjusting to account for geographic differences in labor  costs. Expenditures per encounter at 97 of the 140 facilities we reviewed  were within $51 or one standard deviation—a statistical measure of  variance—of VA’s overall average of $242. According to officials from  VHA’s Office of Finance, one standard deviation is typically used to  identify potential outliers when examining encounter and expenditure  data. For the remaining 43 facilities, our analysis found that expenditures  per encounter at 20 facilities were at least one standard deviation above  the average and at 23 facilities were at least one standard deviation  below, which may indicate potential outliers that VA Central Office and  the networks may need to examine further. (See fig. 3.) Among other  things, this variation may indicate that primary care is being delivered  efficiently at facilities with relatively low expenditures per encounter or  inefficiently at facilities with relatively high expenditures per encounter.  We also analyzed expenditures per unique primary care patient—that is,  a patient with at least one primary care encounter in fiscal year 2014— and found similar variation across VA’s facilities. (See app. I.) We found  that this variation remained when examining expenditures by encounter  and per unique patient for facilities within the same complexity group.

Of the seven selected facilities, one was among the least expensive  facilities across all VA facilities and another was among the most  expensive, in terms of expenditures per primary care encounter. An  official from the facility that was among the least expensive of the seven  we reviewed, with expenditures per encounter of $158, identified an  increased use of secure messaging and telephone primary care as  primary factors that contributed to a lower expenditure per encounter.  Officials from the network that oversees the facility that was among the  most expensive of the seven we reviewed, with expenditures per  encounter of $330, identified the high cost of living in the area—which  resulted in higher leasing and labor costs—as the primary factor that  contributed to a higher than average cost per encounter. However, our  analysis largely accounted for the higher cost of living in that expenditure  data provided by VA were adjusted to account for geographic differences  in labor costs, which made up 71 percent of this facility’s costs in fiscal  year 2014. The officials also explained that part of the reason for the high  expenditures per encounter was that the facility was not appropriately  accounting for telephone-based primary care services it provided for the  entire network. As a result, primary care encounters and expenditures for  the selected facility included encounters and expenditures for telephone  primary care services for other facilities within the network. According to  network officials, steps are being taken to ensure that the facility is  allocating these expenditures appropriately going forward.

section_title: VA Central Office and Networks Verify and Use Facilities’ Encounter and Expenditure Data for Financial Purposes, But Not to Monitor Primary Care
paragraphs: While VA Central Office and networks verify and use facilities’ encounter  and expenditure data for financial purposes, VA’s policies governing  primary care do not require VA Central Office and networks to use these  data to monitor facilities’ management of primary care. Federal internal  control standards state that agencies need both operational and financial  data to determine whether they are meeting strategic goals and should  use such data to assess the quality of performance over time.

We found that the Office of Finance in VA Central Office independently  verifies facilities’ encounter and expenditure data to help ensure their  reliability and uses the data for cost accounting and budgetary  purposes. Similarly, chief financial officers or their designees at six of  the seven networks that oversee the facilities we reviewed routinely  examine encounter and expenditure data to identify outliers for the  purposes of ensuring data reliability and for cost accounting. However,  the Primary Care Operations Office in VA Central Office does not use  encounter and expenditure data, even though officials stated that  examining such data would likely help them monitor facilities’  management of primary care. Furthermore, primary care officials at the  seven networks we examined generally do not use these data to monitor  facilities’ management of primary care. Some officials told us that they do  not use encounter and expenditure data for monitoring primary care  delivery because panel sizes are the most effective means of measuring  efficiency within primary care.

By not using encounter and expenditure data to monitor facilities’  management of primary care, VA may be missing opportunities to identify  facilities—such as those that experience higher than average  expenditures per encounter or significant changes in expenditures over  time—that may warrant further examination and to strengthen the  efficiency and effectiveness of the primary care program. Using panel  size data in conjunction with encounter and expenditure data, would allow  VA Central Office and networks to assess facilities’ capacity to provide  primary care services and the efficiency of care delivery.

section_title: Conclusions
paragraphs: The absence of reliable panel size data and oversight processes could  significantly inhibit VA’s ability to ensure that facilities are providing  veterans with timely, quality care and delivering that care efficiently. While  VA planned to address some of the data reliability issues through new  software to help VA facilities record data more accurately, development of  this software is currently on hold, and VA could not provide any estimates  of when the software would be implemented at its facilities. Even if this  software is implemented, VA Central Office and networks will still be  relying on self-reported data on primary care panel sizes from its facilities.  By not having in place a process to verify the reliability of facilities’ panel  size data or to monitor wide variations between facilities’ reported and  modeled panel sizes, VA will likely continue to receive unreliable data and  miss opportunities to assess the impact of panel sizes on veterans’  access to care. VA Central Office and the networks are also missing  opportunities to use readily available encounter and expenditure data to  potentially improve the efficiency of primary care service delivery.  Consistent with federal internal control standards, using such data in  conjunction with reliable panel size data could be a potent tool in “right- sizing” panel sizes to best serve veterans’ needs and deliver primary care  efficiently.

section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of the Department of Veterans Affairs,  direct the Undersecretary for Health to take the following two actions to  improve the reliability of VA’s primary care panel size data and improve  VA Central Office and the networks’ oversight of facilities’ management of  primary care:  Incorporate in policy an oversight process for primary care panel  management that assigns responsibility, as appropriate, to VA Central  Office and networks for (1) verifying each facility’s reported panel size  data currently in PCMM and in web-PCMM, if the software is rolled- out nationally, including such data as the number of primary care  patients, providers, support staff, and exam rooms; and (2) monitoring  facilities’ reported panel sizes in relation to the modeled panel size  and assisting facilities in taking steps to address situations where  reported panel sizes vary widely from modeled panel sizes.

Review and document how to use encounter and expenditure data in  conjunction with panel size data to strengthen monitoring of facilities’  management of primary care.

section_title: Agency Comments and Our Evaluation
paragraphs: VA provided written comments on a draft of this report, which we have  reprinted in appendix II. In its comments, VA agreed with our conclusions,  concurred with our two recommendations, and described the agency’s  plans to implement our recommendations. VA also provided technical  clarifications and comments on the draft report, including the  recommendations contained in the draft report. We incorporated these  comments, as appropriate. In particular, we modified our first  recommendation in the draft report and now recommend that VA verify  each facility’s panel size data in PCMM and, if the latter is available, in  web-PCMM. We made this change to reflect the continued uncertainty  over the implementation of the web-PCMM software. In addition, we  modified our second recommendation in the draft report and no longer  recommend VA incorporate into existing VA policy a requirement that the  agency and its networks use encounter and expenditure data to  strengthen the monitoring of facilities’ management of primary care. We  made this change to reflect that VA officials were not prepared to  incorporate such a requirement without first examining how to use these  data for monitoring purposes.

To address our first recommendation, VA stated that it plans to issue  guidance by September 2016 clarifying VA Central Office’s and the  networks’ oversight responsibilities with regard to primary care panel size  data. This guidance will include a process—developed by the Offices of  Primary Care Services and Primary Care Operations—for addressing  medical facilities whose panel sizes differ significantly from similar  facilities’ panels. In its response, however, VA did not provide information  on how it plans to address unreliable panel size data facilities record and  report in PCMM. We would encourage VA, in the guidance it plans to  issue in 2016, to assign responsibility for verifying each facility's reported  panel size data as we recommended. To address our second  recommendation, VA stated that it will take steps to understand encounter  and expenditure data and determine how best to utilize these data to  improve patient care with a target completion date for presenting its  findings and decisions by September 2018.

As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 14 days from the  report date. At that time, we will send copies to the appropriate  congressional committees and the Secretary of Veterans Affairs. In  addition, the report will be available at no charge on the GAO website at  http://www.gao.gov.

If you or your staffs have any questions about this report, please contact  me at (202) 512-7114 or williamsonr@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs are on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix III.

section_title: Appendix I: Expenditures per Unique Primary Care Patient in Fiscal Year 2014
paragraphs: We analyzed Department of Veterans Affairs (VA) fiscal year 2014 data  on primary care expenditures and calculated expenditures per unique  primary care patient. We found that expenditures per unique primary  care patient varied widely across facilities in fiscal year 2014, ranging  from $558 to $1,544 after adjusting to account for geographic differences  in labor costs across facilities. We found that the expenditures per  unique patient at 102 of the 140 facilities we reviewed were within $167 or  one standard deviation—a statistical measure of variance—of VA’s  overall average of $871. For the remaining facilities, expenditures per  unique patient were at least one standard deviation above the average  (19 facilities) or were at least one standard deviation below the average  (19 facilities), which may indicate potential outliers that VA Central Office  and the networks may need to examine further. (See fig. 4.)

section_title: Appendix II: Comments from the Department of Veterans Affairs

section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Rashmi Agarwal, Assistant  Director; James Musselwhite, Assistant Director; Kathryn Black;   Krister Friday; Cathleen Hamann; Aaron Holling; Emily Wilson; and  Michael Zose made key contributions to this report.

section_title: Related GAO Products
paragraphs: Department of Veterans Affairs: Expanded Access to Non-VA Care  Through the Veterans Choice Program. GAO-15-229R. Washington,  D.C.: Nov 19, 2014.

VA Health Care: Actions Needed to Ensure Adequate and Qualified  Nurse Staffing. GAO-15-61. Washington, D.C.: Oct 16, 2014.

VA Health Care: Ongoing and Past Work Identified Access, Oversight,  and Data Problems That Hinder Veterans’ Ability to Obtain Timely  Outpatient Medical Care. GAO-14-679T. Washington, DC: Jun 9, 2014.

VA Health Care: VA Lacks Accurate Information about Outpatient   Medical Appointment Wait Times, Including Specialty Care Consults.   GAO-14-620T. Washington, D.C.: May 15, 2014.

VA Health Care: Ongoing and Past Work Identified Access Problems  That May Delay Needed Medical Care for Veterans. GAO-14-509T.  Washington, D.C.: Apr 9, 2014.


Question-summary annotations: questions: question: What did GAO find regarding the VA's data on primary care panel sizes?
answer: GAO found that the Department of Veterans Affairs' (VA) data on primary care panel sizes—that is, the number of patients VA providers and support staff are assigned as part of their patient portfolio—are unreliable across VA's 150 medical facilities and cannot be used to monitor facilities' management of primary care. Specifically, as part of its review, GAO found missing values and other inaccuracies in VA's data.
child_questions: question: How did VA officials respond to these findings?
answer: Officials from VA's Primary Care Operations Office confirmed that facilities sometimes record and self-report these data inaccurately or in a manner that does not follow VA's policy and noted that this could result in the data reliability concerns GAO identified.

question: How did GAO process collected data from primary care facilities?
answer: GAO obtained updated data from six of seven selected facilities, corrected these data for inaccuracies, and then calculated the actual panel sizes for the six facilities.
child_questions: question: What did this data reveal?
answer: GAO found that for these six facilities the actual panel size varied from 23 percent below to 11 percent above the modeled panel size, which is the number of patients for whom a provider and support staff can reasonably deliver primary care as projected by VA.
child_questions: question: What concerns does this data raise?
answer: Such wide variation raises questions about whether veterans are receiving access to timely care and the appropriateness of the size of provider workload at these facilities.

questions: question: What issues did GAO find with the VA's data management?
answer: Moreover, GAO found that while VA's primary care panel management policy requires facilities to ensure the reliability of their panel size data, it does not assign responsibility to VA Central Office or networks for verifying the reliability of facilities' data or require them to use the data for monitoring purposes.
child_questions: question: How does this compare to federal internal quality control standards?
answer: Federal internal control standards call for agencies to clearly define key areas of authority and responsibility, ensure that reliable information is available, and use this information to assess the quality of performance over time.

question: How does not meeting the federal control standards negatively affect the VA?
answer: Because VA's panel management policy is inconsistent with federal internal control standards, VA lacks assurance that its facilities' data are reliable and that the facilities are managing primary care panels in a manner that meets VA's goals of providing efficient, timely, and quality care to veterans.

questions: question: What did GAO find about VA's primary care encounter and expenditure data in contrast to panel data?
answer: In contrast to VA's panel data, GAO found that primary care encounter and expenditure data reported by all VA medical facilities are reliable, although the data show wide variations across facilities. For example, in fiscal year 2014, expenditures per primary care encounter—that is, a professional contact between a patient and a primary care provider—ranged from a low of $150 to a high of $396 after adjusting to account for geographic differences in labor costs across facilities.
child_questions: question: What might these wide variations indicate?
answer: Such wide variations may indicate that services are being delivered inefficiently at some facilities with relatively higher per encounter costs compared to other facilities.

question: How does the VA use panel size data?
answer: However, while VA verifies and uses these data for financial purposes, VA's policies governing primary care do not require the use of the data to monitor facilities' management of primary care.
child_questions: question: How does this differ from federal internal control standards?
answer: Federal internal control standards state that agencies need both operational and financial data to determine whether they are meeting strategic goals and should use such data to assess the quality of performance over time.

question: How could using panel size data in conjunction with encounter and expenditure data benefit the VA?
answer: Using panel size data in conjunction with encounter and expenditure data would allow VA to assess facilities' capacity to provide primary care services and the efficiency of their care delivery.

question: How does not using panel size data negatively affect the VA?
answer: By not using available encounter and expenditure data in this manner, VA is missing an opportunity to potentially improve the efficiency of primary care service delivery.

questions: question: What was GAO asked to examine?
answer: GAO was asked to examine these data and VA's oversight of primary care.
child_questions: question: What this report address?
answer: This report examines (1) VA's panel size data across facilities and how VA uses these data to oversee primary care, and (2) VA's encounter and expenditure data across facilities and how VA uses these data to oversee primary care.

question: How did GAO collect data for this report?
answer: GAO analyzed fiscal year 2014 data on primary care panel size, encounters, and expenditures for all VA facilities. GAO also conducted a more in-depth, nongeneralizable analysis of data and interviewed officials from seven facilities, selected based on geographic diversity and differences in facility complexity. GAO also interviewed VA Central Office and network officials to examine their oversight of primary care, including the extent to which they verify the data and use it to monitor the management of primary care.