Veterans Health
 

Administration
 Review of Veterans’ Access 

to Mental Health Care 

OFFICES OF AUDITS AND EVALUATIONS

AND HEALTHCARE INSPECTIONS

April 23, 2012 

12-00900-168 
Review of Veterans’ Access to Mental Health Care 

ACRONYMS AND ABBREVIATIONS 

ARC Allocation Resource Center 

CBOC Community Based Outpatient Clinic 

CPT Current Procedural Terminology 

DSS Decision Support System 

Dx Diagnosis 

EBT Evidence Based Psychotherapy 

FTE Full-Time Employee Equivalent 

IOP Intensive Outpatient Substance Use Treatment Program 

Mgmt Management 

MH Mental Health 

OEF Operation Enduring Freedom 

OIF Operation Iraqi Freedom 

OIG Office of Inspector General 

OPES Office of Productivity, Efficiency, and Staffing 

PAR Performance and Accountability Report 

PCT Post-Traumatic Stress Disorder Clinical Team 

PTSD Post-Traumatic Stress Disorder 

RVU Relative Value Units 

SUD Substance Use Disorder 

Tx Treatment 

VHA Veterans Health Administration 

VISN Veterans Integrated Service Network 

VistA Veterans Health Information Systems and Technology Architecture 

VSSC Veterans Health Administration Support Service Center 

WSDTT Women’s Stress Disorders Treatment Team 

WVRU Work Relative Value Units 

To Report Suspected Wrongdoing in VA Programs and Operations:
 

Telephone: 1-800-488-8244
 

E-Mail: vaoighotline@va.gov
 

(Hotline Information: http://www.va.gov/oig/contacts/hotline.asp)
 

VA Office of Inspector General 
Review of Veterans’ Access to Mental Health Care 

TABLE OF CONTENTS
 

Page 

Executive Summary............................................................................................................ i
 

Introduction ....................................................................................................................... 1
 

Purpose............................................................................................................................. 1
 

Background...................................................................................................................... 1
 

Results and Conclusions.................................................................................................... 2
 

Issue 1 – VHA’s Mental Health Performance Data Is Not Accurate or Reliable ........... 2
 

Issue 2 - VHA Measures Do Not Fully Reflect Critical Dimensions of
 

Mental Health Care Access ............................................................................ 10
 

Appendixes ....................................................................................................................... 36
 

A. Scope and Methodology .......................................................................................... 36
 

B. Statistical Sampling Methodology........................................................................... 38
 

C. Previous OIG Reports Highlighting Need for Improved Appointment Scheduling
 

and Peformance Measures ....................................................................................... 43
 

D. Detailed Legend for Figures .................................................................................... 45
 

E. Under Secretary for Health Comments.................................................................... 47
 

F. OIG Contact and Staff Acknowledgments .............................................................. 53
 

G. Report Distribution .................................................................................................. 54
 

VA Office of Inspector General 
Review of Veterans’ Access to Mental Health Care 

Executive Summary
 

Results in Brief 

The Office of Inspector General (OIG) conducted this review at the request of the 

Chairman and Ranking Member, U.S. Senate Committee on Veterans’ Affairs, the 

Secretary of Veterans’ Affairs, the Chairman and Ranking Member of the U.S. House 

Veterans’ Affairs Committee and the Chairwoman and Ranking Member of the 

Subcommittee on Health, House Committee on Veterans’ Affairs, after they expressed 

concerns that veterans may not be able to access the mental health care they need in a 

timely manner. The request asked the OIG to determine how accurately the Veterans 

Health Administration (VHA) records wait times for mental health services for both 

initial (new patients) and follow-up (established patients) visits and if the wait time data 

VA collects is an accurate depiction of the veteran’s ability to access those services. 

Background 

VHA policy requires all first-time patients referred to or requesting mental health 

services receive an initial evaluation within 24 hours and a more comprehensive 

diagnostic and treatment planning evaluation within 14 days. The primary goal of the 

initial 24-hour evaluation is to identify patients with urgent care needs and to trigger 

hospitalization or the immediate initiation of outpatient care when needed. Primary care 

mental health providers, other referring licensed independent providers, or licensed 

independent mental health providers can conduct the initial 24-hour evaluation. 

One method VHA uses to monitor access to mental health services is to calculate 

patients’ waiting times by measuring the elapsed days from the desired dates of care to 

the dates of the treatment appointments. Medical facility schedulers must enter the 

correct desired dates of care in the system to ensure the accuracy of this measurement. 

VHA’s goal is to see patients within 14 days of the desired dates of care. 

VHA’s Mental Health Performance Data Is Not Accurate or Reliable 

VHA does not have a reliable and accurate method of determining whether they are 

providing patients timely access to mental health care services. VHA did not provide 

first-time patients with timely mental health evaluations and existing patients often 

waited more than 14 days past their desired date of care for their treatment appointment. 

VHA’s Measurement of a First-Time Patient’s Access to a Full Mental Health Evaluation 

was not a Meaningful Measure of Waiting Time 

In VA’s FY 2011 Performance and Accountability Report (PAR), VHA reported 

95 percent of first-time patients received a full mental health evaluation within 14 days. 

However, this measure had no real value as VHA measured how long it took VHA to 

VA Office of Inspector General i 
Review of Veterans’ Access to Mental Health Care 

conduct the evaluation, not how long the patient waited to receive an evaluation. For 

example, if a patient’s primary care provider referred the patient to mental health service 

on September 15 and the medical facility scheduled and completed the evaluation on 

October 1, VHA’s data showed the veteran waited 0-days for their evaluation. In reality, 

the veteran waited 15 days for their evaluation. 

VHA’s measurement differed from the defined objective of the measure that stated 

veterans should have further evaluation and initiation of mental health care in 14 days of 

a trigger encounter. VHA defined the trigger encounter as the veterans contact with the 

mental health clinic or the veteran’s referral to the mental health service from another 

provider. VHA needs to redefine their measurement to ensure it meets the intent of the 

stated objective that is to make sure veterans receive a full mental health evaluation 

within 14 days. 

VHA was not Providing All First-Time Patients a Full Mental Health Evaluation Within 

14 Days 

Using the same data VHA used to calculate the 95 percent success rate shown in the 

FY 2011 PAR, we selected a statistical sample of completed evaluations to determine the 

starting and ending points of the elapsed day calculation. We conducted an independent 

assessment by reviewing patient records to review patients’ visit dates, clinical notes, and 

consult records to identify the exact date of the trigger encounter (the date the patient 

initially contacted mental health seeking services, or when another provider referred the 

patient to mental health). We determined when the full evaluation containing a patient 

history, diagnosis, and treatment plan was completed. Based on our analysis of that 

information, we calculated the number of days between initial contact in mental health 

and the full mental health evaluation. Our analysis projected that VHA provided only 

49 percent (approximately 184,000) of their evaluations within 14 days. On average, for 

the remaining patients, it took VHA about 50 days to provide them with their full 

evaluations. 

VHA Overstated Its Success in Providing Veterans New and Follow-Up Appointments 

for Treatment Within 14 Days 

VHA does not consider the full mental health evaluation as an appointment for treatment, 

but rather the evaluation is the prerequisite for VHA to develop a patient-appropriate 

treatment plan. Once VHA provides the patient with a full mental health evaluation, 

VHA schedules the patient for an appointment to begin treatment. We found that VHA 

did not always provide both new and established patients their appointments within 

14 days of the patients’ desired date—VHA’s goal for timely patient access to care. 

VHA defines the desired date as the date on which the patient or the provider wants the 

patient to be seen without regard to schedule capacity. We reviewed patient records to 

identify the desired date (generally located in the physician’s note as the date the patient 

needed to return to the clinic or shown as a referral from another provider) and calculated 

VA Office of Inspector General ii 
Review of Veterans’ Access to Mental Health Care 

the elapsed days to the date of the patient’s completed treatment appointment date. We 

projected nationwide that in FY 2011, VHA: 

	 Completed approximately 168,000 (64 percent) new patient appointments for 

treatment within 14 days of their desired date; thus, approximately 

94,000 (36 percent) appointments nationwide exceeded 14 days. VHA data showed 

that 95 percent received timely care. 

	 Completed approximately 8.8 million (88 percent) follow-up appointments for 

treatment within 14 days of the desired date; thus, approximately 1.2 million 

(12 percent) appointments nationwide exceeded 14 days. Although we based our 

analysis on dates documented in VHA’s medical records, we have less confidence in 

the integrity of this date because providers at three of the four medical centers we 

visited told us they requested a desired date of care based on their schedule 

availability. VHA data showed that 98 percent received timely care for treatment. 

VHA Schedulers did not Consistently Follow Procedures 

VHA schedulers were not following procedures outlined in VHA Directives, and, as a 

result, VHA’s reported waiting time data was not accurate or reliable. For new patients, 

the scheduling clerks frequently stated they used the next available appointment slot as 

the desired appointment date for new patients. For established patients, medical 

providers told us they frequently scheduled the return to clinic appointments based on 

their known availability rather than the patient’s clinical need. For example, providers 

may not have availability for 2–3 months, so they specify that as the return to clinic time 

frame. 

OIG previously reported concerns with VHA’s calculated wait time data in our Audit of 

VHA’s Outpatient Scheduling Procedures, Report No. 04-02887-169, July 8, 2005 and 

Audit of VHA’s Outpatient Wait Times, Report No. 07-00616-199, September 10, 2007. 

During both audits, OIG found that schedulers were entering an incorrect desired date. 

Given VHA’s inability to correct this long-standing problem, VHA should reassess their 

training, competency, and oversight methods and develop appropriate controls to collect 

reliable and accurate appointment data. 

Mental Health Staff Vacancies May Be Affecting VHA’s Ability to Meet Timeliness 

Goals 

According to VHA, from 2005 to 2010, mental health services increased their staff by 

46 percent and treated 39 percent more patients. Despite the increase in mental health 

care providers, VHA’s mental health care service staff still did not believe they had 

enough staff to handle the increased workload and consistently see patients within 

14 days of the desired date. 

VA Office of Inspector General iii 
Review of Veterans’ Access to Mental Health Care 

In July 2011, the U.S. Senate Committee on Veterans’ Affairs requested VA to conduct a 

survey that among other questions asked mental health professionals whether their 

medical center had adequate mental health staff to meet current veteran demands for care; 

71 percent responded their medical center did not have adequate numbers of mental 

health staff. Based on our interviews at four medical centers, staff in charge of mental 

health services reported VHA’s greatest challenge has been to hire and retain 

psychiatrists. Three of the four sites we visited had vacant psychiatry positions. We 

determined that a patient at the Salisbury VA Medical Center had to wait, on average, 

86 days to see a psychiatrist. This was based on an analysis of VHA’s data that identifies 

a provider’s third next available appointment date for treatment. Staff at that facility told 

us they were still trying to replace three psychiatrists who left to go to a private practice 

in the past year. A comprehensive staffing analysis can help VHA determine if 

psychiatrist, or other mental health provider, vacancies are systemic issues impeding 

VHA’s ability to meet mental health timeliness goals. 

Measuring Access to VHA Mental Health Care 

The data and measures needed by decision makers for effective planning and service 

provision may differ at the national, Veterans Integrated Service Network, and facility 

level. No measure of access is perfect or paints a complete picture in isolation. 

Meaningful analysis and decision making requires reliable data, on not only the 

timeliness of access but also on trends in demand for mental health services, treatments, 

and providers; the availability and mix of mental health staffing; provider productivity; 

and treatment capacity. These demand and supply variables in turn feedback upon a 

system’s ability to provide treatment that is patient centered and timely. 

VHA’s 14-day follow-up measure provides decision makers with a limited picture of a 

new patient’s ability to access and begin mental health treatment. Additionally, 

depending on a veteran’s point of access, this metric does not truly measure VHA’s 

stated objective “to ensure timely access for all veterans who are new to mental health.” 

A series of timeliness and treatment engagement measures might provide decision 

makers with a more comprehensive view of the ability with which new patients can 

access mental health treatment. Furthermore, although VHA collects and reports mental 

health staffing and productivity data, the complexity of the computations and 

inaccuracies in some of the data sources, limits the usability of productivity information 

to fully assess current capacity, determine optimal resource distribution, evaluate 

productivity across the system, and establish mental health staffing and productivity 

standards. 

Private sector entities with whom we spoke reported that their managers use multiple 

measures to assess a range of access parameters. These entities disseminate their 

dashboard reports to all levels of management thereby facilitating timely response to 

changing access dynamics. Beyond measures of timeliness (or delay) to mental health 

care, a dashboard of user friendly measures that incorporate aspects of patient demand, 

VA Office of Inspector General iv 
Review of Veterans’ Access to Mental Health Care 

provider supply, clinic capacity, and provider productivity, anchored by a consistent set 

of business rules, might provide VHA decision makers with a more robust perspective 

from which to assess and timely respond to changes in access parameters. 

Conclusion 

VHA does not have a reliable and accurate method of determining whether they are 

providing patients timely access to mental health care services. VHA did not provide 

first-time patients with timely mental health evaluations and existing patients often 

waited more than 14 days past their desired date of care for their treatment appointment. 

As a result, performance measures used to report patient’s access to mental health care do 

not depict the true picture of a patient’s waiting time to see a mental health provider. 

Although no measure of access is perfect or paints a complete picture in isolation, 

meaningful analysis and decision making requires reliable data, on not only the 

timeliness of access but also on trends in demand for mental health services, treatments, 

and providers; the availability and mix of mental health staffing; provider productivity; 

and treatment capacity. A series of timeliness and treatment engagement measures might 

provide decision makers with a more comprehensive view of the ability with which new 

patients can access mental health treatment. 

Recommendations 

1.	 We recommend the Under Secretary for Health revise the current full mental health 

evaluation measurement to ensure the measurement is calculated to reflect the 

veteran’s wait time experience upon contact with the mental health clinic or the 

veteran’s referral to the mental health service from another provider to the 

completion of the evaluation. 

2.	 We recommend the Under Secretary for Health reevaluate alternative measures or 

combinations of measures that could effectively and accurately reflect the patient 

experience of access to mental health appointments. 

3.	 We recommend the Under Secretary for Health conduct a staffing analysis to 

determine if mental health staff vacancies represent a systemic issue impeding the 

Veterans Health Administration’s ability to meet mental health timeliness goals, and 

if so, develop an action plan to correct the impediments. 

4.	 We recommend the Under Secretary for Health ensure that data collection efforts 

related to mental health access are aligned with the operational needs of relevant 

decision makers throughout the organization. 

VA Office of Inspector General v 
th

concurred with the OIG’s findings and recommendations

y committed to providing Veterans the best care possible.

would act rapidly on all findings that may improve

care

onsider the planned actions acceptable and will follow up

Y

l

ental Health Care

H

The Under Secretary for ommendations

and stated VHA is une care possible.

may improve

Veterans’ access to mental of the Under

We will follow up

LINDA A. HALLI

Assistant Inspector Ge

Audits and Evaluat

M.D.

neral for

ns

Access to

The Under Secretary for Health concurred with the OIG’s findings and

and stated VHA is unequivocally committed to providing Veterans the

VHA would act rapidly on all findings

for the full

We consider the planned actions acceptable

JOHN D. DAIGH,

Assistant Inspector

In

Review of Veterans’ Access to Mental Health Care 

Under Secretary for Health Comments 

The Under Secretary for Health concurred with the OIG’s findings and recommendations 

and stated VHA is unequivocall quivocally committed to providing Veterans the best care possible. 

The Under Secretary stated VHA would act rapidly on all findings that may improve 

Veterans’ access to mental health care. (See Appendix E for the full text of the Under 

Secretary’s comments.) We consider the planned actions acceptable and will follow up 

on the implementation. 

LINDA A. HALLIDAY JOHN D. DAIGH, JR., M.D. 

Assistant Inspector General for Assistant Inspector General for 

Audits and Evaluations Healthcare Inspectio spections 

VA Office of Inspector General vi 
Review of Veterans’ Access to Mental Health Care 

Introduction
 

Purpose 

The Chairman and Ranking Member of the U.S. Senate Committee on Veterans’ Affairs, 

the Secretary of Veterans’ Affairs, the Chairman and Ranking Member of the U.S. House 

Veterans’ Affairs Committee and the Chairwoman and Ranking Member of the 

Subcommittee on Health, House Committee on Veteran’s Affairs requested this review of 

veterans’ access to mental health care. Specifically, they requested the Office of 

Inspector General (OIG) determine how accurately the Veterans Health Administration 

(VHA) records wait times for mental health (MH) for both initial (new patients) and 

follow-up (established patients) visits and if the wait time data VHA collects provides an 

accurate depiction of the veteran’s ability to access those services. 

Background 

VHA policy requires all new patients referred to or requesting mental health services 

receive an initial evaluation within 24 hours, and a more comprehensive diagnostic and 

treatment planning evaluation within 14 days. VHA’s goal is to see patients within 

14 days of the patient’s desired date of care. 

To ensure reliable waiting times, schedulers must input the correct desired date of care 

when creating appointments for treatment. According to VHA Directive 2010-027: 

	 For new patients, the patient defines the desired date without regard to schedule 

capacity. Once the desired date is established, schedulers must not alter the date to 

reflect an appointment date the patient acquiesces to accept for lack of appointment 

availability on the desired date. 

	 For established patients, the provider or scheduler will communicate a specific or a 

general time frame and the patient establishes the actual desired date. The scheduler 

is to offer and schedule an appointment on or as close to the desired date as possible. 

If there is a discrepancy between the patient and provider desired date, the scheduler 

must contact the provider for a decision on the return appointment time frame. 

VHA defines new patients as those who have not received care in a defined stop code or 

stop code group at that facility within the past 2 years; established patients represent all 

others. VHA measures the elapsed days from the desired dates of care to the dates of the 

appointments. Schedulers must enter the correct desired dates of care in the system to 

ensure the accuracy of this measurement. 

VA Office of Inspector General 1 
Review of Veterans’ Access to Mental Health Care 

Results and Conclusions
 

Issue 1: VHA’s Mental Health Performance Data Is Not 

Accurate or Reliable 

VHA’s measurement of a first-time patient’s access to a full mental health evaluation was 

not a meaningful measure to assess the time a veterans waits for a mental health 

appointment and VHA’s performance data for mental health treatment appointments is 

not accurate or reliable. Our assessment showed VHA did not provide first-time patients 

with timely mental health evaluations and existing patients often waited more than 

14 days past their desired dates of care for their treatment appointments. Based on our 

review, this occurred because VHA did not have an appropriate method in place to 

evaluate how well they were providing first-time patients a timely full mental health 

evaluation. Scheduling staff not following prescribed procedures, staffing shortages, and 

increasing workloads also contributed to inaccurate and unreliable data as well as delays 

in mental health care. As a result, VHA’s leadership and decision makers’ ability to 

make informed decisions for improving timely access to mental health care is 

constrained. 

VHA’s Mental Health Performance Data Is Not Accurate or Reliable 

VHA does not have a reliable and accurate method of determining whether they are 

providing patients timely access to mental health care services. VHA did not provide 

first-time patients with timely mental health evaluations and existing patients often 

waited more than 14 days past their desired date of care for their treatment appointment. 

VHA’s Measurement of a First-Time Patient’s Access to a Full Mental Health Evaluation 

was not a Meaningful Measure of Waiting Time 

In VA’s FY 2011 Performance and Accountability Report (PAR), VHA reported 

95 percent of first-time patients received a full mental health evaluation within 14 days. 

However, this measure had no real value as VHA measured how long it took VHA to 

conduct the evaluation, not how long the patient waited to receive an evaluation. For 

example, if a patient’s primary care provider referred the patient to mental health service 

on September 15 and the medical facility scheduled and completed the evaluation on 

October 1, VHA’s data showed the veteran waited 0-days for their evaluation. In reality, 

the veteran waited 15 days for their evaluation. 

VHA’s measurement differed from the defined objective of the measure that stated 

veterans should have further evaluation and initiation of mental health care in 14 days of 

a trigger encounter. VHA Directive 2010-027 requires that all patients seeing a VA 

mental health provider for the first time receive a full evaluation within 14 days of either 

VA Office of Inspector General 2 
Review of Veterans’ Access to Mental Health Care 

the patient’s referral to mental health or the patient’s self-contact seeking mental health 

services. According to VHA senior officials, a full evaluation consists of patient history, 

diagnosis, and treatment plan. According to VHA criteria, this measures the ability of 

VHA to ensure veterans new to mental health services will have initial mental health care 

and further evaluation in less than 15 days of either the patient’s: 

	 Walk-in to the clinic or directly accessing mental health clinic staff through other 

means, such as telephone (referred to as a trigger encounter). 

	 Referral to mental health service from either the primary care provider or other 

specialty care provider. 

VHA needs to redefine their measurement to ensure it meets the intent of the stated 

objective that is to make sure veterans receive a full mental health evaluation within 

14 days. 

OIG’s Assessment of How Long Patients Waited to Receive a Full Mental Health 

Evaluation 

Using the same data VHA used to calculate the 95 percent success rate shown in the 

FY 2011 PAR, we selected a statistical sample of completed evaluations to determine the 

starting and ending points of the elapsed day calculation. Specifically, we reviewed 

patient records and used information in VHA’s Compensation and Pension Records 

Interchange to review patients’ visit dates, clinical notes, and consult records to identify 

when the patient initially contacted mental health seeking services or when another 

provider referred the patient to mental health. We determined if the full evaluation 

contained patient history, diagnosis, and treatment plan. Based on that information, we 

calculated the number of days between initial contact in mental health and the full mental 

health evaluation. Our analysis projected that VHA provided only 49 percent 

(approximately 184,000) of first-time patients their evaluation within 14 days. On 

average, for the remaining patients, it took VHA about 50 days to provide them with their 

full evaluations. 

OIG’s Site Visits to Four VHA Medical Centers 

We visited four medical centers to evaluate their processes of scheduling first-time 

patients for mental health care. All four medical centers had a process in place to 

conduct an initial assessment within 24 hours to comply with VHA directives and ensure 

the patient was not in immediate risk. At that point, the scheduling processes varied, as 

highlighted in the procedures identified below. 

	 The Spokane VA Medical Center established an intake clinic staffed by multiple 

providers consisting of social workers and psychologists. The clinic contacted 

VA Office of Inspector General 3 
Review of Veterans’ Access to Mental Health Care 

patients within 24 hours and scheduled them for a full evaluation within 14 days. 

Clinic staff uses the 24-hour contact process to assess whether the veteran is in an 

immediate danger. If there is an emergency, the inpatient psychiatrists handle the 

intervention with the patient. Following the full evaluation, an intake committee, 

consisting of social workers, psychologists, and psychiatrists, meets every Tuesday 

and Thursday to discuss if the recommended treatment plan is appropriate. Because 

psychiatrists are in short supply, this process appears to make the best use of staff. 

	 The Salisbury VA Medical Center used a telephone triage system to contact the 

patient and assess the urgency of their condition. This triage (Current Procedural 

Terminology (CPT) code 98968) qualifies as a full evaluation because the triage 

performed collects a brief history, diagnosis, and treatment plan to follow up with a 

provider in the future. However, in discussions with triage staff, we found that this 

procedure consisted of talking to the veteran for at least 21 minutes using a template 

with a list of questions. The telephone triage staff would schedule the patient for their 

intake appointment, and this was usually at least 3 months in the future. The 

telephone triage staff stated that although VHA considers this effort a full evaluation 

and records it as such under CPT code 98968. They agreed that in 21 minutes they 

could only provide a minimum evaluation. 

VHA Overstated Its Performance Outcomes to Provide Veterans New and 

Follow-Up Appointments for Treatment Within 14 Days 

Once VHA provides the patient with a full mental health evaluation, VHA schedules the 

patient for an appointment to begin treatment. VHA does not consider the full mental 

health evaluation as an appointment for treatment, but rather the evaluation is the 

prerequisite for VHA to develop a patient-appropriate treatment plan. VHA did not 

always provide patients their appointments within 14 days of the patients’ desired dates— 

VHA’s goal for timely patient access to care. Desired date is the date on which the 

patient or the provider wants the patient to be seen without regard to schedule capacity. 

OIG’s Assessment of How Long Patients Waited for their Mental Health Care Treatment 

Appointment 

For treatment appointments of patients new to a specific mental health clinic, as 

prescribed by VHA Directives and training modules, we used the date a provider referred 

the patient to the clinic as the desired date. For example, a physician treating a patient in 

a substance abuse clinic determines the patient needs treatment for depression and on 

July 1 refers the patient to an appropriate clinic specializing in treating depression. We 

used July 1 as the desired date. When we could not identify a consult referral date, we 

used the appointment creation date as the desired date. In FY 2011, we projected 

nationwide that VHA completed approximately 168,000 (64 percent) new patient 

appointments within 14 days of their desired date; thus, approximately 

VA Office of Inspector General 4 
Review of Veterans’ Access to Mental Health Care 

94,000 (36 percent) appointments nationwide exceeded 14 days. VHA data showed that 

95 percent received timely care. 

For follow-up treatment appointments, we used the date the provider requested for the 

patient’s return to the clinic. Although we based our analysis on dates documented in 

VHA’s medical records, we have less confidence in the integrity of this date, because 

providers at three of the four medical centers we visited told us they requested a desired 

date of care based on their schedule availability. For example, if they knew they could 

not see the patient for another 2–3 months, they scheduled the patient’s return to clinic in 

that time frame to align with their availability. VHA completed approximately 

8.8 million (88 percent) follow-up appointments within 14 days of the desired date; thus, 

almost 1.2 million (12 percent) appointments nationwide exceeded 14 days. VHA data 

showed that 98 percent received timely care. 

VHA’s National Access List 

VHA uses the national access list to monitor access and identify delays in mental health 

care. As of January 2012, VHA’s national access list, which included both first-time 

patients and patients receiving follow-up appointments for treatment, reported about 

15,000 mental health care appointments that exceeded VHA’s standard of scheduling 

appointments within 14 days of their desired date. Approximately 12,600 (84 percent) 

appointments were scheduled from 15 to 60 days, and approximately 2,400 (16 percent) 

of the appointments were scheduled more than 60 days, from the patient’s desired 

appointment date. We did not validate the completeness or accuracy of this list. 

Next Available Appointment 

VHA captures data to determine the number of days until the third next available 

treatment appointment for specific clinics. Calculating the wait time to the third next 

available appointment is a common practice for assessing a provider’s ability to see 

patients in a timely manner. According to VHA data, the number of completed 

appointments in mental health clinics during FY 2011 was about 10.3 million with 

7.3 million (71 percent) of these showing the third next available appointment was within 

14 days. In other words, in FY 2011, VHA generally had the capacity to schedule 

71 percent of their patients within 14 days. 

Determining the wait time to the third next available appointment is not an absolute 

measure of VHA’s ability to provide timely access (an appointment within 14 days) 

because there are two available appointment times prior to the third next available 

appointment that clinics can use to schedule patients. However, in July 2011, the 

U.S. Senate Committee on Veterans’ Affairs requested VA to conduct a survey that 

among other questions asked mental health professionals whether they could schedule a 

patient appointment within 14 days in their own mental health clinics. Of the 

VA Office of Inspector General 5 
Review of Veterans’ Access to Mental Health Care 

respondents, 63 percent said they could schedule a new patient appointment, and 

61 percent said they could schedule an established patient appointment within 14 days. 

The survey appears to corroborate VHA’s data of the third next available appointment. 

Scheduling Procedures Were Not Followed 

VHA schedulers were not following procedures outlined in VHA Directives and, as a 

result, VHA’s reported waiting time data was not accurate or reliable. For new patients, 

the scheduling clerks frequently stated they used the next available appointment slot as 

the desired appointment date for new patients. Even though a consult referral, or contact 

from the veteran requesting care, may have been submitted weeks or months earlier than 

the patient’s appointment date, the desired appointment date was determined by and 

recorded as the next available appointment date. For established patients, medical 

providers told us they frequently scheduled the return to clinic date based on their known 

availability rather than the patient’s clinical need. For example, providers may not have 

availability for 2–3 months, so they specify that as the return to clinic time frame. 

Using inappropriate and inconsistent scheduling practices greatly distorts the actual 

waiting time for appointments. We analyzed VHA’s data and found that: 

	 Of the new patient treatment appointments, VHA staff scheduled 81 percent 

(211,000) on the patient’s desired date of care resulting in a 0-day wait. 

	 Of the established patient treatment appointments, VHA staff scheduled 88 percent 

(8.9 million) on the patient’s desired date of care resulting in a 0-day wait. 

Based on discussions with medical center staff and our review of data, we contend it is 

not plausible to have that many appointments scheduled on the exact day the patient 

requested. In our opinion, this data reflects inappropriate scheduling practices and it is 

not representative of how long the patient waited to receive care. 

OIG previously reported concerns with VHA’s calculated wait time data in our Audit of 

VHA’s Outpatient Scheduling Procedures, Report No. 04-02887-169, July 8, 2005 and 

Audit of VHA’s Outpatient Wait Times, Report No. 07-00616-199, September 10, 2007. 

During both audits, OIG found that schedulers were entering an incorrect desired date. 

VHA Directives require all scheduling staff to complete training on scheduling 

procedures and require VA Medical Center directors to: 

 Ensure every scheduler successfully completes all training modules.
 

 Ensure all schedulers receive an annual competency assessment.
 

 Ensure completion of an annual scheduler audit of the timeliness and appropriateness
 

of scheduling actions, and the accuracy of desired dates. 

 Be vigilant in the identification and avoidance of inappropriate scheduling activities. 

VA Office of Inspector General 6 
Review of Veterans’ Access to Mental Health Care 

Our review did not fully analyze if the construct of VHA’s training and oversight 

program contributed to the problems we identified. Regardless, given VHA’s inability to 

correct this long-standing problem, VHA should reassess their training, competency, and 

oversight methods to ensure reliable and accurate appointment data is captured. 

Increasing Workload and Staffing Shortages 

According to VHA, from 2005 to 2010, mental health services increased their staff by 

46 percent and treated 39 percent more patients. Despite the increase in mental health 

care providers, VHA’s mental health care service staff still did not believe they had 

enough staff to handle the increased workload and consistently see patients within 

14 days of the desired dates. 

The VA survey asked mental health professionals whether their medical center had 

adequate mental health staff to meet current veteran demands for care; 71 percent 

responded their medical center did not have adequate numbers of mental health staff. 

Based on our interviews at four medical centers, staff in charge of mental health services 

reported VHA’s greatest challenge has been to hire and retain psychiatrists. We analyzed 

access to psychiatrists at the four medical centers we visited by determining how long a 

patient would have to wait for the physician’s third available appointment. On average, a 

patient had to wait 41 days. Specifically: 

 19 days at the Denver VA Medical Center 

 28 days at the Milwaukee VA Medical Center 

 80 days at the Spokane VA Medical Center 

 86 days at the Salisbury VA Medical Center 

Three of the four sites we visited had vacant psychiatry positions. The Salisbury VA 

Medical Center had an average wait of 86 days for the psychiatrists’ third next available 

appointment. Staff at that facility told us they lost three psychiatrists to private practice 

facilities in the past year. VHA needs to conduct a staffing analysis to determine if 

psychiatrist, or other mental health provider, vacancies are systemic issues impeding 

VHA’s ability to meet mental health timeliness goals. 

Conclusion 

Most first-time patients waited more than 14 days for their full mental health evaluation 

and VHA could not always provide existing patients their treatment appointments within 

14 days of their desired dates. Performance measure data for mental health appointments 

was not accurate or reliable. Without accurate and appropriate data, VHA’s leadership 

and decision makers cannot make informed decisions for improving access to mental 

health care. 

VA Office of Inspector General 7 
Review of Veterans’ Access to Mental Health Care 

Recommendations 

1.	 We recommend the Under Secretary for Health revise the current full mental health 

evaluation measurement to ensure the measurement is calculated to reflect the 

veteran’s wait time experience upon contact with the mental health clinic or the 

veteran’s referral to the mental health service from another provider to the 

completion of the evaluation. 

2.	 We recommend the Under Secretary for Health reevaluate alternative measures or 

combinations of measures that could effectively and accurately reflect the patient 

experience of access to mental health appointments. 

3.	 We recommend the Under Secretary for Health conduct a staffing analysis to 

determine if mental health staff vacancies represent a systemic issue impeding the 

Veterans Health Administration’s ability to meet mental health timeliness goals, and 

if so, develop an action plan to correct the impediments. 

Under Secretary for Health Comments 

The Under Secretary for Health concurred with the OIG’s findings and recommendations. 

VHA has convened a work group to examine how best to measure wait time and develop 

an action plan to create new metrics. They are committed to developing measures that 

will reliably and accurately capture appointment scheduling performance for new and 

ongoing treatment. VHA began collecting vacancy data for mental health staff on 

January 31, 2012, and will assess the impact of vacancies on operations and develop 

recommendations for improvement. We consider the planned actions acceptable and will 

follow up on the implementation. 

The Under Secretary for Health provided the following technical comments. The Under 

Secretary stated that the metric labeled in the report as “First-Time Patient’s Access to a 

Full Mental Health Evaluation,” was never intended to measure waiting time, but rather, 

as the OIG suggests, it was intended to measure how long it took VHA to conduct the 

evaluation. The Under Secretary also stated that the OIG did not use VHA’s 

methodology of the desired date to assess the accuracy of the metric for new patient 

waiting time. Rather, OIG assessed waiting time based on the time the appointment was 

entered into the system, which was the create date. 

OIG Response to VHA’s Technical Comments 

The OIG does not agree that the metric was intended to measure how long it took VHA 

to conduct the evaluation. The metric states that veterans defined as new to mental health 

will have further evaluation and initiation of mental health care in less than 15 days of 

trigger encounter (walk in or direct access to mental health clinic) or a referral to mental 

VA Office of Inspector General 8 
Review of Veterans’ Access to Mental Health Care 

health service from either primary care provider or other specialty care provider. 

Therefore, we believe VHA should be measuring the time from when the patient directly 

contacts mental health service or the initial consult referral from a primary or specialty 

care provider. 

We used the premise that new patients want to be seen as soon as possible to assess the 

accuracy of the metric for new patient waiting time. For appointments with a consult 

referral we used the referral date as the desired date. For appointments without a consult 

referral we used the appointment create date as the desired date since that was the earliest 

date indicating the veteran wanted mental health care. 

VA Office of Inspector General 9 
Review of Veterans’ Access to Mental Health Care 

Issue 2: VHA Measures Do Not Fully Reflect Critical 

Dimensions of Mental Health Care Access 

VHA’s primary mental health access measure, the 14-day follow-up measure, provides 

decision makers with a limited picture of a new patient’s ability to access and begin 

mental health treatment at VA. Additionally, depending on a veteran’s point of access, it 

may not truly measure VHA’s objective “to ensure timely access for all veterans who are 

new to mental health.” A series of timeliness and treatment engagement measures that 

better reflect the various dimensions of access may provide VHA decision makers with a 

more comprehensive view of the ability with which new patients can access mental 

health treatment. Developing strong access measures not only requires an understanding 

of how veterans access and demand mental health services but also what VA’s supply 

and availability is for services. 

Defining Access 

In its 1993 report on Access to Health Care in America, the Institute of Medicine defined 

access as the timely use of personal health services to achieve the best possible outcomes. 

The authors noted their definition forces one to identify those areas of medical care in 

which services can be shown to influence health status and then to ask whether the 

relatively poorer outcomes of some population groups can be explained by problems 

related to access. The Institute of Medicine definition also emphasized the need to go 

beyond typical approaches relying mainly on enumerating health care providers, the 

uninsured, or encounters with health care providers to identify access problems. The 

authors also commented, “no matter how generally efficacious a particular health service 

may be, a good outcome cannot always be guaranteed. The most important consideration 

is whether people have an opportunity for a good outcome—especially in those instances 

in which medical care can make a difference.”1 

Access to mental health care is a multi-dimensional concept that can be assessed relative 

to a variety of domains including: 

 Geographic location (travel time, distance)
 

 Temporal (time to appointments, waiting time, time of day, day of week)
 

 Setting (primary care versus mental health, general mental health care versus
 

specialized mental health care) 

 Urgency (emergency mental health treatment, routine treatment) 

 Type of mental health service (medication management, psychotherapy, case 

management, group versus individual treatment) 

1 Millman, M., Access to Healthcare in America, Institute of Medicine, Washington D.C., The National Academies 

Press, 1993. 

VA Office of Inspector General 10 
Review of Veterans’ Access to Mental Health Care 

	 Provider (psychiatrist, psychologist, social worker, nurse practitioner, addictions 

counselor) 

 Subjective consumer satisfaction 

 Affordability2 and utilization (number of encounters) 

 Penetrance (percent of at-risk patients receiving treatment) 

Overview and Definition of VHA Mental Health Access Measures 

At the November 30, 2011, Senate Committee on Veterans’ Affairs hearing, “VA Mental 

Health Care: Addressing Wait Times and Access to Care,” VHA reported that to address 

mental health access, a new four-part mental health measure would be included in the 

performance contract for VHA leadership. The measures in the performance contract 

would define what leadership is accountable to accomplish. Accordingly, in the 

FY 2012 Network Director Performance Plans, one of the items used to evaluate the 

performance of Veterans Integrated Service Network (VISN) Directors is “the Network 

Director assures timely and appropriate access to mental health services.” The four 

measures referred to at the hearing are incorporated in the FY 2012 performances 

measures listed for this element. The performance plan includes a fifth measure based on 

desired date. The limitations of the desired date measure were discussed in an earlier 

section of this report. Table 1 includes an explanation of the numerator and denominator 

used for the four referenced measures. The four measures are: 

1. Percentage of eligible patient evaluations documented within 14 days of new mental 

health patient index encounter 

2. Mental health follow-up within 7 days of discharge from an inpatient mental health 

unit 

3. Percentage of Operation	 Enduring Freedom/Operation Iraqi Freedom (OEF/OIF) 

veterans with a new diagnosis of post-traumatic stress disorder (PTSD), receiving 

eight therapy visits within a 14-week period within 1 year of the initial mental health 

visit 

4. Percentage of patients with an activated suicide high-risk flag placed on charts who 

receive four follow-up visits within a 4-week period following inpatient hospital 

discharge 

2 Managing Managed Care: Quality Improvement in Behavioral Health, Institute of Medicine, Washington D.C., 

The National Academies Press, 1997. 

VA Office of Inspector General 11 
Review of Veterans’ Access to Mental Health Care 

Table 1. Measures/Monitors Used by VHA To Monitor Access to Mental Health Care 

Measure Numerator Denominator 

1. Percentage of eligible patient 

encounters documented within 

14 days of new mental health 

patient index encounter 

(14-day follow-up for new patients) 

Veterans with encounter in specific 

mental health stops with a full 

evaluation < 15 days from date of 

initial trigger encounter or referral 

(consult).3 

Veterans with encounters in any 

stop code except compensation and 

pension, neuropsychological testing 

and smoking cessation for a mental 

health diagnosis /problem and no 

prior encounters in specific mental 

health stops in the previous 

24 months at that specific medical 

center. Veterans with a Mental 

health diagnosis who are treated in 

Primary Care and not referred to 

mental health are not in this monitor. 

2. Mental health Inpatient 7 day 

follow-up 

(7-day follow-up discharge) 

Mental health discharges with a faceto-face, telehealth, or telephone 

encounter in a mental health stop 

code during the 7 days after 

discharge. The initial follow up 

encounter cannot be on the same day 

as the discharge from inpatient. If 

initial follow-up contact is by telephone 

within 7 days, a face-to-face or 

telemental health follow up must occur 

within 14 days. 

VA inpatient discharges in which the 

patient had at least 1 bed day of 

care on a mental health service. 

3. OEF/OIF veterans with 

8 psychotherapy visits in 14 weeks 

(OEF/OIF Psychotherapy) 

Those patients are then required to 

have 8 psychotherapy sessions (using 

the CPT codes 90801, 90806, 90807, 

90808, 90809, 90818, 90819, 90821, 

90822, 90853) within 14 weeks of one 

another sometime during that FY. 

Operation Enduring 

Freedom/Operation Iraqi Freedom 

(OEF/OIF) veterans who have 

primary PTSD diagnoses in two 

separate outpatient encounters are 

included in the denominator during 

the month of their second 

diagnoses. 

4. High-risk suicide monitor 

(High Risk for suicide) 

Total number of patients flagged as 

Suicide High Risk at discharge that 

received a qualifying follow-up mental 

health encounter with a specific 

provider Person Class in each of 

4 weeks following discharge.4 

Total number of patients with the 

health factor “Suicide High Risk PRF 

Placed on Chart” activated. 

Activation can be in either in or 

outpatient setting. 

Source: Table is from VA OIG. Numerators and Denominators are from VHA VSSC website. 

3 A stop code is a three-digit outpatient workload identifier that indicates the main clinical group responsible for 

care. Outpatient mental health clinics are indicated by stop codes 500–599. For patient encounters, primary and 

secondary stop codes can be indicated. 

4 A Person Class is a six-digit identifier used to classify providers by discipline, specialty, and subspecialty. 

VA Office of Inspector General 12 
Review of Veterans’ Access to Mental Health Care 

Points of Access and Flow of Veterans With PTSD and Substance Use Disorder 

through VHA 

Understanding how patients might access and move through VHA care provides a 

framework from which to construct meaningful access measures. The discussion that 

follows illustrates that in isolation, the first measure (14-day follow-up for new patients), 

provides decision makers with a limited picture of a new patient’s ability to access and 

begin mental health treatment. A series of timeliness and treatment engagement 

measures might provide decision makers with a more comprehensive view of the ability 

with which new patients can access mental health treatment. Additionally, on closer 

review we found that depending on a veteran’s point of access, measure 1 does not truly 

measure VHA’s stated objective “to ensure timely access for all veterans who are new to 

mental health.” 

Further, meaningful analysis and decision making requires reliable data on not only the 

timeliness of access but also on trends in demand for mental health services, treatments, 

and providers; the availability and mix of mental health staffing; provider productivity; 

and long-term treatment capacity. These demand and supply variables in turn feedback 

upon a system’s ability to provide treatment that is patient centered and timely. 

To understand access measures within the broader context of VHA mental health care, it 

is important to understand how a patient may access and move through VHA care. In 

this section, we describe a qualitative model of flow into VHA care for patients with a 

PTSD diagnosis (Figure 1) and depict a model of flow through VHA for patients with 

PTSD and patients with a substance use disorder (Figures 2 and 3). These models 

illustrate that VHA care has multiple access points and that demand for care is dynamic. 

At a given point in time, not all patients who need care seek it, and of those patients who 

seek care, not all continue care. 

Flow of Veterans with PTSD. Figure 1 depicts the demand for VHA PTSD care. 

VHA’s U.S. Vets database is a list of all veterans in the United States regardless of 

whether or not they have accessed or utilized VA care or benefits. Within the U.S. Vets 

population, some veterans may have been diagnosed with PTSD while in the military. 

Other veterans in the population may not have PTSD, did not display symptoms of PTSD 

while in the military, or were not diagnosed with PTSD during their military service. 

Of those diagnosed with PTSD in the Department of Defense, some veterans do not seek 

mental health care after leaving the military (for example, those treated in Department of 

Defense with resolution of symptoms or veterans not seeking care due to stigma related 

concerns); some veterans seek care in the private, community mental health, or 

alternative public sectors (such as through Tricare); some veterans seek care through 

VHA; and others may initially seek care elsewhere and then through VHA at a later point 

in time. 

VA Office of Inspector General 13 
Review of Veterans’ Access to Mental Health Care 

Of those without a Department of Defense (DOD) PTSD diagnosis, some veterans are not 

in need of mental health care after leaving the military; some develop PTSD symptoms 

and seek care through private, community, or alternative public sector sources; some seek 

care through VHA for non-mental health-related conditions, and a sub-set of these 

veterans may subsequently develop PTSD symptoms; while other veterans develop PTSD 

symptoms and initially seek care through VHA for this condition. Figure 1, depicts the 

flow of patients with a PTSD diagnosis (indicated by Dx) into VHA. The green box and 

green arrow toward the bottom of the figure are used to represent the demand for PTSD 

evaluation and treatment within VHA. 

Of note, the demand at a given VHA health care facility is dynamic and may fluctuate in 

relation to trends not only in the percentage of patients already within VHA care that 

need and/or seek mental health treatment at that location but to trends in the number of 

veterans who develop or initiate access to treatment within VHA for mental healthrelated conditions. 

Figure 1. DEMAND=f(x) POPULATION, PTSD BURDEN, PTSD Dx 

U.S. Veterans 

With DOD 

PTSD Dx 

Without DOD 

PTSD Dx 

No MH Care Non-VA 

MH Care 

No MH Care Non-VA 

MH Care 

VHA Care 

With VA PTSD Dx Without VA 

PTSD Dx 

PTSD Treatment 

VA Office of Inspector General 14 
Review of Veterans’ Access to Mental Health Care 

Not all patients in need of mental health treatment seek treatment. Some patients seek or 

are referred to care but do not keep appointments. Other patients may enter into care but 

are difficult to engage in sustained treatment. Timely entry into treatment and the receipt 

of services can influence the quality of mental health and substance use services. 

Measure 1: 14-Day Follow-Up (New Patients) 

A 2005 study of referrals to the Johns Hopkins Bayview Medical Center community 

psychiatry adult and child/adolescent outpatient programs from 1995 to 2000 found 

increased wait times for an initial appointment for services at a community mental health 

center adversely affected the rate of kept appointments. In their clinic population, the 

study’s authors found a linear relationship between appointment delay and cancelations 

and no-shows. The rate of cancelations and no-shows was 12 percent for patients who 

were given an appointment on the same day as the initial contact and 23 percent for the 

patients who were given an appointment the day after the initial contact. The rate rose to 

42 percent among the patients whose appointments were 7 days after initial contact and 

reached a maximum of 44 percent among patients for whom the initial appointments 

were 13 days following initial contacts. Of note, the effect of wait time to initial mental 

health appointment might differ in other clinics where the patient population 

characteristics differ from the population in the community mental health center utilized 

in the study.5 

A 2011 study examined the impact of days to treatment admission on the probability of 

completing four sessions of care within outpatient addiction treatment program units 

funded by the Network for the Improvement of Addiction Treatment. Data was analyzed 

for a cohort of patients from October 2003 to December 2004 and a second cohort of 

patients from January 2005 to March 2006. The study findings demonstrated a strong 

decrement in the probability of completing four sessions of treatment with increasing 

time between the clinical assessment and first treatment session.6 These studies suggest 

that the motivation to show for and engage in mental health evaluation in treatment may 

wane as the delay between referral and receipt of initial evaluation and treatment 

increases. 

VHA Handbook 1160.01, Uniform Mental Health Services in VA Medical Centers and 

Clinics, states that evaluations and treatment for mental health conditions can be provided 

through primary care and other medical settings or by arrangements with non-VA 

community services. New patients requesting or referred for mental health services must 

receive an initial evaluation within 24 hours and a more comprehensive diagnostic and 

treatment planning evaluation within 14 days. The primary goal of the initial 24-hour 

5 Gerard Gallucci, M.D., M.H.S.; Wayne Swartz, L.C.S.W.-C.; Florence Hackerman, L.C.P.C., Brief Reports:
 

Impact of the Wait for an Initial Appointment on the Rate of Kept Appointments at a Mental Health Center,
 

Psychiatric Services, Vol. 56 No.3, March 1, 2005.

6 Hoffman, K.A., et al., Days to treatment and early retention among patients in treatment for alcohol drug
 

disorders, Addictive Behaviors (2011), doi:10.1016/j.addbeh.2011.01.031.
 

VA Office of Inspector General 15 
Review of Veterans’ Access to Mental Health Care 

evaluation is to identify patients with urgent care needs and to trigger hospitalization or 

the immediate initiation of outpatient care when needed. The initial 24-hour evaluation 

can be conducted by primary care, mental health providers, other referring licensed 

independent providers, or by licensed independent mental health providers.7 

The Deputy Undersecretary for Health for Operations and Management’s Monitors and 

Guidelines for FY 2010 included what we refer to in this report as measure 1 within the 

domain of access. The monitor was titled “ensure timely access for all new veterans who 

need mental health care.” According to VHA monitor documentation, the objective of 

measure 1 is to ensure timely access for all veterans who are new to mental health. 

Veterans defined as new to mental health will have further evaluation and the initiation of 

mental health care will occur within 15 days of a trigger encounter (walk in or direct 

access to mental health clinic) or a referral to mental health service from either primary 

care provider or other specialty care provider.8 

In practice, how VHA collects and reports data for measure 1 differs in part from the 

objective described for the measure. The trigger for measure 1 is not usually a referral or 

consult. Instead, it is typically an initial visit at a mental health clinic 500 series stop 

code. This distinction is illustrated in Figure 2, which represents the flow of patients with 

PTSD symptoms through possible VHA outpatient treatment settings, toward possible 

treatment options in primary care and specialty mental health treatment venues. Not all 

specialized mental health clinics depicted in the diagram are located at each VHA site, 

such as a women’s stress disorders treatment team (WSDTT), military sexual trauma 

clinic, or substance use PTSD treatment team clinic. 

Alternative paths that trigger measure 1 are depicted in Figure 2. (See Appendix D for 

detailed legend of Figure 2.) The data underlying measure 1 informs decision makers 

regarding the timeliness within which comprehensive new patient evaluations are 

completed but not necessarily the timeliness within which patients requesting or referred 

for mental health services receive a comprehensive evaluation or treatment. 

7 VHA Handbook 1160.01, Uniform Mental Health Services in VA Medical Centers and Clinics, September 11,
 

2008.
 8 “Mental Health 14-Day Follow-Up Monitor Documentation,” Last Updated 4/20/2011. Accessed at VHA Support
 

Service Center (VSSC) website
 

VA Office of Inspector General 16 
1

Review of Veterans’ Access to Mental Health Care 

Figure 2. Flow of Patients With PTSD 

MENTAL HEALTH TREATMENT 

Veteran Population With PTSD 

Seeking Care Through VA Medical Centers and CBOCs Not Seeking Care at 

VAMC or CBOC 

Vet 

Centers 

Community 

Mental 

Health/ 

Private 

Emergency 

Department 

Primary 

Care 

Women’s 

Clinic 

Mental 

Health 

Walk-In 

Clinic 

Primary Care/ 

Mental Health 

Integration 

Points of Access Treatment Options 

OUTCOMES 

Fee-Basis 

Treatment/ 

Contracted 

Care 

Mental 

Health 

Clinic 

PCT/PTSD 

Clinic 

TeleMental 

Health 

WSDTT, 

Military Sexual 

Trauma, Other 

Specialty Programs 

Substance 

Use PTSD 

Team 

Case 

Mgmt EBT Other 

Therapy 

Med 

Mgmt Counseling 

& Education 

Inpatient 

Treatment 

(Acute or 

Residential) 

Counseling: 

individual, 

Family or 

Group 

1 

1 

1 

1 

1 

2 

1 

1 

2 

EBT 

Mental 

Health 

Clinic 

Other 

Therapy 

1 

3 3 

Source: VA OIG 

VA Office of Inspector General 17 
Review of Veterans’ Access to Mental Health Care 

As is apparent in Figure 2, there is not one uniform route or channel toward treatment for 

all patients but multiple potential channels through which patients may access evaluation 

and PTSD care and by which measure 1 will or will not be triggered: 

1.	 Patients may utilize VHA primary care services but may seek mental health 

treatment outside of VHA through a private or community provider. 

2.	 Patients may not seek or may decline PTSD treatment at the medical center or 

Community Based Outpatient Clinic (CBOC) but may seek counseling at a 

readjustment-counseling center (Vet Center). Evaluation at a vet center is not 

captured in measure 1. 

3.	 Patients triaged in a primary care or mental health clinic may, under certain 

circumstances, be referred outside of VHA for treatment by a fee basis provider. 

Unless the fee basis provider treats patients on VA facility grounds, the time to 

completed evaluation is not captured in measure 1 for fee basis treatment. 

4.	 Patients with PTSD symptoms may initially be screened and evaluated by a primary 

care provider in a primary care clinic or a comprehensive women’s clinic in those 

facilities that have a women’s clinic. Patients evaluated in these settings may 

decline referral to mental health and opt to be treated for PTSD symptoms by their 

primary care provider. These patients would not be captured by measure 1. 

5.	 Patients presenting to a primary care or other medical clinic with PTSD symptoms 

may be referred to a specialized mental health primary care-mental health 

integration clinic provider who is co-located and or/collaborates with the patient’s 

primary care provider. An integrated primary care-mental health visit is captured by 

the decision support system (DSS) stop code/identifier 534. If the patient was seen 

for comprehensive evaluation by the mental health clinician on the same day, the 

visit would be picked up by measure 1 as a 0 day wait. If a comprehensive 

evaluation cannot be completed by a primary care-mental health integrated clinic 

provider on that day, then the measure captures the time from the index primary 

care- mental health integrated clinic provider visit to the time of the next visit with a 

primary care-mental health integrated clinic provider at which the evaluation is 

completed. 

6.	 If a patient is seen in a primary care clinic, another non-mental health medical 

clinic, or the emergency department and a referral is made for a mental health 

appointment at a specialized mental health clinic, for example mental health clinic 

or PTSD Clinical Team (PCT) clinic, the clock for measure 1 is not triggered until 

evaluation is initiated on the day the patient presents at the mental health clinic for 

the appointment. If a comprehensive evaluation was completed within the index 

visit to the mental health clinic, this would be captured as a 0 day wait. If the 

VA Office of Inspector General 18 
Review of Veterans’ Access to Mental Health Care 

evaluation is completed at a subsequent visit, then the time from the initial visit to 

the subsequent visit is captured by the measure. 

7.	 If a patient calls PCT or another mental health clinic directly and sets up an 

appointment, again, the clock for measure 1 begins on the day of presentation (not 

when the appointment is requested) at the clinic, and the clock stops on the day a 

comprehensive evaluation is completed. If the evaluation was completed on the day 

of an initial trigger presentation, this would be captured as a 0 day wait. If the 

patient receives a triage evaluation at the initial visit, then the clock for measure 1 

begins with the initial mental health visit and ends when evaluation is completed at 

a subsequent mental health clinic or PCT visit. 

8.	 If a new patient spontaneously presents to a mental health walk-in clinic and a 

comprehensive evaluation is completed on the same day, this would be captured by 

measure 1 as having a zero day wait. A new patient spontaneously presenting to a 

walk-in clinic that receives an initial or triage evaluation and is scheduled for 

evaluation in another mental health clinic would trigger measure 1 at the time of 

presentation to the walk-in clinic. In this scenario, measure 1 would capture the 

time from the walk-in clinic visit to the day of completion of evaluation in the 

second mental health clinic. 

9.	 While measure 1 provides information regarding completion of new patient 

evaluations within 14 days of an index mental health encounter, it does not capture 

the timeliness within which new patients are evaluated and treated from when they 

are initially referred or initially request an appointment. In addition, it would not 

capture the timeliness with which established patients are seen for a second, third, or 

subsequent mental health appointments. 

10.	 Once a comprehensive evaluation is completed in a specialized mental health clinic, 

referral to a different specialized mental health clinic would not be captured in 

measure 1. For example, if a patient is evaluated and treated in a general mental 

health clinic for PTSD but is referred to in an intensive outpatient substance use 

treatment program (IOP) during the course of their treatment in the mental health 

clinic, the wait time from referral by the mental health clinic to treatment in the IOP 

clinic is not captured by measure 1 or another measure. The completion of an 

evaluation starting and ending during the initial or a subsequent visit to the PCT 

clinic would also not be captured in the measure because the patient is not new to 

mental health. 

11.	 Access to specific mental health providers (psychiatrist, psychologist, social worker, 

nurse practitioner, substance use counselor) may vary by VHA site. Timeliness 

measures in isolation would not capture access to specific providers for which 

staffing, capacity, and productivity data would be informative. 

VA Office of Inspector General 19 
Review of Veterans’ Access to Mental Health Care 

Measure 2: 7-Day Follow-Up Discharge 

The weeks following discharge from inpatient mental health hospitalization represent a 

period of increased suicide risk. A British study found that patients receiving closer 

follow-up and more intensive aftercare were less likely to commit suicide.9,10 A large 

Danish study found two sharp peaks for suicide around psychiatric hospitalization—one 

in the first week after admission and another in the first week after discharge.11 

Measure 2 (follow-up within 7 days post hospitalization) is depicted in Figure 2. For 

measure 2, the clock for the measure begins when a discharge is entered into the system 

and ends when a face-to-face, telehealth, or telephone encounter is completed. The OIG 

will be collecting information related to measure 2 as part of the onsite Combined 

Assessment Program review process during the spring 2012 Combined Assessment 

Program cycle. 

Measure 3: OEF/OIF Veterans with Eight Psychotherapy Visits in 14 Weeks 

Access to specific mental health treatment services including medication management, 

case management, individual or group evidence-based psychotherapy (EBT) for PTSD, 

other therapies (such as supportive therapy), and counseling and educational groups, may 

differ at each VHA site (such as a VA Medical Center or CBOC). The occurrence and 

duration of therapy appointments can be captured with CPT codes, but the content of the 

therapy provided, whether it is a form of EBT (such as cognitive behavioral therapy) or 

another type of therapy (such as supportive therapy), cannot be captured by existing 

administrative codes. Measure 3 (OEF/OIF veterans with a new diagnosis of PTSD, 

receiving eight therapy visits within a 14-week period within 1 year of the initial mental 

health visit), is used by VHA as an indirect marker of whether new OEF/OIF patients are 

receiving a course of evidence-based therapy (a specific mental health treatment). 

While the intent of the measure appears sound, upon review of recent data for this 

measure we noted that totals of veterans meeting the measure for certain VISNs exceeded 

the national total, and the variation among VISNs was very wide. The Office of Mental 

Health Service explained these discrepancies by noting that data extraction techniques for 

the measure were still being refined in order to resolve counting effects of patients 

receiving therapy in overlapping 14-week periods and fiscal years. 

9 Hunt I.M., Kapur N., Webb R., Robinson J., Burns J., Shaw J., and Appleby L., Suicide in recently discharged
 

psychiatric patients: a case-control study, Psychological Medicine, 2009, 39, 443-449.
 10 Suicide After Psychiatric Hospitalization. Review of Hunt I.M. et al. (cited in previous footnote), Journal Watch
 

Psychiatry, May 11, 2009.

11 Qin, Ping and Nordentoft, Merete, Suicide Risk in Relation to Psychiatric Hospitalization, Archives of General
 

Psychiatry, Volume 62, April 2005, p.427-432.
 

VA Office of Inspector General 20 
Review of Veterans’ Access to Mental Health Care 

Measure 4: High-Risk Suicide Monitor 

Measure 4, which requires four follow-up visits in 4 weeks following discharge for 

patients flagged as high risk, is not depicted in Figure 2 but would be represented in a 

fashion similar to that shown for measure 3 with an arrow initially emanating from 

inpatient care. The OIG will be collecting information related to measure 4 as part of the 

on-site Combined Assessment Program review process during the Spring 2012 Combined 

Assessment Program cycle. 

Flow of Veterans With Substance Use Disorders Through VHA. Figure 3 depicts the 

multi-channel flow of patients with substance use issues through possible VHA treatment 

settings, toward possible treatment options in primary care and specialty mental health 

treatment venues. The conventions are similar to those used in Figure 2. The complexity 

of flow is heightened by the potential need for inpatient or outpatient detoxification, 

residential versus outpatient rehabilitation and recovery treatment, and relapse potential. 

As in Figure 2, the small squares containing a number 1 and a number 2 refer to measures 

1 and 2. The small square containing a number 5 is used to depict a VHA monitor that 

captures the percent of patients beginning a new episode of treatment for substance use 

disorder who maintain continuous treatment involvement for at least 90 days as 

demonstrated by at least 2 days with visits every 30 days for a total of 90 days in any 

outpatient specialty substance use clinic. This is a measure of treatment engagement. As 

with patients with PTSD, there are multiple points of access and measure 1 invariably 

reflects access contingent on the patient’s point of entry into treatment. 

VA Office of Inspector General 21 
Review of Veterans’ Access to Mental Health Care 

Figure 3. Flow of Patients with SUD 

Veteran Population With Substance Use Disorder (SUD) 

Seeking Care Through VA Medical Centers and CBOCs Seeking Care Outside 

of VA Medical Center 

Community 

Support – 

AA, NA, 

other 

Community 

Mental 

Health or 

Private SUD 

Treatment 

Emergency 

Department 

Primary 

Care 

Women’s 

Clinic 

Mental Health 

Walk-In Clinic 

Admit Required for Detoxification? VA or non-VA? 

OUTPATIENT TREATMENT 

Points of Access Treatment Options 

OUTCOMES 

Mental 

Health 

Clinic 

Intensive 

Outpatient 

Treatment 

Outpatient 

Substance Abuse 

Treatment 

Program 

Opiate 

Replacement 

Treatment 

Motivational 

Enhancement 

Therapy 

Coordination 

With Justice 

System 

Individual 

Therapy 

Case 

Management 

Inpatient or 

Residential 

Treatment 

Regular Outpatient Substance Use Treatment, Intensive 

Outpatient Treatment, or Intensive VA Residential 

Treatment? 

Relapse 

Group 

Therapy 

PsychEducation 

Groups 

12-Step 

Facilitation 

Peer 

Support 

Detoxification 

at Non-VA 

Facility 

1 

1 

Dual 

Diagnosis 

Treatment 

Medication 

Management 

1 

SUD-Family 

Therapy 

Contingency 

Management 

Therapy 

2 2 

1 

Fee-Basis 

Treatment/ 

Contracted 

Care 

Treatment 

Group 

Therap 

5 

Source: VA OIG 

VA Office of Inspector General 22 
Review of Veterans’ Access to Mental Health Care 

Bundled and Competing Access Measures 

The data and measures needed by decision makers for planning and service provision 

may differ at the national, VISN, and facility level. No measure is perfect or paints a 

complete picture in isolation. 

A bundle of related timeliness measures, if feasible, might include a measure of time 

from referral, no matter where a patient presents (by self or by primary care or other 

medical provider), to initial triage evaluation (time t0 – t1); time from initial evaluation to 

completed comprehensive evaluation (time t1–t2); time from the date of completed 

evaluation to the next treatment appointment (time t2 –t3) or from the initial evaluation to 

initiation of treatment (t1-t3); an engagement measure (percentage of patients seen a 

threshold number of times during the first tx days of treatment, such as 45 days), and a 

measure from time of referral by one specialized mental health clinic (such as General 

Mental Health) to other select specialized mental health clinics (for example, PCT or 

IOP). 

Beyond measure of timeliness (or delay) to mental health care, a composite or family of 

measures to broadly inform access might also include: (1) provider panel size; (2) a 

measure of demand for services; (3) measures of supply for services encompassing both 

short and long-term treatment capacity, available supply of treatment, and utilization of 

treatment; (4) trend measure of scheduled clinic appointments either missed or cancelled 

by patients and by clinics; (5) mental health provider productivity data; and (6) balancing 

or competing measures. 

As discussed to this point, meaningful analysis and decision making requires reliable data 

not only on the timeliness of access but also on trends in demand for mental health 

services, treatments, and providers. However, understanding the availability and mix of 

mental health staffing, provider productivity, and treatment capacity is critical as well. 

VHA Mental Health Staffing and Productivity Data 

According to VHA’s informal survey of frontline mental health professionals, 71 percent 

reported that, in their opinion, their facilities did not have adequate mental health staff to 

meet current demand for care. Supply of mental health staff can be reflected in terms of 

the number and mix of providers, as well as the productivity of available providers. 

Although VHA collects and reports mental health staffing and productivity data, the 

complexity of the computations and inaccuracies in some of the data sources presently 

hinders the usability of productivity information by VHA decision makers to fully assess 

current capacity, determine optimal resource distribution, evaluate productivity across the 

system, and establish mental health staffing and productivity standards. 

The primary offices that generate mental health staffing information in VHA are the 

Allocation Resource Center (ARC) and the Office of Productivity, Efficiency, and 

VA Office of Inspector General 23 
Review of Veterans’ Access to Mental Health Care 

Staffing (OPES). The ARC and OPES have different purposes for generating and 

disseminating this information—the ARC generates the information for budget planning 

and allocation purposes; OPES generates it to construct productivity models intended to 

assist VHA managers in making resource management decisions, primarily at the facility 

level. 

ARC Staffing Reports. In 2005, the ARC introduced a series of Mental Health Tracking 

Reports to report VHA’s status on the Mental Health Enhancement Initiative, a special 

purpose fund created to support VA’s implementation of a mental health strategic plan. 

ARC staffing reports show full-time employee equivalents (FTE) for “Core Mental 

Health Staff,” which broadly includes psychiatrists; psychologists and psychology aides 

and technicians; social workers and social work aides and technicians; registered nurses, 

licensed practical nurse, licensed vocational nurses, and nurses aides; physician 

extenders, including nurse practitioners, physician assistants, clinical nurse specialists, 

and pharmacists; and therapists, including physical therapists, occupational therapists, 

and other types of therapists. Users can also drill down in the ARC’s web-based staffing 

reports to determine FTE levels for specific disciplines, as well as analyze staffing at the 

facility, VISN, or national level. The reports do not include FTE for non-VA providers, 

such as contract or fee basis physicians. The primary data sources for the ARC reported 

FTE are DSS, VA’s cost accounting system, and Financial Management System. 

As Table 2 shows, the number of core mental health staff has increased by about 

29 percent since 2008; although, the annual rate of increase each year has declined. The 

largest increases over the 4-year period have occurred with social worker FTE (not 

including social work aides or technicians) and psychologist FTE (not including 

psychology aides or technicians), for which FTE increased by 66 and 46 percent, 

respectively, between 2008 and 2011. Physician FTE increased 23 percent during the 

same period. 

Table 2. ARC Selected Mental Health Staffing for 2008–2011 

FTE 2008 2009 

Percent 

Change 

2008–09 

2010 

Percent 

Change 

2009–10 

2011 

Percent 

Change 

2010–11 

Overall 

Change 

2008–11 

Mental Health 

Core Staffing 17,127.2 19,252.5 12.4% 20,853.9 8.3% 22,032.0 5.6% 28.6% 

Physician Only 2,073.1 2,288.1 10.4% 2,452.8 7.2% 2,544.1 3.7% 22.7% 

Psychologist 

Only 1,788.5 2,122.7 18.7% 2,464.9 16.1% 2,613.5 6.0% 46.1% 

Social Worker 

Only 2,844.8 3,510.9 23.4% 4,155.2 18.4% 4,725.7 13.7% 66.1% 

Source: Allocation Resource Center, Core Mental Health Staff by VISN accessed on February 2, 2012 

VA Office of Inspector General 24 
Review of Veterans’ Access to Mental Health Care 

OPES Staffing and Productivity Reports. OPES’ role is to develop tools that VHA 

managers can use to assess and optimize clinical productivity.12 Examples of resources 

available from OPES include workforce reports for specialty programs, including mental 

health, and the Physician Productivity Cube. 

OPES’ Mental Health Workforce reports show staffing levels for physicians 

(psychiatrists) involved in direct patient care and associated mental health providers, 

including psychologists, social workers, clinical nurse specialists, nurse practitioners, and 

physician assistants. Data for the reports is extracted from VA’s payroll system and, for 

the physicians, labor-mapping information from DSS. Labor mapping is a process by 

which VHA facilities allocate employees’ time to various activities, including inpatient 

and outpatient and clinical, administrative, research, and educational activities. (VHA’s 

labor mapping process and definitions are discussed in more detail in the next section on 

productivity.) 

For physician FTE, OPES uses the number of worked hours from the Personnel and 

Accounting Integrated Data System and reported direct patient care time by provider in 

DSS, plus estimated FTE for contract and fee physicians. The estimated contract and fee 

physician FTE is calculated based on the average workload of VA paid physicians. Since 

social workers, clinical nurse specialists, nurse practitioners, and physician assistants are 

also used by other specialties, such as medicine, and because DSS labor mapping 

information is not available at the individual provider level, OPES uses Budget Object 

Codes (accounting codes used to describes types of goods or services being purchased) 

and mental health Clinic Stop Codes to estimate the associated provider time dedicated to 

mental health. 

OPES’ Physician Productivity Cube, is intended for facility managers to use, in 

conjunction with other data, to evaluate productivity at the service or department level. 

According to OPES officials, the cube is most useful when it is combined with other 

facility-level information, such as practice models or patient complexity. The cube can 

assist facility managers in decision making about staffing distributions, hiring and 

contracting, and other resource allocation decisions. 

The focus of the cube is to provide information on physician productivity. It draws from 

multiple data sources to calculate physician productivity, and it relies on a complex set of 

business rules to make the calculations. The methodology involves multi-step processes 

to determine: (1) workforce, (2) workload, and (3) productivity. 

Workforce. The physician FTE reported in the cube represents worked, direct patient 

care (or clinical) FTE of full- and part-time VA paid physicians. Productivity is not 

12 VHA Support Service Center (VSSC) website, OPES Mission, http://opes.vssc.med.va.gov/Pages/Default.aspx, accessed on 

1/30/12. 

VA Office of Inspector General 25 
Review of Veterans’ Access to Mental Health Care 

calculated for non-physician providers. Determining physician direct care FTE involves 

several steps. 

	 Person Class. Every physician in VA is supposed to be mapped into a specialty (such 

as internal medicine, psychiatry, or surgery). This is done by assigning them a 

“Person Class” in the Veterans Health Information Systems and Technology 

Architecture (VistA), VA’s integrated system of patient care software applications, in 

order to exercise their clinical privileges and to have their workload captured in the 

Patient Care Encounter application.13 Person Class, represented by numerical codes, 

was originally developed in the private sector to classify providers for billing 

purposes; it reflects aggregate groupings, provider types, and areas of specialization 

or focus. For example, Person Class 182906 corresponds to aggregate grouping by 

medical doctor, provider-type psychiatry, and specialization geriatric psychiatry. 

	 Labor Mapping. Direct care FTE is computed using facility-reported data on the 

allocation of physicians’ time. VHA policy requires that facilities map the time all 

full- and part-time VA paid physicians and dentists spend in direct patient care, 

administration, research, and education.14 Labor mapping is done in DSS, the 

designated managerial cost accounting system for VA. VHA Directive 2011-009 

describes the activities included in each time category (called Account Level 

Budgeter Cost Centers). 

	 Direct Patient Care includes time spent in preparing, providing, and following 

up on the clinical needs of patients. This may include reviewing patient data, 

consulting with colleagues about patients, contacting patients or caregivers, 

and supervising residents. Direct patient care also includes attending education 

programs aimed at maintaining or enhancing clinical skills. 

	 Administration includes time spent performing managerial or administrative 

duties, such as preparing performance reviews, meeting reporting 

requirements, managing a program or service, and participating in service, 

facility, national or professional society committees. 

	 Research includes time spent performing formal, approved health care 

research or activities that support approved research, such as working on 

projects, serving on research committees, supervising research, and writing or 

presenting publications. Research does not include clinical research where 

workload is captured under clinical care. 

	 Education includes time spent preparing for and providing formal, didactic 

training, such as giving classroom lectures or conferences or managing 

13 VHA Directive 2005-059, Person Class File Taxonomy, December 7, 2005. 14 VHA Directive 2011-009, Physician and Dentist Labor Mapping, February 28, 2011. 

VA Office of Inspector General 26 
 

Review of Veterans’ Access to Mental Health Care 

residency programs and serving on medical school committees. Education 

does not include programs taken to maintain or enhance clinical skills. 

Using the Person Class and DSS labor-mapping information, OPES calculates clinical 

time for physicians (annotated as MD FTE(C)). The difference (Δ) between MD FTE 

and MD FTE(C) represents potential physician resources currently involved in other 

activities such as administration, research, and education. Based on OPES’ supporting 

data, non-clinical psychiatrist FTE associated with administration, research, and 

education on aggregate represents roughly 15 percent of total physician FTE, with some 

facilities reporting up to 32 percent. 

For psychiatrists (and other non-surgical specialties), they further adjust this FTE to 

exclude time mapped to inpatient activities. According to OPES officials, they do this 

because of issues with the reliability and completeness in how facilities are capturing 

inpatient workload data for non-surgical specialties. Therefore, the Adjusted MD 

FTE(C) used for calculating psychiatrist FTE (and productivity) reflects worked, 

outpatient, direct patient care FTE. 

Workload. Within the cube, workload is allocated to physicians using several business 

rules in an effort to reduce double counting or overlap. Workload is reported in terms of 

patient encounters and work relative value units (WRVU).15 Workload is only captured 

for work performed “within the walls” of VHA facilities—it does not include work 

performed by off-site, fee basis, or contract physicians. 

	 Patient Encounters and RVUs. To determine provider workload, the cube uses 

monthly extracts from the National Patient Care Database, which includes outpatient 

and inpatient CPT codes transmitted from the Patient Care Encounter package. 

Workload is assigned to a specialty based on the Person Class of the first attending 

physician associated with the encounter. If during an encounter, a patient is seen by 

two physicians from different specialties, the workload is assigned to each physician. 

However, if the patient is seen by two providers from the same specialty, the 

workload is assigned only to the first listed attending physician to avoid possible 

double counting. Using these rules, workload is aggregated for each provider, and 

WRVUs are calculated using values contained in Centers for Medicare and Medicaid 

Services Physician Fee Schedule RVU files. 

	 Inpatient Workload. At present, there are limitations in the cube for accurately 

reporting inpatient workload for non-surgical specialties, including mental health. 

15 RVUs are widely used in the health care industry to reflect the resources required to perform medical procedures 

(designated by Current Procedural Terminology or CPT codes) and are used as the basis for reimbursement under 

Medicare and other health insurance. Procedures comprise three RVUs—Work RVUs reflect the time and intensity 

to perform a procedure in relation to other procedures, Malpractice RVUs reflect the costs related to maintaining 

malpractice insurance, and Practice RVUs reflect the costs associated with maintaining a medical practice. For 

workload measurement and productivity, only the WRVUs are relevant. 

VA Office of Inspector General 27 
Review of Veterans’ Access to Mental Health Care 

Although VHA Directive 2009-002, issued in January 2009, requires mental health 

providers to “document and enter encounter data on all mental health professional 

services provided in an inpatient setting,”16 there does not appear to be full or 

consistent compliance with the directive. Therefore, it is not clear how accurate or 

complete the inpatient workload shown in the cube is for the non-surgical specialties. 

Anecdotally, OPES officials know that some medical facilities are diligent about 

reporting inpatient workload, while others are underreporting workload or not 

reporting it at all. 

Productivity. Despite the fairly complex business rules for calculating direct patient care 

FTE and workload, the productivity calculation for psychiatrists is straightforward:

ሻܥሺ ܨܶܧ ܦ ܯ ݁݀ݐݏݑ ܣ݆݀ ൊ ܷܸܴ ܹ ൌ ݕݐ݅ݒ݅ݐܿݑ ݀ܲ݋ݎ

In this calculation, Adjusted MD FTE(C) represents the direct outpatient clinical time 

physicians worked in a given fiscal period. WRVU reflects workload as measured by 

Work RVUs. It is important to note that if a facility is diligent about capturing inpatient 

psychiatry workload, the WRVU will reflect both inpatient and outpatient workload, even 

though the FTE reflects only outpatient effort, thereby potentially overstating 

productivity. 

Limitations with Productivity Data. While the Physician Productivity Cube may 

provide a ballpark measure for looking at and benchmarking service-level or facility level 

productivity, it has several limitations that prevent using it as a nationwide tool to gauge 

the true supply and availability of mental health providers in VA. These limitations are 

primarily due to incomplete or inaccurate information in the data sources used to 

calculate productivity. Specific limitations include: 

	 The cube is based on numerous complex business rules resulting in various 

adjustments to and calculations of the source data. Consequently, reconciling to 

source data in the cube is cumbersome. 

	 The cube only shows productivity for VA paid full- and part-time physicians. It does 

not compute productivity for other physicians, such as contract, fee basis, or without 

compensation physicians, who serve veterans at VHA facilities or at non-VA 

facilities. 

	 The cube currently does not show productivity for non-physicians who may play 

significant roles in delivery of mental health care, including psychologists, social 

workers, clinical nurse specialists, nurse practitioners, and physician assistants. 

16 VHA Directive 2009-002, Patient Care Data Capture, January 23, 2009. 

VA Office of Inspector General 28 
Review of Veterans’ Access to Mental Health Care 

	 In calculating psychiatrist productivity, the cube uses only outpatient FTE because of 

OPES’ concern that facilities are not fully or accurately capturing inpatient workload. 

However, for those facilities that do accurately capture inpatient workload, 

productivity may be overstated since the denominator is just outpatient FTE, yet the 

numerator is both outpatient and inpatient workload. 

	 FTE (and consequently productivity) is calculated based on physicians’ direct patient 

care time, excluding administration, research, and education activities. As a result, 

decision makers may not get a complete picture as to the impact that non-clinical 

activities have on meeting patient workload demands or in terms of VA’s full capacity 

to provide services. 

	 The cube relies heavily on labor-mapping information contained in DSS and Person 

Class identified in VistA, which are self-reported by facilities. However, there is 

variability in how accurately, frequently, and consistently facilities are updating DSS 

labor mapping data, which makes it difficult to compare productivity across facilities. 

	 Although the cube includes a measure of facility complexity, it cannot account for 

local practice approaches (such as use of interdisciplinary teams), case mix, 

availability of support staff, or quality issues, which limits the cube’s usefulness for 

benchmarking between medical facilities or identifying outliers. 

The next section of this report describes what measures a few, large private sector health 

care organizations use to monitor and manage the timeliness of access to mental health 

services and will review what other data VHA already collects that may inform internal 

assessment of access to mental health care. 

Measuring Access to Mental Health Care in the Private Sector 

The challenge of meeting the demand for mental health services and measuring access 

are not unique to VHA. We interviewed the directors of behavioral health services for 

three private sector health care organizations based in the Pacific Northwest, California, 

and the Southeast. The organizations included a large, national health care system and 

two regionally-based systems. All three organizations have multiple mental health clinics 

distributed throughout their geographic service delivery area. These clinics vary widely 

in size, available resources, and patient demographics. Two of the organizations use 

electronic medical record systems for all aspects of patient care. All three have a 

mechanism to refer patients to providers outside their organizations to augment the 

capacity or range of services provided. 

VA Office of Inspector General 29 
Review of Veterans’ Access to Mental Health Care 

Despite a decrease in outpatient utilization since 2009,17 the private sector health care 

organizations reported increased demand for mental health services by as much as 

20 percent over the past 2 years. The directors attributed this increase to multiple factors, 

including declining public community mental health resources and worsening economic 

and financial pressures on individuals and families. Their primary strategy to address 

increased demand involves actively managing their clinic capacity and efficiency. For 

the most part, this includes reallocating resources to open new triage and evaluation slots, 

transitioning stable patients back to primary care, or transitioning eligible patients from 

individual to group encounters. 

Private sector health care organizations we contacted all used performance “dashboards” 

(or “scorecards”) that report different dimensions of access, such as timeliness, continuity 

of care, efficacy of treatment engagement, capacity, patient satisfaction, and quality. 

Dashboards are the key decision support tools managers use to monitor and respond to 

the changes in demand for mental health services. The dashboards provide managers at 

all levels of the organization real-time data to assess supply and demand, allowing local 

managers to rapidly adjust and reallocate clinical resources to meet needs and upper-level 

managers to quickly assess the adequacy of responses and ensure that quality of care is 

not compromised. Different types of dashboard reports are available within the 

organizations, from high-level aggregated executive summaries to point-of-care tracking 

sheets. Wide dissemination and visibility of the dashboards is critical to ensure 

performance goals are communicated and met within the organizations. For example, in 

local clinics, dashboards are displayed in areas for all employees to see. 

Dashboards display different measures designed to track various dimensions of access 

(that is, timeliness, continuity of care, efficacy of treatment engagement, capacity, patient 

satisfaction, and quality). Directors typically use multiple measures to monitor trends 

within each of the dimensions. Frequently measures are complementary—improved 

performance for one measure may be offset by performance decline of another. The most 

common dimensions reflected in the dashboards are discussed below. 

Timeliness of Care. When measuring timeliness of mental health services, the private 

sector organizations consider both new patient appointments requested through selfreferral or consult requests and follow-up appointments for established patients. 

Measuring timeliness of new patient appointments is a key focus. For new patients, the 

underlying assumption is that most patients want to initiate treatment as soon as possible 

and typically want appointments within 2 weeks. 

The organizations apply similar approaches for new patient appointments—triage, then 

intake, followed by treatment initiation with a goal of rapid stabilization. Triage 

17 Johnson, Avery, Jonathan D. Rockoff, and Anna Wilde Mathews. “Americans Cut Back on Visits to Doctors,” 

The Wall Street Journal, July 29, 2010. 

VA Office of Inspector General 30 
Review of Veterans’ Access to Mental Health Care 

management is also consistent between the organizations. Patients requesting services, 

either as walk-ins or by phone, are immediately triaged. Depending on urgency, a patient 

will receive active treatment for emergent cases in less than 24 hours (ideally within 

6 hours), urgent cases within 48–72 hours, and routine cases within 2 weeks. For consult 

requests or referrals by primary and specialty care providers, clinic staff contact patients 

within 24 business hours after the consult request to schedule appointments. 

The organizations measure “time to first appointment” as the number of days between the 

time the patient first requests an appointment or a consult is created by a referring 

provider to the day of the patient’s first appointment with a mental health provider for 

initiation of treatment with a psychiatrist, psychologist, or mental health nurse 

practitioner. The scheduling software electronically captures T0 when a clerk opens the 

appointment schedule to create a new patient appointment. Two of the directors we 

interviewed stated that they expect clinics to provide 95 percent of new patient 

appointments within 2 weeks of a patient’s first request or a consult. 

Continuity of Care and Follow-Up Appointments. The private sector organizations 

share a uniform goal of rapid stabilization of symptoms early in the treatment course. 

They strive to achieve close monitoring of new patients during the initial phase of 

treatment to allow for symptom assessment, initialization of medications, and monitoring 

for medication side effects. Their dashboards include measures of continuity of care, 

timely follow-up appointments, and the ability clinics have to continue treatment after a 

new patient’s initial visit. 

The directors at each organization established a pre-determined average number of visits 

within the first 45–60 days of an initial new patient appointment. How they achieve this 

goal varies. Typically, a form of recurring scheduling is used to guarantee access to 

4-6 appointments within a pre-determined time frame goal. Some clinics achieve this by 

scheduling the entire set of appointments at the time of initial presentation; others 

schedule the first several appointments at the conclusion of the first treatment session. 

On average, behavioral health directors expect new patients to have at least four visits 

within the first 45–60 days of treatment. Accordingly, the organization dashboards 

include measures of the length of time between subsequent visits, that is the time to the 

second (T2), third (T3), and fourth (T4) visits. 

Treatment Engagement. The private sector behavioral health directors consider 

treatment engagement an essential element of their access performance measures. They 

define engagement as both the ability to provide treatment to patients who initially seek 

care (capture), as well as their ability to provide continued and sustained participation in 

treatment after the initial visit (engagement). 

As a reflection of patient engagement in treatment, the dashboards report the percent of 

patients who continue treatment after their initial appointment. The percent of patient 

VA Office of Inspector General 31 
Review of Veterans’ Access to Mental Health Care 

engagement—that is, the percent of patients who complete a second appointment—can 

be expressed as: 

Engagement (E2) = (N patients with T2 appointment/N patients seeking care T0) ×100 

Treatment engagement measures may also indirectly reflect the quality of care delivered. 

The behavioral health directors expect at least 60 percent of new patients to return for a 

second appointment. Some also track the patient’s ability to see the same provider for 

their follow-up care, an important factor in establishing a strong patient-provider 

relationship. 

Capacity. Closely monitoring availability of future appointment slots is essential to be 

able to respond to increased demands for service. Dashboard capacity measures include 

time to first available appointment, third-next available appointment,18 and the number of 

available new appointments per week. 

This dashboard information is particularly useful to local clinic managers and providers 

who can then reallocate resources required to meet patient needs. Local managers review 

the number of open appointments available each day, the following week, and over the 

following month. They assess their capacity by the type of appointment—new intakes, 

return appointments, medication management, and individual or group therapy. Capacity 

for each type of provider (such as psychiatrist, psychologist, nurse practitioners or 

clinical specialists, and social workers) is tracked to assist in matching patients’ 

individual needs with the clinics’ available resources. Local clinics can also predict the 

average number of new patients expected for treatment. If their current schedule cannot 

accommodate their predicted demand, they initiate an informal team meeting with 

providers and staff and identify ways to open more appointment slots. 

In tandem with capacity measures, managers closely track the ability to maximize 

available patient care resources at local clinics. These measures include monitoring 

appointment “fill rate” (the percent of available appointment times actually utilized to 

deliver services), as well as “no-show” (patient fails to arrive for a scheduled 

appointment) and “cancelation” rates. 

Most of the local private sector clinics reportedly use waiting lists when there are 

unexpected spikes in demand. According to one director, for about 20 percent of 

patients, going on a wait list was preferable to the option of seeing a provider outside the 

organization because these patients perceived the organization’s staff to be more 

desirable. The longest wait, on average, was 3–4 weeks from the time a patient requested 

services. However, all of the directors we interviewed told us their local clinics routinely 

review fill rates, no-show, and cancelations and manage their wait lists in an effort to 

18 The third-next available appointment open for scheduling eliminates overestimates of available services caused by 

high no-show or cancelation rates, which reduce the “1st next available” wait time. 

VA Office of Inspector General 32 
Review of Veterans’ Access to Mental Health Care 

quickly move patients with the longest wait times to more timely appointments (that is, 

within the 2-week goal). 

Patient Satisfaction. Patient satisfaction is also a key element in the dashboards. All of 

the organizations solicit patient input through surveys and patient-initiated feedback. The 

directors especially rely on patient satisfaction measures for assessment of contracted 

care services, where access to electronic scheduling data is not available. Some include 

these measures as an ongoing deliverable in their contracts. Regional managers closely 

review discrepancies between local clinics patient satisfaction data and other dashboard 

performance measures. Such conflicts frequently serve as early indicators of access, 

engagement, or quality issues. 

Application of Private Sector Measures to VHA 

VHA currently reports mental health access metrics on the VHA Support Service Center 

(VSSC) Systems Redesign website. Some timeliness data captured by VSSC are similar 

to those we reviewed from the private sector. The current VHA scheduling system is 

able to electronically capture a “creation date” when a clerk opens the scheduler to create 

a new appointment. VHA reports can include a metric parallel to the private sector “time 

to first appointment” by measuring the time between the “creation date” and the date of 

the first evaluation and treatment provided by a psychiatrist, psychologist, social worker, 

or other mental health provider. With adaptation and testing, the first use of an 

International Classification of Disease (ICD-9) code for a specific mental health 

diagnosis (for example, PTSD) has potential for use as the trigger to start the clock for a 

measure to timeliness of mental health evaluation and treatment. 

Similar to private sector entities, measures of time to the second, third, and fourth mental 

health visit could be reported. To ensure that treatment visits are captured, the measures 

would need to specify the type of provider (for example, psychologist or psychiatrist) or 

type of service (such as evaluation, treatment visit) to avoid inclusion of complementary 

visits (such as educational groups). An engagement measure, such as the percentage of 

patients seen a threshold number of times during an initial period of treatment (for 

example, the first 45 days), would indirectly reflect both a facility’s ability to provide 

serial follow-up visits during early treatment and to sustain patient involvement in the 

initial phase of treatment. 

The inability to readily track capacity imposes a major limitation on the support tools 

available to VISN and facility decision makers. VHA officials reported the present 

scheduling software is 25 years old and the software interface is not “user-friendly.” 

Schedulers must click between several screens during the scheduling process requiring 

anywhere from 30 seconds to 5 minutes to schedule an appointment. VHA officials 

asserted that calendar-based systems used in the private sector are quicker and more 

user-friendly. 

VA Office of Inspector General 33 
Review of Veterans’ Access to Mental Health Care 

Additionally, VHA officials reported that ascertaining provider availability using the 

existing scheduling software is challenging. Each provider may have assigned several 

stop codes under which the provider sees patients. The stop codes are used in part to 

allow for reporting of utilization of specific clinics (such as PCT). For example, a 

psychiatrist may use DSS stop codes 509 (psychiatry-individual), 557 (psychiatry-group), 

and 562 (PTSD-individual) among others to capture their outpatient work on a given day. 

The system will treat each stop code as if each were a separate clinic with separate 

schedules. 

While local facilities do construct ad hoc reports to identify future appointment 

availability (appointments available that day, the following week, and the following 

month), the process is labor intensive. Measures of capacity by type of appointment (new 

patient, medication review, follow-up), or by type of provider (psychiatrist, psychologist, 

social worker, nurse practitioner) are not readily available. 

Conclusions 

The data and measures needed by decision makers for planning and service provision 

may differ at the national, VISN, and facility level. No measure of access is perfect or 

paints a complete picture in isolation. Meaningful analysis and decision making requires 

reliable data, on not only the timeliness of access but also on trends in demand for mental 

health services, treatments, and providers; the availability and mix of mental health 

staffing; provider productivity; and treatment capacity. These demand and supply 

variables in turn feed back upon a system’s ability to provide treatment that is patient 

centered and timely. 

Our review found that VHA’s 14-day follow-up measure provides decision makers with a 

limited picture of a new patient’s ability to access and begin mental health treatment. 

Additionally, depending on a veteran’s point of access, this metric does not truly measure 

VHA’s stated objective “to ensure timely access for all veterans who are new to mental 

health.” A series of timeliness and treatment engagement measures might provide 

decision makers with a more comprehensive view of the ability with which new patients 

can access mental health treatment. 

Furthermore, although VHA collects and reports mental health staffing and productivity 

data, the complexity of the computations and inaccuracies in some of the data sources 

limits the usability of productivity information to fully assess current capacity, determine 

optimal resource distribution, evaluate productivity across the system, and establish 

mental health staffing and productivity standards. 

Private sector entities with whom we spoke reported that their managers use multiple 

measures to assess a range of access parameters. These entities disseminate their 

dashboard reports to all levels of management thereby facilitating timely response to 

changing access dynamics. 

VA Office of Inspector General 34 
Review of Veterans’ Access to Mental Health Care 

Beyond measures of timeliness (or delay) to mental health care, a dashboard of user 

friendly measures that incorporate aspects of patient demand, provider supply, clinic 

capacity, and provider productivity, anchored by a consistent set of business rules, might 

provide VHA decision makers with a more robust perspective from which to assess and 

timely respond to changes in access parameters. 

Recommendations 

4. We recommend the Under Secretary for Health ensure that data collection efforts 

related to mental health access are aligned with the operational needs of relevant 

decision makers throughout the organization. 

Under Secretary for Health Comments 

The Under Secretary for Health concurred with the OIG’s findings and recommendation. 

VHA’s Office of Mental Health Operations has developed a comprehensive mental 

health information system. This operations group will review available data and add 

other data elements as necessary, ensuring the information system is aligned to the 

operational needs of the organizational structure. We consider the planned actions 

acceptable and will follow up on the implementation. 

VA Office of Inspector General 35 
Review of Veterans’ Access to Mental Health Care 

Appendix A Scope and Methodology 

We conducted this review from December 2011 to March 2012. To address our review 

objective, we reviewed applicable laws, regulations, policies, procedures, guidelines, and 

studies. Additionally, we focused on measures contained with the FY 2011 PAR: 

	 Percent of eligible patient evaluations documented within 14 days of the new mental 

health patient index encounter. 

	 Percent of specialty care appointments completed within 14 days of the desired date. 

To evaluate whether patients received full evaluations within 14 days, we obtained the 

VSSC data “Percent of Eligible Patient Evaluations Documented within 14 days of New 

Mental Health Patient Index Encounter (Formerly Mental Health 14-Day Follow-up 

Monitor)” report for FY 2011. VHA officials identified this as the source data used to 

support the 95 percent reported in VA’s FY 2011 PAR. We then obtained a statistically 

random sample to test the data. For further details on the sampling methodology, see 

Appendix B. 

To evaluate whether patients received their mental health appointments within 14 days of 

the desired date, we obtained the VSSC Report “Wait Time Final Cube (Completed 

Appointments)” from VHA Systems Redesign. We received all new and established 

patient mental health appointments, excluding Compensation and Pension visits. We 

then obtained a statistically random sample of new patient appointments and established 

patient appointments to test the data. For further details on the sampling methodology, 

please refer to Appendix B. 

We interviewed the directors of behavioral health services for three private sector health 

care organizations based in the Pacific Northwest, California, and the Southeast. The 

organizations included a large, national health care system and two regionally based 

systems. All three organizations have multiple mental health clinics distributed 

throughout their geographic service delivery area. These clinics vary widely in size, 

available resources, and patient demographics. Two of the organizations use electronic 

medical record systems for all aspects of patient care. All three have a mechanism to 

refer patients to providers outside their organizations to augment the capacity or range of 

services provided. During the review, we visited four VA medical centers located in: 

 Denver, CO 

 Milwaukee, WI 

 Salisbury, NC 

 Spokane, WA 

VA Office of Inspector General 36 
Review of Veterans’ Access to Mental Health Care 

Appendix A Scope and Methodology (cont) 

While on site at VA healthcare facilities, we conducted interviews with schedulers, 

providers and medical center management to identify the steps used to schedule 

appointments, the origins of return to clinic dates used by providers, and the clinic 

availability of the providers. The audit team used these interviews to determine if VHA 

personnel followed established scheduling procedures. 

We assessed the reliability of automated data by comparing three selected data 

elements—create date of appointment, desired date of care, date of completed 

appointment—to the electronic medical records. To test completeness of the data, we 

captured and compared all completed appointments from Spokane found in VistA and 

confirmed the completed appointments were included in VHA’s universes used to 

calculate and support the performance measures. We concluded that the data used to 

accomplish the audit objective was sufficiently reliable. 

Our assessment of internal controls focused only on those controls related to the accuracy 

of veterans’ waiting times and access to care. The Office of Audits and Evaluations and 

Office of Healthcare Inspections completed this review jointly in accordance with The 

Council of the Inspectors General on Integrity and Efficiency’s Quality Standards for 

Inspection and Evaluation. 

VA Office of Inspector General 37 
Review of Veterans’ Access to Mental Health Care 

Appendix B Statistical Sampling Methodology 

To determine whether veterans can access the mental health care that they need in a 

timely manner, we reviewed three random samples to ensure veterans seeking VHA 

mental health services for the first time received timely evaluations, and to ensure 

accurate recording of waiting times for new and established mental health patients. 

 Eligible Patient Evaluations – New Mental Health Patient Index Encounter 

 Completed mental health appointments for new patients 

 Completed mental health appointments for established patients 

Population 

The population of encounters (appointments) of veterans seeking mental health care at 

VHA for the first time (Sample 1) consisted of over 373,000 in FY 2011. These veterans 

have had no prior encounters in specific mental health clinics in the previous 24 months 

at that specific medical center, and require further evaluation and treatment in mental 

health. In the FY 2011 PAR, VHA reported 95 percent of these veterans received an 

extensive evaluation and initiation of appropriate mental health care within 14 days of a 

trigger encounter. 

The population of appointments for veterans new to a specific mental health clinic 

(Sample 2) consisted of over 262,000 completed appointments in FY 2011. The 

definition of a new patient is someone not seen by a high-level qualified provider within 

the prior 24 month. A patient will only move into the “established” category once a highlevel, qualified physician has seen them. The population of appointments for veterans 

established in the mental health clinic (Sample 3) consisted of over 10 million completed 

appointments in FY 2011. 

In the FY 2011 PAR, VHA reported they completed 95 percent of new and established 

specialty care appointments within 14 days of the desired date. We obtained and 

reviewed only the mental health patient (all 500 series stop codes) appointments and 

separated them into universes of new and established patient appointments. VHA’s 

mental health appointment data showed 95 percent of the new patient and 98 percent of 

the established patient appointments were completed within 14 days of the desired date. 

Sampling Design 

We used a simple random sampling approach to select the sample from each of the three 

populations, based on a design precision of no more than 8.2 percent of the estimated 

number of cases, a 90 percent confidence level, and an expected error rate of no more 

than 10 percent of the total. This sampling technique provided equal chance of selection 

of all records in the respective population. Based on this sampling criterion, we used a 

VA Office of Inspector General 38 
Review of Veterans’ Access to Mental Health Care 

Appendix B Statistical Sampling Methodology (cont) 

classical simple random formula to calculate the sample sizes. 

We segregated the populations of records into two groups based on two judgmentally 

selected sites and all remaining sites. Each randomly selected sample was within the two 

groups. OIG statisticians used a random number generator to generate a random 

selection of appointments, which automatically selected records from the population. 

Table 3. Sampling Approach 

New Mental Health Evaluations 

Group Total No. 

Of Sites 

Total 

Evaluations 

Sample 

Size 

Certainty Sites 2 5,700 53 

Remaining Sites 139 367,600 93 

Total 141 373,300 146 

Source: VA OIG 

Table 4. Sampling Approach 

New Patient Completed Appointments 

Group Total No. 

Of Sites 

Total 

Appointments 

Sample 

Size 

Certainty Sites 2 4,300 60 

Remaining Sites 139 257,900 200 

Total 141 262,200 260 

Source: VA OIG 

Table 5. Sampling Approach 

Established Patient Completed Appointments 

Group Total No. 

Of Sites 

Total 

Appointments 

Sample 

Size 

Certainty Sites 2 135,800 60 

Remaining Sites 139 9,908,700 200 

Total 141 10,045,500 260 

Source: VA OIG 

VA Office of Inspector General 39 
Review of Veterans’ Access to Mental Health Care 

Appendix B Statistical Sampling Methodology (cont) 

Sampling Methodology 

The 146 sampled evaluations in Sample 1, and the 260 sampled appointments each in 

Samples 2 and 3 represented particular segments of the overall universes. We accounted 

for differences in the probability of selection between groups by weighting the sample 

results. 

To evaluate access to mental health care and determine if veterans received a full 

evaluation within 14 days, we identified when the patient initially contacted mental 

health seeking services, or when another provider referred the patient to mental health. 

Specifically, we used VA’s Compensation and Pension Records Interchange to review 

patients’ visit dates, clinical notes, and consult records. We determined if the full 

evaluation contained patient history, diagnosis, and treatment plan. Based on that 

information, we calculated the number of days between initial contact in mental health 

and the full mental health evaluation. 

To evaluate wait times for mental health appointments, we determined if VHA personnel 

followed established procedures when selecting the types of appointments and veterans’ 

desired dates of care. Specifically, we reviewed the desired date of care requested by the 

provider and documented in medical records or the veteran’s desired date of care as 

recorded in the VistA scheduling and consult packages by the scheduler. Based on that 

information, we calculated the number of days between the desired date and the 

completed appointment date. 

Estimates and Margins of Error 

Based on the results of our three samples, we projected nationwide totals and percentages 

of delays to evaluations, and waiting times for new and established mental health 

appointments. We used WesVar software to calculate the weighted population estimates 

and associated sampling errors. WesVar employs replication methodology to calculate 

margins of error and confidence intervals that correctly account for the complexity of the 

sample design. The margins of error and confidence intervals are indicators of the 

precision of estimates. If we repeated this audit with multiple samples, the confidence 

intervals would differ for each sample, but would include the true population value 

90 percent of the time. The projected numbers are rounded to agree with the numbers in 

the report. 

VA Office of Inspector General 40 
Review of Veterans’ Access to Mental Health Care 

Appendix B Statistical Sampling Methodology (cont) 

Sample 1 

We projected that only 184,000 (49 percent) of over 373,000 new mental health patient 

evaluations were completed within VHA’s goal of 14 days. Table 6 shows the results of 

our sample. 

Table 6. Projections 

New Mental Health Patient Evaluations 

Attribute Projected 

Evaluations 

Projected 

Percent 

Margin of Error Based on 90 

Percent Confidence Interval Sample 

Margin of Size 

Error 

Lower 

Limit 

Upper 

Limit 

Evaluation 

Within 14 

Days 

184,000 49.4 32,764 151,643 217,171 67 

Evaluation 

Later Than 

14 Days 

161,000 43.1 32,206 128,784 193,195 68 

Evaluation 

Not 

Completed 

28,000 7.5 16,770 11,123 44,662 11 

Total 373,000 100 16,775 356,513 390,064 146 

Source: VA OIG 

VA Office of Inspector General 41 
Review of Veterans’ Access to Mental Health Care 

Appendix B Statistical Sampling Methodology (cont) 

Sample 2 

We projected that approximately 168,000 (64 percent) of new patient appointments were 

completed within 14 days of the desired date. Table 7 shows the results of our sample. 

Table 7. Projections 

New Patient Completed Appointments 

Attribute Projected 

Appointments 

Projected 

Percent 

Margin of Error Based on 90 

Percent Confidence Interval Sample 

Margin Size

of Error 

Lower 

Limit 

Upper 

Limit 

Wait Time 

Within 14 Days 168,000 64.1 14,497 153,509 182,503 175 

Wait Time Over 

14 Days 94,000 35.9 14,497 79,734 108,728 85 

Total 262,000 100.0 260 

Source: VA OIG 

Sample 3 

We projected that approximately 8.8 million (88 percent) of follow-up appointments 

within 14 days of the desired date. Table 8 shows the results of our sample. 

Table 8. Projections 

Established Patient Completed Appointments 

Attribute Projected 

Appointments 

Projected 

Percent 

Margin of Error Based on 90 

Percent Confidence Interval 

Sample Margin Size 

of 

Error 

Lower 

Limit 

Upper 

Limit 

Wait Time Within 

14 Days 8,838,900 88.0 376,936 8,460,951 9,214,822 227 

Wait Time Over 

14 Days 1,207,000 12.0 376,936 829,665 1,583,536 33 

Total 10,045,000 100.0 260 

Source: VA OIG 

VA Office of Inspector General 42 
Review of Veterans’ Access to Mental Health Care 

Appendix C Previous OIG Reports Highlighting Need 

for Improved Appointment Scheduling 

and Performance Measures 

Audit of VHA’s Outpatient Scheduling Procedures, Report No. 04-02887-169, 

July 8, 2005 

Schedulers did not follow established procedures when selecting the type of appointment 

when entering the desired appointment date, which affected how the waiting time was 

calculated. The then Under Secretary for Health agreed to ensure medical facility 

managers require schedulers to create appointments following established procedures and 

monitor the schedulers’ use of correct procedures when creating appointments. 

Audit of VHA’s Outpatient Waiting Times, Report No. 07-00616-199, 

September 10, 2007 

Schedulers were not following established procedures for making and recording medical 

appointments, and schedulers were not receiving annual training. The then Under 

Secretary for Health agreed to establish procedures to routinely test the accuracy of 

reported waiting times and completeness of electronic waiting lists and take corrective 

action when testing shows questionable differences between the desired dates of care 

shown in medical records and documented in the VistA scheduling package. The Under 

Secretary also agreed to ensure schedulers received required annual training. 

Audit of Alleged Manipulation of Waiting Times in VISN 3, Report 

No. 07-03505-129, May 19, 2008 

The U.S. Senate Committee on Veterans’ Affairs requested we review allegations that 

leadership at VISN 3 was manipulating procedures to misrepresent patient waiting times. 

Schedulers were not following appointment scheduling procedures or inputting the 

correct desired appointment date. The then Under Secretary for Health did not agree with 

the recommendations to monitor compliance with established policies since the issues 

cited for VISN 3 reflected the need for national solutions that VHA was already 

addressing in response to previous reports. 

VA Office of Inspector General 43 
Review of Veterans’ Access to Mental Health Care 

Appendix C Previous OIG Reports Highlighting Need 

for Improved Appointment Scheduling 

and Performance Measures (cont) 

Healthcare Inspection of Access to VA Mental Health Care for Montana Veterans, 

Report No. 08-00069-102, March 31, 2009 

VHA needed to increase the availability of evidence-based psychotherapy for mental 

health services in rural Montana. The then Under Secretary for Health agreed with the 

findings and took action to expand evidence-based psychotherapy with additional 

resources and developing contracts with providers throughout the state. 

Healthcare Inspection of Electronic Waiting List Management for Mental Health 

Clinics Atlanta VA Medical Center, Report No. 10-02986-215, July 12, 2011 

The inspection suggested that VHA improve performance measures for mental health 

care waiting times because medical centers may be compliant with current performance 

measures but may not be providing timely treatment. 

Review of Select Patient Care Delays and Reusable Medical Equipment Review, 

Central Texas Veterans Health Care System, Report No. 11-03941-61, 

January 6, 2012 

Appointments were routinely made incorrectly by using the next available appointment 

date instead of the patient's desired date. The Director agreed to ensure staff follow VA 

policy for scheduling outpatient appointments and monitor compliance. 

VA Office of Inspector General 44 
Review of Veterans’ Access to Mental Health Care 

Appendix D Detailed Legend for Figures 

For the figures in this report: 

 The color green is used to illustrate primary care venues and the flow 

of patients seeking care at VA facilities who have not received a completed 

comprehensive diagnostic and treatment planning evaluation by a specialized 

mental health (MH) provider. 

 The color blue is used to denote specialized MH venues and the flow 

of patients once they have received a completed comprehensive diagnostic and 

treatment planning evaluation. 

	 Light purple is used to denote care at VA readjustment counseling centers (Vet 

Centers). 

	 Light red is used to denote care at non-VHA settings including private MH 

providers, community MH services, and MH services accessed via other agencies 

(e.g. Tricare-DOD, Indian Health Service) or fee basis care arranged through 

VHA. 

 Orange is used to denote VHA inpatient/residential treatment venues. 

	 A small square containing the number 1 is used to denote VHA access measure 

1 (percent of eligible patient evaluations documented within 14 days of new MH 

patient index encounter). Placement of the small square completely within the 

lower corner of a MH treatment venue box is used to indicate that the MH index 

encounter and completed comprehensive evaluation have occurred within the 

same day at the same MH venue. The color inside the small square therefore 

shifts from green (non-MH) to blue (specialty MH) to indicate that the measure 

1 clock essentially starts at point in time 1 and end at point in time 1. 1 The 

arrow that precedes the small square is green while the arrow following the small 

square to reflect the transition to specialty MH treatment. 

	 Placement of a small, completely blue square partly above and partly within an 

upper corner of a MH venue box is used to denote VHA access measure 1 when 

the initial MH trigger encounter precedes the encounter within which a 

comprehensive diagnostic evaluation is completed. The completed evaluation 

may occur within the same specialty MH venue as the trigger MH encounter, for 

example, general MH clinic (MHC) at point in time 1 and MHC at point in time 2 

or within different MH venues, for example, MH walk-in clinic at point in time 1 

and a Women’s Stress Disorders Treatment Team (WSDTT) clinic at point in time 

2. 1 The arrow between the trigger encounter MH venue and the MH venue at 

VA Office of Inspector General 45 
Review of Veterans’ Access to Mental Health Care 

Appendix D Detailed Legend for Figures (cont) 

which the comprehensive evaluation is completed is shaded from green to blue to 

indicate capture by the measure of the time frame between encounters at each 

venue and the transition to specialty MH care. The arrow subsequently 

following the MH venue box would then be shaded completely blue. 

The measure captures the time between the point at 

which a discharge is entered into the system, and the point in time at which the 

follow-up MH encounter occurs. 

	 A small, blue square containing the number 2 is used to denote VHA access 

measure 2, MH follow-up within 7 days 

2 

of discharge from an inpatient or 

residential treatment unit. 

	 A small, blue and yellow square containing the number 3 is used to denote VHA 

access measure 3, OEF/OIF veterans new to MH receiving 8 weeks of 

psychotherapy within 14 weeks. 3 

	 Blue ovals are used in figure 2 to represent specific types of MH treatment 

services; medication management (Med Mgmt, Evidence based psychotherapy 

(EBT), other psychotherapies (Other Therapy), case management (Case Mgmt), 

counseling, and education groups. 

VA Office of Inspector General 46 
Review of Veterans’ Access to Mental Health Care 

Appendix E Under Secretary for Health Comments
 

Department of Memorandum Veterans Affairs 

Date:	 April 12, 2012 

From:	 Under Secretary for Health (10) 

Subj:	 OIG Draft Report, Veterans Health Administration: Review of Veterans’ 

Access to Mental Health Care 

To: Assistant Inspector General for Audits and Evaluations (52) 

1.	 The Veterans Health Administration (VHA) is unequivocally committed 

to providing the best care possible for Veterans and will act rapidly on all 

findings that may improve Veterans’ access to mental health care. I have 

reviewed the draft report and generally agree with the report 

recommendations. VHA leadership and staff will continue to review the 

data, findings, and conclusions drawn in this draft report carefully to 

identify how best to meet Veterans’ needs. 

2.	 VHA is currently reviewing several issues that affect the ability to 

evaluate patient wait time performance measurement. These issues 

include provider and team behavior, scheduler system practices, as well as 

methodology. 

3.	 The Office of Inspector General (OIG) assessed documentation at four 

VHA facilities. VHA will conduct a more extensive review of all VISNs 

using the approach developed by OIG. Based on the time needed to do 

these reviews and the value of the information obtained, VHA may 

expand the analysis to additional facilities in at least some VISNs. The 

goal is to develop a comprehensive set of information about how policies 

and practices work and what needs to be done to accurately assess wait 

times. 

4.	 Attachment A provides our observations about the metrics that OIG 

reviewed. 

VA Office of Inspector General 47 
Review of Veterans’ Access to Mental Health Care 

5.	 Thank you for the opportunity to review the draft report. A complete 

action plan is attached. If you have any questions, please contact Linda H. 

Lutes, Director, Management Review Service (10A4A4) at (202) 461­

7014. 

(original signed by:) 

Robert A. Petzel, M.D. 

Attachments 

VA Office of Inspector General 48 
Review of Veterans’ Access to Mental Health Care 

ATTACHMENT A
 

DESCRIPTION OF MEASURES
 

A. Metric: First-Time Patient’s Access to a Full Mental Health Evaluation 

1.	 The first metric, labeled in the report as “First-Time Patient’s Access to a Full Mental 

Health Evaluation,” was developed in 2007. It assesses the time that it takes for a 

Veteran to receive a full mental health evaluation and treatment initiation if they have 

never previously had such a mental health service at VA. VHA mandated this be 

completed within 14 days after an initial 24-hour period in which a triage evaluation 

was completed (i.e., a total of 15 days from initial referral). This distinction is 

important because in the first 24 hours after the determination that a Veteran without 

mental health treatment in the last 24 months needs mental health care, a triage 

evaluation is required, but this often is done without a specific appointment; the 

mental health provider typically sees the Veteran in Primary Care, the Emergency 

Department, at a walk-in clinic, or in some other way for this triage evaluation within 

24 hours. After triage, the patient may be immediately admitted to an inpatient 

service if the need is urgent. If not, the appointment must be made at that time and 

occur with the next 14 days. 

2.	 The OIG reports (page i) that this measure “is not a meaningful measure of waiting 

time.” We agree with this finding. This was never intended to measure wait time, 

but rather, as the OIG suggests, it was intended to “measure how long it took VHA to 

conduct the evaluation.” 

B.	 Metric: New Patient Waiting Time 

1.	 The second metric, “New Patient Waiting Time” as measured per the scheduling 

directive, refers to appointments for patients who are new to a specific mental health 

clinic. That is, they may already have contact in one or more other mental health 

clinics, so they are not new to VA mental health care in general. The VHA 

scheduling directive requires schedulers to ask Veterans when they want to be seen 

for a new appointment. This is the Desired Date. The report indicates that OIG did 

not use that prescribed VHA methodology to assess the accuracy of this waiting time 

metric. Rather, OIG assessed waiting time based on the time that the appointment 

was actually entered into the system - the Create Date. 

2.	 We agree with OIG’s concerns that use of the Desired Date leads to ambiguity of the 

interaction between the patient and the scheduler. A simplified methodology would 

significantly improve the reliability of the new patient scheduling data. This is one of 

the issues that VHA is currently reviewing. 

C. Metric: Follow-up Appointments. 

The third metric, “Follow-up Appointments” uses a similar method as described for the 

second metric “New Patient Waiting Time.” We generally agree that some revision of 

the metric is required. This is one of the issues that VHA is currently reviewing. 

VA Office of Inspector General 49 
Review of Veterans’ Access to Mental Health Care 

VETERANS HEALTH ADMINISTRATION
 

Action Plan
 

OIG Draft Report, Veterans Health Administration: Review of Veterans’ Access to 

Mental Health Care 

Date of Draft Report: March 23, 2012 

Recommendations/ Status Completion 

Actions Date 

Recommendation 1: We recommended that the Under Secretary for Health revise 

the current full mental health evaluation measurement to ensure the 

measurement is calculated to reflect the veteran’s wait time experience upon 

contact with the mental health clinic or the veteran’s referral to the mental health 

service from another provider to the completion of the evaluation. 

VHA Comment 

Concur 

The Veterans Health Administration (VHA) concurs with the Office of Inspector General 

(OIG) about the importance of ensuring that VHA’s measurement of waiting times is 

meaningful and reflective of the Veteran’s wait time experience. VHA leadership has 

consistently emphasized the critical importance of ensuring Veterans have timely 

access to mental health care with Veteran Integrated Service Networks (VISN) and 

facilities. While VHA changed its definition of access and designated measures in 

Fiscal Year (FY) 2012 to capture information about timely follow up care for specific 

high risk populations, these new measures do not directly address the “wait time 

experience” for all Veterans seeking mental health services. 

VHA has convened a work group to specifically examine how best to measure Veterans’ 

wait time experiences. This work group will develop an action plan to create new 

metrics about how long it takes to get care (such as wait time from referral to evaluation, 

or timeliness of treatment initiation) for Veterans using mental health treatment. The 

action plan will be provided to the Under Secretary for Health (USH) for review on or 

before July 1, 2012. 

In process	 Work group plan will be provided to the Under Secretary for Health 

(USH) for review NLT July 1, 2012 

Recommendation 2: We recommended that the Under Secretary for Health 

reevaluate alternative measures or combinations of measures that could 

effectively and accurately reflect the patient experience of access to mental 

health appointments. 

VA Office of Inspector General 50 
Review of Veterans’ Access to Mental Health Care 

VHA Comment 

Concur 

VHA is committed to developing a combination of measures that will reliably and 

accurately capture appointment scheduling performance for new and ongoing 

treatment. 

Appointment scheduling measures should be sufficiently flexible to accommodate a 

Veteran’s condition and the phase of treatment. For example, for Veterans who need 

weekly evidence-based psychotherapy training, a 14-day policy for time between 

appointments could be too long. At the other end of the spectrum, for a Veteran on a 

maintenance schedule after successful treatment, a return appointment in 14 days 

would be much too soon. Instead, a thoughtful, individualized maintenance plan should 

be developed to guide the timing of return appointments. The work group described in 

the response to Recommendation 1 will develop a set of options for measuring 

scheduling processes based upon Veterans’ treatment needs considering the 

complexity and severity of problems identified and the phase of treatment. 

In Process Work group plan to be provided to USH for review NLT July 1, 2012 

Recommendation 3: We recommended that the Under Secretary for Health 

conduct a staffing analysis to determine if mental health staff vacancies 

represent a systemic issue impeding the Veterans Health Administration’s ability 

to meet mental health timeliness goals, and if so, develop an action plan to 

correct the impediments. 

VHA Comment 

Concur 

VHA facility mental health staffing has historically been determined at the facility level to 

allow flexibility to staff based upon provider pools, types of programs offered, and 

Veteran demand. 

VHA has recently developed a prototype staffing model for general mental health 

outpatient care using the same methodology as was used to successfully develop its 

Primary Care staffing model. The methodology included a literature review, interviews 

with other healthcare systems, and use of available productivity data. The prototype 

mental health staffing model is currently being piloted within three Veterans Integrated 

Service Networks (VISN) to validate its effectiveness prior to field implementation. 

National implementation of the model is targeted for September 30, 2012. 

VHA began collecting monthly vacancy data about mental health direct care positions 

as of January 31, 2012. A survey collects information by facility and discipline on the 

VA Office of Inspector General 51 
Review of Veterans’ Access to Mental Health Care 

number of authorized positions for mental health and the number of vacant positions so 

that vacancy rates can be accurately calculated. VHA is currently assessing the impact 

of vacancies on operations and developing recommendations for improvement. 

In Process	 National mental health staffing guidance to be implemented NLT 

September, 30, 2012. 

VHA evaluation of vacancy data and related recommendations to 

be provided to USH NLT July 31, 2012 

Recommendation 4: We recommended that the Under Secretary for Health ensure 

that data collection efforts related to mental health access are aligned with the 

operational needs of relevant decision makers throughout the organization. 

VHA Comment 

Concur 

In March 2011, VHA established an Office of Mental Health Operations with operational 

oversight of the VHA Mental Health Program. Each VISN has identified a Mental Health 

Lead to collaborate with VHA Central Office and facility mental health providers. This 

VHA mental health organizational structure facilitates information sharing and best 

practices. 

In FY 2011, VHA developed a comprehensive mental health information system. This 

system is available to all VHA staff to support management decisions and quality 

improvement efforts. Current VHA performance metrics and quantitative data are made 

available. The system provides normative information on utilization and penetration of 

VHA mental health services by facilities within each VISN, or nationally within treatment 

domains (e.g., substance abuse services, post-traumatic stress disorder, and general 

mental health). VHA will review existing available data and add other data elements 

related to access as developed by the work group referenced in the response to 

Recommendation 1 to ensure that the mental health information system is aligned to the 

operational needs of the established VHA mental health organizational structure. 

In Process	 Data elements related to mental health access identified by the 

work groups referenced in the response to Recommendation 1 will 

be added to the mental health information system NLT September 

30, 2012 

Veterans Health Administration 

April 2012 

VA Office of Inspector General 52 
Review of Veterans’ Access to Mental Health Care 

Appendix F OIG Contact and Staff Acknowledgments
 

OIG Contact For more information about this report, please contact the 

Office of Inspector General at (202) 461-4720. 

Acknowledgments Larry Reinkemeyer, Director 

Dr. Michael Shepherd, Office of Healthcare Inspections 

Marilyn Barak 

Beverly Carter 

Lee Giesbrecht 

Dr. Monika Gottlieb 

Lance Kramer 

Brad Lewis 

Russ Lewis 

Claire McDonald 

Karen McGoff-Yost 

Daniel Morris 

Ken Myers 

Carla Reid 

Jason Schuenemann 

Nelvy Viguera Butler 

Sarah White 

Oscar Williams 

VA Office of Inspector General 53 
Review of Veterans’ Access to Mental Health Care 

Appendix G Report Distribution 

VA Distribution 

Office of the Secretary 

Veterans Health Administration 

Veterans Benefits Administration 

National Cemetery Administration 

Assistant Secretaries 

Office of General Counsel 

Non-VA Distribution 

House Committee on Veterans’ Affairs 

Subcommittee on Health, House Committee on Veterans’ Affairs 

House Appropriations Subcommittee on Military Construction, Veterans Affairs, and 

Related Agencies 

House Committee on Oversight and Government Reform 

Senate Committee on Veterans’ Affairs 

Senate Appropriations Subcommittee on Military Construction, Veterans Affairs, and 

Related Agencies 

Senate Committee on Homeland Security and Governmental Affairs 

National Veterans Service Organizations 

Government Accountability Office 

Office of Management and Budget 

: 

This report will be available in the near future on the OIG’s Web site at 

http://www.va.gov/oig/publications/reports-list.asp. This report will remain on the OIG 

Web site for at least 2 fiscal years. 

VA Office of Inspector General 54