Defense Health Care: DOD Could Improve Its Beneficiary Feedback Approaches
Published: 1998-02-06T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a congressional request, GAO reviewed: (1) whether the Department of Defense (DOD) solicits feedback from beneficiaries of its managed health care program, TRICARE, and, if so, how this is done and what the data show; (2) what other means are available to beneficiaries to provide feedback and what such beneficiary-initiated feedback could reveal about TRICARE's success; and (3) how DOD's approaches to obtaining feedback compare with the private sector's and whether opportunities exist to improve DOD's beneficiary feedback tracking and reporting.
section_title: What GAO Found
paragraphs: GAO noted that: (1) DOD obtains and uses TRICARE beneficiary feedback in several ways across the military health system (MHS); (2) DOD conducts a broad annual beneficiary questionnaire survey and a monthly survey of patients' perceptions of military treatment facilities (MTF) outpatient visits--both of which are based on private-sector models--to measure levels of satisfaction with TRICARE; (3) DOD reports the survey results throughout the MHS; (4) DOD does not conduct such surveys of MTF inpatient users or civilian network care users, though DOD officials told GAO that they are now planning to develop an MTF inpatient survey; (5) as TRICARE continues to be phased in across the MHS, DOD's annual surveys are indicating fairly levels of overall beneficiary satisfaction with the program, but lower satisfaction levels with aspects of military care; (6) DOD also tracks and reports beneficiary-initiated feedback--complaints and other comments--in ways that vary throughout the MHS; (7) a wide range exists in how much feedback information is tracked and in how the different levels of units that compose TRICARE--and other DOD offices--do the tracking; (8) beneficiary-initiated feedback reporting throughout the MHS varies as well; (9) because of the variability of DOD's recording of these data, reliably depicting the range, magnitude, or frequency of beneficiary feedback about TRICARE is not possible; (10) private health care managers rely extensively on beneficiary feedback; (11) surveys, which provide data about whole customer populations, and customer-initiated complaints, which show where specific problems have occurred, are used together as key tools to measure plan performance and identify systemic problems; (12) while no direct private-sector parallel to MHS exists, DOD's feedback efforts are somewhat similar to the private sector's, although adopting certain private practices might improve DOD's feedback systems; (13) more reliable beneficiary feedback data would also help DOD to make customer satisfaction an outcome measure in the next round of TRICARE contracts, which DOD is trying to base more on outcomes and less on process; and (14) to improve its beneficiary feedback approaches, DOD will need to consider a number of cost-benefit issues, the varying sophistication levels of beneficiary feedback management throughout MHS, and other matters.

section_title: Letter
paragraphs: Medical care is of great importance to the 8 million people eligible to receive health care services through the $15 billion-per-year Military Health System (MHS). In 1993, the Department of Defense (DOD) began implementing a major change in its health care system: conversion to a managed care program known as TRICARE. Just as in the private sector, where customer feedback is used as a key management tool, an important measure of TRICARE’s success should be whether beneficiaries are satisfied with TRICARE and what their views are of DOD’s new health care system.
In light of the importance of TRICARE, you asked that we review (1) whether DOD solicits TRICARE beneficiaries’ feedback and, if so, how this is done (such as through surveys) and what the data show; (2) what other means are available to beneficiaries to provide feedback and what such beneficiary-initiated feedback could reveal about TRICARE’s success; and (3) how DOD’s approaches to obtaining feedback compare with the private sector’s and whether opportunities exist to improve DOD’s beneficiary feedback tracking and reporting.
In doing this work, we interviewed and obtained documents from MHS officials, including officials of the Office of the Assistant Secretary of Defense (Health Affairs), the Army’s Office of the Surgeon General, the Navy’s Bureau of Medicine and Surgery, and the Air Force’s Office of the Surgeon General. We visited seven military treatment facilities (MTF) in three TRICARE regions: Southeast, Southern California, and Southwest. Also, we interviewed and obtained documents from headquarters and field office representatives of two TRICARE managed care support contractors: Foundation Health Federal Services and Humana Military Healthcare Services. To compare DOD’s beneficiary feedback approaches with private approaches, we interviewed representatives from the Joint Commission on the Accreditation of Healthcare Organizations (JCAHO); the National Committee for Quality Assurance (NCQA); Kaiser Permanente, a major commercial health maintenance organization (HMO); and Inova Health System, a Northern Virginia hospital chain. Finally, we discussed TRICARE beneficiaries’ perceptions of the program with representatives of one beneficiary group, the National Military Family Association. Although we identified the varying ways that DOD and its contractors gather and process beneficiary-initiated comments and obtained numerous sample comments, we did not review whether or how individual complaints were resolved or whether they were valid. Details of our scope and methodology appear in appendix I.
subsections: section_title: Background
paragraphs: DOD’s primary military medical mission is to maintain the health of 1.6 million active duty service personnel and be prepared to deliver health care during wartime. Also, as an employer, DOD offers health services to 6.6 million additional military-related beneficiaries, including active duty members’ dependents and military retirees and their dependents. Most care is provided in 115 hospitals and 471 clinics—called military treatment facilities—operated by the Army, Navy, and Air Force worldwide. This direct delivery health system is supplemented by DOD-funded care provided in civilian facilities. In fiscal year 1997, DOD spent about $12 billion for direct care and about $3.5 billion for civilian facility care.
In the late 1980s, in response to increasing health care costs and uneven access to care, DOD initiated, with congressional authority, a series of demonstrations to evaluate alternative health care delivery approaches. On the basis of this experience, DOD designed TRICARE as its managed health care program. TRICARE is intended to ensure a high-quality, consistent health care benefit; preserve choice of health care providers for beneficiaries; improve access to care; and contain health care costs. TRICARE is designed to give beneficiaries a choice among three approaches to health care: TRICARE Prime, an HMO-like option; TRICARE Extra, which is similar to a preferred provider option; and TRICARE Standard, a fee-for-service-type option.
The TRICARE program uses regional managed care support contracts to augment its MTFs. The contractors’ responsibilities include developing civilian provider networks, performing utilization management functions,processing claims, and providing such support functions as beneficiary education and enrollment. The 11 TRICARE regions in the United States are covered by seven managed care support contracts, and health care delivery has commenced under five of the contracts (see fig. 1).
The Office of the Assistant Secretary of Defense (Health Affairs) (hereafter referred to as Health Affairs) sets TRICARE policy and has overall responsibility for the program. The managed care support contractors are overseen by the TRICARE Support Office (TSO), a part of Health Affairs. The Army, Navy, and Air Force Surgeons General have authority over the MTFs in their respective services. To coordinate MTF and contractor services, each region is headed by a “lead agent,” which is led by a designated MTF commander and supported by a joint-service staff. The lead agent responds to direction from Health Affairs, but the services retain authority and control over their medical facilities and personnel. Therefore, lead agents seek to affect operations by working cooperatively with the MTFs in their region and the regional managed care support contractor.
section_title: DOD Uses Surveys to Solicit Some Beneficiary Feedback
paragraphs: DOD conducts beneficiary satisfaction surveys—a common private sector health care practice—to measure TRICARE’s performance and reports the results throughout the MHS. Health Affairs currently conducts two such ongoing surveys: an annual systemwide survey of all eligible beneficiaries and a monthly survey of patients’ perceptions of outpatient visits at MTFs.Both surveys are based on widely used private sector survey instruments. Health Affairs’ TRICARE Marketing Office also conducted a survey of TRICARE Prime enrollees’ satisfaction in 1996. Health Affairs officials told us that a systemwide survey targeted to MTF inpatient care is currently being planned, and a survey targeted to civilian TRICARE network care is under discussion. DOD policy requires most other beneficiary surveys—whether proposed by the services, MTFs, or managed care support contractors—to first be approved by Health Affairs. The annual surveys have indicated generally high overall satisfaction levels, with mixed results for satisfaction with particular aspects of military health care. The MTF outpatient surveys have shown satisfaction levels higher than civilian HMOs’, and the TRICARE Prime enrollee survey showed satisfaction levels somewhat lower than those of the private sector. However, officials also told us that it is too soon to use DOD’s survey results as a measure of TRICARE’s overall success. Detailed descriptions of the surveys are provided in appendix II.
subsections: section_title: Annual Surveys
paragraphs: Public Law 102-484 requires DOD to conduct an annual beneficiary survey. The survey’s purpose is to provide a comprehensive look at how beneficiaries view their health care—including their health status, the availability of health services, and related matters. The questions and scales used in the annual survey were based on private sector surveys that had been extensively tested for reliability and validity. DOD uses the survey responses to represent all eligible beneficiaries’ views and reports results for each MTF catchment area.
DOD’s 1996 annual survey results show that active duty family members’ satisfaction generally increased when compared with 1994-95 results, while satisfaction decreased for retirees and their family members. But retirees’ satisfaction generally remained higher than that of active duty family members in both surveys. Moreover, active duty family members’ satisfaction was slightly higher in regions in which TRICARE had been implemented than in the other regions. In the 1994-95 survey, retirees and their family members in TRICARE regions reported higher satisfaction than their counterparts in the other regions, but in 1996 the two results were about the same, as shown in figure 2.
While overall satisfaction levels were fairly high, satisfaction with certain aspects of military health care was somewhat lower, according to the 1996 annual survey (see fig. 3).
DOD survey officials told us it was too soon to use these annual survey results to assess TRICARE because the program is new and not yet implemented nationwide. Also, they said two surveys constitute an insufficient basis from which to identify trends, and several more annual surveys are needed of the fully implemented program before the results can be used as an overall system performance measure. Nonetheless, the lead official for DOD’s survey efforts told us of uses already being made of the annual survey’s results. For example, the 1994-95 results showed that beneficiaries were more satisfied with civilian care than with military care, which led Health Affairs and the service Surgeon General offices to design a survey targeting MTF outpatients’ perceptions of the care they received. (This survey will be discussed further below.) Also, in implementing its new Enrollment Based Capitation financing approach, DOD is using the annual survey’s health status measures and results to adjust the various MTF enrollee populations for their projected health care needs. DOD is risk-adjusting the enrollee populations on the basis of such demographic factors as age, sex, beneficiary category, and military service, which correlate with differing health care service need levels.
section_title: Outpatient Surveys
paragraphs: Health Affairs also conducts a monthly MTF survey of patients’ perceptions of outpatient visits. The survey provides detailed information on specific visits to individual clinics at all MTFs in the 50 states. Health Affairs officials told us that because the 1994-95 annual survey results showed that beneficiaries were more satisfied with civilian care than with military care, this survey was designed to more closely examine MTF care. The MTF outpatient survey was also based on survey questions developed, tested, and used by the private sector, which has facilitated comparisons of MTF and civilian care satisfaction levels. Health Affairs provides detailed survey results reports to MTFs and summary reports to lead agents and service commands.
DOD provided us with April, May, and June 1997 MTF outpatient survey results for each service and region. The results measure satisfaction on a 5-point scale in three areas: (1) access to care for a single visit, (2) quality of care during that visit, and (3) staff interaction with the survey respondent during the visit. The reports also include private sector survey results that show how civilian HMO users rate their satisfaction in the same areas. Figure 4 shows results for the entire MHS, each service, and the civilian managed care industry. Satisfaction among the three services’ MTFs is similar, and averages for all three are somewhat higher than national civilian benchmarks. Results by region are also consistent across the MHS, and all of the region averages exceed civilian HMO benchmarks. See appendix II for each region’s results and comparative civilian HMO scores in corresponding geographic areas.
section_title: TRICARE Prime Survey
paragraphs: In 1996, to help direct TRICARE marketing and beneficiary education efforts, Health Affairs’ TRICARE Marketing Office conducted a telephone survey of beneficiaries enrolled in TRICARE Prime. The survey addressed enrollees’ understanding of the Prime program, satisfaction with program aspects, perceptions about access and quality changes after Prime’s implementation, and intentions regarding reenrolling in TRICARE Prime.
Health Affairs compared the survey results with civilian managed care programs’ satisfaction levels.
DOD’s survey report describes high overall satisfaction levels, with about two-thirds of Prime enrollees either satisfied or very satisfied with TRICARE, and slightly higher ratings from non-active duty TRICARE Prime enrollees. Only 7 percent of respondents said they were unlikely to reenroll in TRICARE Prime, while 88 percent said they were likely or very likely to do so. DOD reported, however, that overall satisfaction levels with TRICARE Prime trailed the civilian sector average by about 16 percentage points. The report notes, though, that the results may be skewed by response format differences between DOD’s questionnaire and the civilian instrument. Table 1 shows the survey results for overall satisfaction.
section_title: Efforts Are Under Way to Survey Inpatient and Contractor Network Care
paragraphs: DOD does not currently conduct systemwide surveys targeted to MTF inpatient or network user satisfaction. However, Health Affairs officials told us that a working group of representatives from the Army, Navy, and Air Force Surgeons’ General Offices is planning to develop a means of surveying beneficiaries about their MTF inpatient care. The group has begun by reviewing inpatient surveys currently used by MTFs and the civilian health care industry.
Also, DOD recently eliminated a contract requirement that each managed care support contractor conduct its own annual beneficiary survey. Health Affairs officials told us they concluded that contractor-conducted surveys might lack the appearance of independence and were somewhat at odds with Health Affairs’ interest in standardizing surveys and reducing the survey burden on beneficiaries. Officials of Health Affairs, the services, and managed care support contractors are now discussing how best to obtain beneficiary views on network care using such a targeted survey.
section_title: DOD Obtains and Processes Beneficiary-Initiated Comments in Varying Ways Across the MHS
paragraphs: DOD documents, analyzes for trends, and reports on TRICARE beneficiaries’ complaints and compliments in differing ways throughout the MHS. All MHS levels, from DOD headquarters offices to TSO to MTFs and managed care support contractors, receive beneficiary-initiated feedback through such means as phone calls, letters, and personal visits. Like the private sector, DOD officials told us they use this information to identify problems and gauge performance of various MTF services. We obtained many examples of beneficiary-initiated complaints and other comments covering a host of issues. However, because beneficiary comments were not consistently documented, the examples we obtained cannot be viewed as representative of all TRICARE beneficiary-initiated feedback. Nevertheless, the examples do illustrate the types of issues military health care beneficiaries choose to raise. Detailed descriptions of feedback-related processes are provided in appendix III.
Because neither DOD nor the services require MTFs to follow a standard procedure for tracking and reporting beneficiary comments, MTFs are free to establish their own feedback systems. As a result, the MTFs we visited have differing ways of obtaining, documenting, and analyzing beneficiary-initiated feedback. The MTFs also have different ways of reporting their feedback to MTF management and others within the facility. We also found, with few exceptions, that most reporting of feedback to entities outside MTFs is not done systematically.
Lead agents also capture information on beneficiary-initiated concerns in varying ways. Each of the three lead agents we visited has systems according to which its MTFs and the regional managed care support contractor report certain TRICARE-related issues to the lead agent, including issues emanating from beneficiary comments. All three lead agents also track in some way the beneficiary feedback-related issues that they learn of. Lead agent officials told us that they consider the complaints they receive to be a valuable source of information about possible problems in their regions. None of the offices provide formal feedback-related issues reports to Health Affairs or the Surgeons General, although all have a variety of informal ways of reporting issues to them.
Health Affairs, the Surgeons General, and TSO also receive beneficiary feedback and have their own procedures for handling it. These offices maintain tracking systems for the beneficiary feedback they receive, but these systems primarily track who is responsible for handling the case and response timeliness, not the specific categories the beneficiary comments fall into. Staff in these offices told us that they use the complaints they receive as indicators of possible TRICARE problems.
Representatives of both of the managed care support contractors we contacted told us that they extensively track complaints and use them to identify system problems, and that their TRICARE tracking systems mirror the systems they use for their commercial health plans. While the managed care support contracts require periodic reports that include beneficiary feedback volume and response timeliness, DOD does not require the contractors to report their complaint tracking results to the government. Yet, managed care support contractor officials told us that they consider systematically tracked beneficiary feedback and rigorous analysis of the root causes of members’ complaints to be hallmarks of a customer service-oriented managed care plan.
Officials at the various MHS organizations we visited told us how their complaint tracking procedures have led to problem identification and elimination. For example, one MTF’s deputy commander told us that he saw an increase in “staff attitude” complaints from patients at his facility. In response, he required all facility staff to take customer service training. In another case, lead agent officials told us how their tracking of complaints indicated that TRICARE Prime enrollees were being required to drive more than an hour for an MTF’s specialty care, though this exceeded the TRICARE requirement. The lead agent staff found that driving time to the MTF routinely exceeded 1 hour because of heavy traffic in parts of the MTF’s catchment area. As a result, the staff arranged for beneficiaries in those areas to go instead to closer network providers.
Further, one contractor learned through complaints that civilian providers were referring beneficiaries to collection agencies because of unpaid bills. The contractor identified a number of problems caused by beneficiary and provider mistakes, which led to improved beneficiary and provider education efforts. This investigation also identified a DOD policy that was causing claims to be inappropriately denied. When a beneficiary needs medical care that cannot be provided at an MTF, the facility can complete a “nonavailability statement” certifying that the facility does not have the required resources to provide the care needed and authorizing the beneficiary to receive the care from a civilian provider. The contractor’s investigation found that when the computer record erroneously showed that a nonavailability statement had not been issued, DOD’s policy was to not accept a paper copy of such a statement. The contractor called this problem to the attention of DOD officials, and the policy was changed.
subsections: section_title: Sample Comments Cover a Wide Range of Issues
paragraphs: DOD officials at MTFs and other offices, contractor officials, and a beneficiary organization’s representatives provided us with more than 2,600 examples of military health care beneficiary complaints and compliments. The comments covered a wide range of areas, including health care and administrative service quality, cost issues, and access to care.
Because of the sample comments’ many forms, it is not possible to generalize across the system or to draw conclusions about comment frequency, the full range of categories that complaints or other comments may fall into, the number of comments in any particular category, how types of comments vary over time, or how complaints were resolved. Nonetheless, the following sample comments illustrate the types of concerns and favorable comments that DOD health care beneficiaries have expressed.
Examples of complaints about MTF quality of care or services included the following:
An MTF doctor unfamiliar with how to prescribe a drug gave a patient incorrect instructions on how often to take the medicine. The patient’s mother caught the mistake and confirmed it by calling the MTF pharmacy.
The daughter of a retired military member who was admitted to an MTF for cancer treatment complained that her father was not well cared for. In particular, she complained that his clothes were soiled but no one had cleaned him. Upon inquiry, MTF staff told family members where they could get supplies to clean him themselves. The daughter also complained that she had found his intravenous bag empty and blood in the tubing, and that the staff had acted as if this were “no big deal.”
Sample complaints about the quality of care or services provided by managed care support contractors follow:
A patient with a previously abnormal mammogram was told by her surgeon that a 6-month follow-up mammogram was necessary. She complained that although she discussed the need for follow-up with her network primary care manager (PCM), the PCM delayed making a referral.The patient later switched PCMs and got the referral, although the test was set for 10 rather than the prescribed 6 months after the first test.
A mother complained that the scale her network pediatrician used to weigh her newborn daughter was faulty. This led to an inadequate assessment of the infant’s weight and, subsequently, the need to hospitalize the child for severe dehydration.
Complaints about MTF access to care included the following:
A patient drove for 3 hours to a 1:00 p.m. MTF appointment for a diagnostic procedure. Upon arriving, he was told his appointment was scheduled for 3:00 p.m. but he would probably not be seen until 4:00 p.m. The patient had not eaten anything for 36 hours—as the procedure required—and now had to wait another 3 hours. He said that his requests for an explanation were not met and that the clinic staff were not attentive to his complaint.
A managed care support contractor’s letter to a lead agent described two incidents in which patients complained to the contractor about inappropriate MTF emergency care delays. In the first case, a woman with a serious medical problem called an MTF emergency room but was told to call the managed care support contractor’s health care information line. The information line nurse, however, told her to go immediately to the emergency room. In the second case, an active duty member who had gone directly to an MTF emergency room was turned away because he had not first called the health care information line. When he called, the nurse said he should return to the emergency room for treatment.
Following are complaints about access to care in contractors’ networks:
When enrolling in TRICARE Prime, a beneficiary chose a gynecologist as her PCM only to find that the doctor, misidentified in the network listing, was a pediatrician. She reported that, as a result, she spent an entire day trying to arrange an appointment with the wrong doctor. After several phone calls and letters, she received a new TRICARE card that still listed the pediatrician as her PCM.
A beneficiary tried in vain to find a TRICARE network provider in her area to treat her swollen knee. On her first call to the contractor’s toll-free number, she was given four doctors’ numbers; two of the numbers had been disconnected, one belonged to a doctor not accepting TRICARE Standard patients, and one was for a hospital emergency room. The patient tried the toll-free number again and got two more numbers, but neither doctor was working that day (Friday). On her third try, she was given six more doctors’ names, but only two came with phone numbers. She was told to look up the other four in the phone book, but none were listed. Of the two phone numbers she received, one was invalid and the other proved to be that of a pediatrician. Thus, after 2-1/2 hours of unsuccessful attempts to find a doctor, she called an MTF she previously had not been able to get through to and was given an appointment that same day.
Examples of complaints related to TRICARE costs and other financial issues follow:
A TRICARE Prime enrollee referred by her MTF to a civilian specialist complained that the doctor told her the reimbursement from the managed care support contractor was “not sufficient to perform the surgery [or cover] the cost of supplies.”
A TRICARE Prime enrollee referred by his civilian PCM to a civilian specialist began to receive bills for the care. The managed care support contractor told the enrollee that the civilian doctor was using an incorrect identification number and that the doctor should resubmit the claim. The enrollee then received a second bill and was told that the visit was being treated as a point-of-service claim (which would require the patient to pay a large part of the bill), even though his PCM had properly referred him. He was later told to disregard the second bill.
Complaints concerning both access to care and quality of administrative services included the following:
A father was to be contacted within 5 days by an MTF radiology clinic with an appointment time for his child’s procedure. When he was not called, he went to the clinic and was told that “things happen.” He found this response and the lack of an apology to be “rude and uncaring.” Subsequently, when he and the child arrived for the appointment, it had to be cancelled because the child had eaten too recently, although they had not been told of the need to fast before the procedure.
We also obtained the following examples of favorable comments about both the direct care system and contractor functions:
One MTF kept a log of all patients’ comments. The list included compliments about the friendliness, compassion, professionalism, and technical skill of specific staff members, as well as general compliments about, for example, the speed of access to care or the clinic staff in general.
A beneficiary had 6 months of claims processing problems that she described as “a nightmare.” She wrote to the managed care support contractor thanking a specific contractor staff member for resolving her problem. In a letter to a managed care support contractor, an Air Force chief master sergeant complimented staff at the local contractor office. He wrote: “Their enthusiasm and sincerity is definitely the right attitude needed to administer a program that has had the military ‘rank and file’ feeling a little uncomfortable.”
section_title: DOD Could Improve Its Feedback Tracking and Reporting by Further Adopting Private Sector Practices
paragraphs: DOD’s efforts to track beneficiary feedback resemble those of the private sector, but opportunities for improvement exist. Private health care managers make extensive use of customer feedback from surveys and rigorous customer complaint tracking and reporting. While DOD’s current survey efforts and emphasis on addressing beneficiary complaints at the local level are not unlike private practices, additional targeted surveys and more consistent complaint tracking and reporting would better inform DOD managers about beneficiaries’ experiences and more closely reflect private sector approaches to managing such information. Enhancing its current feedback efforts would also help DOD achieve its goal of bringing about a more outcomes-oriented TRICARE health system. Yet, given that the MHS differs in key ways from private sector health care systems, DOD would need to consider several basic cost and implementation issues to improve its beneficiary feedback.
subsections: section_title: Private Sector Uses Surveys Extensively to Solicit Beneficiary Feedback
paragraphs: Customer surveys are a common private sector health care feature. Health plan officials told us they survey plan members to gauge overall satisfaction and conduct targeted user surveys to measure performance in particular areas. One large managed care plan conducts an overall member survey, a survey of members who have recently received health care services under the plan, and surveys targeted toward patients’ perceptions of their doctors. Health care providers also use customer satisfaction surveys. Officials at a hospital system that we contacted told us that every patient is asked to fill out a survey after receiving care in one of the system’s facilities.
Survey results are also reported extensively throughout the private organizations we contacted. Officials told us that the results are used to identify problem areas, measure overall performance, and compare the performance of different parts of the organization. Officials at one managed care organization told us they report survey results to both senior managers and staff throughout the organization. These officials also provide special reports on results in particular areas when departments request them. The hospital system we contacted reports all patient comments, including patient questionnaire responses, to the head of hospital operations on a daily basis, while managers across the organization receive quarterly reports.
In another case, several employers that came together to purchase health care as a group identified extensive beneficiary surveying as a key measure of their system’s performance. For example, the group reports customer satisfaction information from surveys to inform beneficiaries when they are choosing providers. The group also contracts for targeted surveys of particular covered populations, including surveys focused on the health status of children and seniors.
Surveys are also central to the accreditation of managed care organizations. Both NCQA and JCAHO have accreditation programs that require managed care plans that are seeking accreditation to conduct member surveys. Health care purchasers, regulators, and consumers use the results of the accreditation process to assess all aspects of a plan’s delivery systems: physicians, hospitals, other providers, and administrative services.
A survey is also a requirement of the latest version of the Health Plan Employer Data and Information Set (HEDIS). HEDIS is a set of standardized performance measures of health care plans’ performance. HEDIS is designed to provide purchasers and consumers with the information they need to reliably compare managed care plans’ performance. To become part of the HEDIS database, health plans must use NCQA’s Member Satisfaction Survey and be prepared to report the full set of survey results. NCQA makes consolidated results available to consumers for use in selecting among health plans.
section_title: Private Sector Uses Tracked Complaints as a Key Management Tool
paragraphs: Private health care managers also extensively track customer complaints and use them to make system improvements. A large HMO’s member services director, for example, told us that members’ complaints and other comments, whether received in person, over the phone, or in writing, are tracked by computer. The purpose is to resolve members’ problems, identify root causes, and eliminate system flaws. Patient feedback tracking system reports are generated monthly and sent to staff throughout the system, including the Quality Assurance/Quality Improvement Committee.
The hospital system we examined also uses a computer system to track all complaints, whether received in person, over the phone, in writing, or in response to a patient satisfaction survey. Complaints are sent to the senior staff member of the hospital area that the complaint concerns. All patients’ complaints are reported daily to the system’s hospital operations’ vice president, and every quarter to system managers. A senior official told us that complaints are useful for identifying both one-time and systemwide problems. He explained, for example, how patients had complained about giving the same information to different people during the admitting process, which led to the elimination of this redundancy.
Representatives of one California hospital reported that analyzing patient complaints has become the hospital’s least expensive, most accurate method for understanding patients’ perspectives on what needs improvement at the hospital. When facility staff realized that individual complaints had been addressed in the past, but with little documentation or tracking, they designed a comprehensive complaint process that included procedures for capturing all complaints, responding to complaints quickly, measuring complaint severity, analyzing trends to uncover root causes of customer dissatisfaction, and identifying and implementing system changes to prevent future recurrences. The officials also reported that questionnaire surveys are not appropriate for capturing dissatisfied patients’ spontaneous complaints.
Employers who purchase managed care coverage for their employees also see the value of tracking customer complaints. For example, the HMO Performance Standards set by one large employer state that its selected plan “shall track and report to  the number and types of plan aggregate written and verbal complaints received by the HMO.” The standards require an annual report that lists complaints by categories “including but not limited to access, clinical services, providers, pharmacy, mental health/substance abuse, claims, and reception services.”
To obtain accreditation by NCQA and JCAHO as a managed care organization, managed care plans must obtain and use member feedback. Plans are required to track, report, and use customer complaints to identify and address one-time and systemic problems. NCQA standards require that customer feedback analysis include aggregating results; noting trends in results over time; and identifying reasons for the results, such as the causes of dissatisfaction in particular areas. The standards also discuss how managed care organizations should use feedback analysis results to prioritize improvement areas on the basis of their significance to members. NCQA officials told us that no one system is prescribed for managing member complaints. Rather, NCQA surveyors look at a sample of complaints, determine if a system for handling them exists, and decide if the plan is following its own system. Similarly, JCAHO network accreditation standards require health plans seeking accreditation to have customer complaint receipt and management systems.
The extensive use of customer feedback is not just a private sector health care feature; it exists throughout the private sector. A report of the Vice President’s National Performance Review describes extensive customer complaint and survey use by “best-in-business” companies and the applicability of these practices to government. It also refers to Executive Order 12862, which directs federal agencies to perform customer surveys, make complaint systems easily accessible, provide the means to address customer complaints, and measure customer service against the best-in-business. The report also describes customer feedback strategies used by best-in-business companies including facilitating customer complaints through the extensive use of centralized customer help lines, 1-800 numbers, point-of-service complaint or comment cards, and easy-to-use customer appeal processes; encouraging quick responses to customer complaints; using computers to centrally track complaints at the headquarters level; reporting tracking results widely, including to top management; and using the results to identify dissatisfaction trends and root causes to target core processes that need improvement.
section_title: Some DOD Efforts Are Comparable to Those of the Private Sector, but Enhancements Are Possible
paragraphs: Some DOD efforts to track and use beneficiary feedback compare favorably with private sector efforts. For example, DOD’s beneficiary surveys are similar to private health plan and hospital surveys. Also, MTFs and other DOD offices use complaints to help identify problems, as is done in the private sector. But, in our view, DOD could make its current efforts more complete and systematic—and thus more effective.
DOD’s current beneficiary surveys provide a view of beneficiaries’ satisfaction with their care generally and their MTF outpatient care specifically. However, adding targeted surveys of beneficiaries’ satisfaction with MTF inpatient care and TRICARE civilian network care would enhance the usefulness of DOD’s survey data. By doing so, DOD decisionmakers would have a more complete picture of TRICARE’s customer satisfaction.
DOD could also obtain more detailed information about beneficiary- initiated complaints and other comments if it standardized the way it tracks and reports this feedback across the system. Currently, no systemwide approach to tracking and reporting exists. As a result, a serious problem that is surfaced by a complaint in one region or at one MTF, for example, can remain unnoticed in other locales if no one there complains. Moreover, with a consistent approach to tracking and reporting feedback, MHS and contractor personnel could put the complaints they receive into a systemwide perspective, even if they were tracking complaints locally. Further, with standardized tracking and reporting, personnel throughout the MHS could identify trends beyond those at their own location. They would also know the overall complaint volume by type and would probably find that the problems they were seeing had already surfaced and been addressed elsewhere, potentially saving time and resources otherwise spent on reinventing the solutions.
With regular access to systematically tracked and reported complaint data, senior DOD officials could analyze complaint activity across the system, spot trends, and identify possible problems using data currently unavailable to them. Consistent complaint data would also equip senior officials with another tool for evaluating individual MTF performance and making cross-system comparisons.
Standardizing feedback tracking and reporting would also enable DOD to better judge TRICARE’s contractor performance. DOD officials are now working to make future TRICARE contracts less prescriptive in nature and more outcomes based. Past contracts have offered bidding contractors little or no opportunity to use their best commercial practices to introduce innovation or reduce costs to accomplish DOD’s goals. For the new contracts, DOD proposes to set forth its overall objectives, such as maintaining customer satisfaction, and provide a mandatory requirements list. Deciding on an approach to satisfy the objectives and other requirements will be left to the bidders.
In addition, DOD currently plans to use its annual survey and monthly MTF outpatient survey results as program success measures. By adding the other two surveys, DOD decisionmakers could focus more closely on MTF inpatient and civilian network performance and use the level of consequent beneficiary satisfaction as a key performance indicator. DOD officials could be confident that beneficiary complaints were being systematically categorized and reported so that such data could be used as a measure of the performance of managed care support contractors, MTFs, and TRICARE overall.
section_title: Military Health Care Differs in Key Ways From Private Sector Health Care
paragraphs: DOD’s multifaceted MHS role, DOD’s relationship with its managed care support contractors, and the unique chains of authority involved in the roles of the three services in delivering military health care differ from the structure of private sector health care. These differences mean that DOD’s feedback tracking and reporting is more involved than the private sector’s and that civilian standards for this activity are not necessarily easily applicable to the MHS, though the principles driving them apply to all managed care environments, including TRICARE.
Typically, private employers purchase health care coverage for their employees (or individuals purchase it directly) from health plans, which contract with doctors and hospitals to provide covered beneficiaries’ care. DOD operates differently. As the beneficiaries’ employer, it both administers TRICARE and directly provides much of the MHS’ health care through the hundreds of hospitals and outpatient clinics that it operates. Because of DOD’s merged responsibilities, which are usually held by separate entities in the private sector, the checks and balances that exist in civilian business relationships do not exist. For example, a civilian employer that receives numerous complaints about a hospital in the health plan’s network can insist that the plan either drop the hospital or lose the employer’s business. But, should an MTF receive such complaints, DOD’s options would be more limited.
Differences among civilian health care purchasers, plans, and providers are, for the most part, clear cut. In DOD, however, TRICARE is a single health plan operated by two separate entities—the direct care system (MTFs) and the managed care support contractors—each responsible for managing program parts and providing, or arranging for, health care services. Also, the contractors’ role overlaps that of the direct care system, with some patients getting their care directly from DOD, others using the contractor networks, others using non-network civilian providers, and still others using some combination of the sources. Both DOD’s hospitals and DOD’s contractors send patients to each other for some care, but neither has real financial or other authority to control what the other does. Because of the shared care administration and delivery responsibilities, beneficiary-reported problems can appear to each party to be the other’s responsibility.
The role of the three services also distinguishes military from civilian health care. While Health Affairs is responsible for running TRICARE, the MTFs are under the authority of the Army, Navy, and Air Force Surgeons General. And the regional lead agents, which also respond to direction from Health Affairs, cannot direct the activity of the MTFs in their regions but, instead, must rely on the MTFs’ cooperation to implement such new programs as regionwide complaint tracking and reporting. Moreover, neither Health Affairs nor the services can make changes in areas beyond their authority, including changes needed to address problems that surface through beneficiary feedback.
Currently, NCQA requires that a managed care plan seeking accreditation have a single entity that is responsible for the entire plan. An NCQA official told us that because TRICARE uses various sources of care and various entities are responsible for seeing that care is properly delivered, TRICARE has no single accountable entity to examine. Instead, multiple accountability lines exist and, with them, the potential for beneficiary-raised issues to go unaddressed by any responsible organization. Notwithstanding the beneficiary feedback implications, the accountable entity issue could take on greater importance should DOD seek managed care plan accreditation for TRICARE in the future, as DOD officials have told us it may.
section_title: Cost and Implementation Issues Need to Be Worked Through
paragraphs: Within its health care system’s unique context, DOD would need to explore several basic issues to improve beneficiary feedback. The cost of adding surveys and developing a single approach to handling beneficiary complaints would need to be weighed against the benefits sought. Also, DOD would need to decide how reporting the results of complaint tracking should work to ensure that information flowed to the appropriate organization and levels.
Regarding a single complaint tracking system, DOD, private sector, and managed care support contractor representatives told us that care should be taken to ensure that such a system not become overly cumbersome or bureaucratic. Managed care support contractor representatives told us such a system should be collaboratively developed with them, flexible and adaptable to decisionmakers’ changing needs, and not overly prescriptive. They also pointed out that contract-prescribed items are difficult to change because of the time-consuming contract change order process and asked, therefore, that their contracts not prescribe how they should develop such a system. Also, they told us that such a system could be composed of tracking systems that were regional in scope and designed to encourage strong DOD/contractor partnerships.
DOD would also, in our view, need to weigh potential training and other costs of adapting existing MTF and other DOD office beneficiary feedback recording systems. The costs of changing local systems would probably vary from place to place. Locations already capturing a great deal of beneficiary-initiated feedback data would probably find a standardized approach comparatively easier to adopt than those beginning the process for the first time.
Also, DOD would need to consider how to report issues to address the MHS’ multiple lines of authority. Because the services control their respective MTFs, their chains of command would be prospective report recipients. In addition, reporting protocols could include the contractors and DOD contracting officers residing at lead agents and at TSO. Finally, because Health Affairs has overall TRICARE responsibility, it would also logically receive summary feedback, because such information is designed to point up systemwide problems.
section_title: Conclusions
paragraphs: DOD is spending a great deal of money to improve its $15 billion-per-year health care program by implementing TRICARE. An investment of this magnitude heightens the importance of current, accurate, and complete information about how beneficiaries are reacting to and coping with the change. The beneficiary feedback currently available to DOD managers provides useful information about aspects of TRICARE’s performance and possible problem areas. If DOD were to make its current survey efforts more complete and to consistently record and aggregate complaint information across the system, DOD managers would have more valuable information with which to measure TRICARE’s success and identify and eliminate recurring, systemic problems. Enhanced feedback would also help DOD make the outcomes-based assessments it seeks for future TRICARE contracts.
DOD could improve its beneficiary feedback information by conducting a civilian network care survey comparable to its monthly MTF outpatient visit survey, a possibility that is now under discussion. Also, while DOD does not currently have an MTF inpatient care survey, we support DOD’s plans to develop and conduct such a survey. DOD could also benefit by working with the TRICARE contractors to begin restructuring its complaint tracking and reporting systems to more closely parallel private sector managed care practices by consistently recording and aggregating complaint data across the DOD health care system.
section_title: Recommendations
paragraphs: To position DOD to obtain and make better use of beneficiary feedback, both now and in the future, the Secretary of Defense should direct the Assistant Secretary of Defense (Health Affairs) to follow through in weighing the costs and benefits associated with civilian network and MTF inpatient care surveys that are comparable to DOD’s current monthly MTF outpatient survey and, as appropriate, implement these surveys and collaborate with the TRICARE contractors to identify options for, and weight the costs and benefits of, achieving consistency in recording beneficiary complaints, analyzing trends, and reporting beneficiary complaints and, as appropriate, implement the most practical, financially prudent approach.
section_title: Agency Comments and Our Evaluation
paragraphs: In its written comments on a draft of this report, DOD agreed with our recommendations regarding MTF inpatient and civilian network care surveys and a consistent beneficiary complaint tracking and reporting process. DOD added that the Army, Navy, and Air Force are now in various stages of reviewing their TRICARE customer relations approaches and assessing their beneficiary complaint processes.
DOD also suggested that beneficiary complaint tracking is currently done at the lead agent level. However, at the lead agents visited, we found that beneficiary feedback systems varied markedly, as did the amounts and types of complaint data routinely captured. Also, in line with our suggestion that Health Affairs would be a logical recipient of beneficiary feedback data designed to point up systemwide problems, DOD stated it is exploring a centralized process for tracking beneficiary complaints at the Health Affairs level.
DOD also suggested technical report changes, which we incorporated as appropriate. The full text of DOD’s comments is included as appendix IV.
We are sending copies of this report to the Secretary of Defense and will make copies available to others upon request.
Please contact me at (202) 512-7101 or Dan Brier, Assistant Director, at (202) 512-6803 if you or your staff have any questions concerning this report. Other GAO staff who made contributions to this report are David Lewis, Evaluator-in-Charge; Linda Lootens, Senior Evaluator; and Paul Wright, Evaluator.
section_title: Scope and Methodology
subsections: section_title: Department of Defense Survey Efforts
paragraphs: To identify Department of Defense (DOD) efforts to solicit beneficiary feedback through surveys, we interviewed officials of Health Affairs. We also obtained and reviewed documentation, including survey instruments, relating to Health Affairs surveys that included elements of TRICARE beneficiary satisfaction, as well as documents related to other Health Affairs surveys. Through discussion with Health Affairs officials, we determined that three DOD surveys fell within the scope of this review: the Health Care Survey of DOD Beneficiaries (1994-95 and 1996) (the annual survey), the Customer Satisfaction Survey (April/May/June 1997) (the Military Treatment Facility [MTF] outpatient survey), and the TRICARE Prime Enrollee Satisfaction Study (1996).
We obtained DOD reports of these three surveys’ results but did not independently assess the survey instruments’ statistical validity or reliability. In this regard, the DOD official responsible for the Health Affairs survey efforts told us that DOD uses experienced contractors to design and conduct its surveys and that survey questions are based on standard survey questions extensively pretested for validity and reliability by the private sector, and widely used in their surveys. Further, he believes DOD’s rigorous methods for sampling survey populations and weighing survey responses on the basis of numerous proven variables result in statistically valid survey data. DOD survey yield rates are similar to the average 50-percent yield rate for private sector surveys. The annual survey yield rate has been about 60 to 65 percent, and the MTF outpatient survey yield rate has been about 45 percent; both rates have been increasing over time.
section_title: Beneficiary-Initiated Feedback Processes
paragraphs: We interviewed and obtained documents from DOD officials and contractor representatives across the Military Health System (MHS) regarding policies and procedures for documenting, determining trends in, and reporting beneficiary-initiated complaints and compliments. At the DOD headquarters level, we met with Health Affairs officials to discuss tracking beneficiary feedback within Health Affairs. We also reviewed TRICARE Support Office (TSO) requirements for how managed care support contractors are to track and report feedback from beneficiaries and interviewed TSO officials about how they use the beneficiary comments that they receive. In addition, we interviewed representatives of the Army, Navy, and Air Force Surgeon General and Inspector General offices about how their organizations receive and handle beneficiary feedback. We also discussed with all of these officials the means by which they exchange information on feedback-related issues with other MHS locations.
subsections: section_title: Lead Agents
section_title: Army MTFs
section_title: Navy MTFs
section_title: Air Force MTFs
paragraphs: Wilford Hall Medical Center, Lackland Air Force Base, Texas
12th Medical Group Clinic, Randolph Air Force Base, Texas
61st Medical Squadron Clinic, Los Angeles Air Force Base, California We interviewed lead agent and MTF officials at these locations about how they track and report beneficiary comments and obtained documents related to these feedback tracking processes, including comment database formats and summary reports, comment tracking log sheets, complaint/comment forms, and procedures governing beneficiary feedback tracking and reporting.
We also interviewed representatives of two managed care support contractors—Foundation Health Federal Services and Humana Military Healthcare Services—in their headquarters and regional offices and at local contractor offices located in or near the MTFs we visited. We discussed contractors’ feedback tracking and reporting processes, both as they fulfilled DOD requirements and as they met the contractors’ own internal purposes. We also obtained documentation of the contractors’ beneficiary tracking and reporting systems. Although DOD’s contracts require the managed care support contractors to have mechanisms in place for beneficiaries to appeal managed care decisions, we did not examine the appeals process as part of this review.
section_title: Beneficiary-Initiated Feedback Examples
paragraphs: We collected over 2,600 examples of beneficiary-initiated complaints and compliments from lead agents, MTFs, and managed care support contractor officials in the three TRICARE regions we visited as well as from the National Military Family Association, a beneficiary group. For this report, we judgmentally selected example comments to identify the types of issues that beneficiaries raised. However, because of the variability of DOD’s recording of beneficiary comments, we could not determine the range, magnitude, or frequency of beneficiary comments, and we did not review the validity of complaints or how complaints were resolved by the military or contractor organizations that received them.
Because the methods by which beneficiary-initiated comments were documented varied, the set of example complaints and compliments we obtained is not representative of beneficiary comments from either the locations we visited or the MHS as a whole. In some cases, the documentation we reviewed provided only what the beneficiary said; in other cases, particularly in the case of complaints, the documentation also included information about how the complaint was handled. In other cases, the documentation consisted only of brief database entries made by staff of the organization that handled the complaint. We were also told that some complaints and compliments were not recorded in any way.
We did not assess the validity of the beneficiary concerns. However, we noted that in some cases the complaint files included information indicating that the MTF found the complaint to be invalid. For example, a patient who wanted to see a specialist not in the contractor’s network disenrolled from TRICARE Prime in order to avoid paying the substantial cost required of TRICARE Prime enrollees for out-of-network care. But the patient received care from the specialist before the effective date of disenrollment, so the patient was billed the high fee. The patient complained about the bill, but the documentation indicated that the mistake was the patient’s, not the MTF’s or the contractor’s. Another patient complained about being denied care when she could not get an ultrasound test early in her pregnancy. However, her doctor told the MTF staff researching the complaint that the test she wanted was not medically necessary.
Although we did not review whether or how DOD resolved the beneficiary concerns in the example complaints we obtained, we noted that in some cases the available complaint documentation explained DOD or contractor efforts to research and resolve the complaints. There were cases, for example, in which documentation indicated that MTF or contractor staff called the appointment telephone line to test the quality of the service it provided after a beneficiary complained about being left on hold or being given an appointment date weeks or months in the future. According to the case files, the appointment line employees were typically able to set up acceptable appointments for the beneficiaries immediately. Documentation also showed that complaints about inattentive staff in MTF inpatient settings apparently led to special training on the importance of being responsive to patient requests.
section_title: Private Sector Feedback Approaches
paragraphs: To compare DOD approaches to beneficiary feedback with those of the private sector, we interviewed representatives from two health care industry accreditation bodies—the Joint Commission on the Accreditation of Healthcare Organizations (JCAHO) and the National Committee for Quality Assurance—and obtained and reviewed copies of their accreditation standards regarding customer surveys and handling customer comments. We also reviewed the Vice President’s National Performance Review report describing the use of customer complaints by successful companies throughout the private sector and the applicability of such practices to government agencies. Further, we discussed customer surveys and comment tracking with representatives of two private sector health care providers—Kaiser Permanente, a large commercial health maintenance organization, and Inova Health System, a Northern Virginia hospital chain—and obtained documents describing the methods these companies use to track, categorize, and report comments from their customers. Private sector health care accreditation organizations require plans to have procedures for handling appeals of health care decisions, though we did not examine these appeals processes or compare them with those in place under TRICARE.
section_title: DOD Surveys
subsections: section_title: Annual Health Care Survey
paragraphs: The Health Care Survey of DOD Beneficiaries (referred to in this report as the annual survey) has six sections:
Use and source of care. This section asks beneficiaries 22 questions about annual visits, nights spent in a hospital, care sources, and insurance coverage.
Familiarity with benefits. This section contains 13 questions about whether beneficiaries have a source of information for various aspects of their health care benefit.
Health status. This section contains 36 questions, widely used and validated in the private sector, that measure distinct aspects of physical and emotional health.
Access to care. This section contains 25 questions that look at how easily beneficiaries enter the health care system (process measures) and whether they receive necessary care (outcome measures).
Satisfaction with care. This section contains 54 questions about overall satisfaction with care received at military and civilian facilities, and satisfaction with specific aspects of the care.
Demographic information. This section asks about age, education, gender, ethnicity and race, beneficiary group, and length of time in residence as well as other factors important to explaining health-related behaviors and opinions.
The annual survey was designed by a working group composed of survey experts from Health Affairs, each of the three services, and a representative from the Defense Manpower Data Center. The questions and scales used in the annual survey were developed on the basis of a review of private sector surveys that had been extensively tested for reliability and validity. The survey is mailed to a random sample of beneficiaries selected from catchment areas in the United States, overseas, and in noncatchment areas. The 1996 annual survey was mailed to a sample population of 156,838 adult beneficiaries eligible for MHS health care. The survey sample was composed of the following beneficiary types: active duty, active duty family members, retirees under age 65, retirees aged 65 or older, retiree family members under age 65, and retiree family members aged 65 or older. Beneficiaries were included in the sample regardless of whether they were users of military health care—either MTF care or DOD-funded civilian care.
Health Affairs has conducted the annual survey three times, at about 16- to 18-month intervals. The first survey was conducted in late 1994 and early 1995. Because it was conducted just before TRICARE started, it established a baseline against which changes in beneficiaries’ ratings of their health care could be tracked following TRICARE’s implementation. Questions on TRICARE Prime were added to the 1996 and the 1997 survey instruments to (1) gauge how beneficiaries perceive the program and (2) compare responses of beneficiaries enrolled in TRICARE Prime and those who are not.
Health Affairs sends out several reports of the annual survey results. Each TRICARE region receives one report that contains that region’s results by catchment area and by beneficiary group. Health Affairs sends each regional report to the lead agent, who is then responsible for distributing the results to the MTFs in that region. According to DOD officials, it is important to get the information to the local level where local officials can use the information to make improvements. Also, Health Affairs sends to each service Surgeon General a summary-level report that includes results for each of that service’s MTFs.
Health Affairs uses annual survey results as measures, along with a wide variety of other measures, in its MHS Performance Report Cards and in its Annual Quality Management Reports. The report cards, which provide MTF commanders with data on their facility’s health care delivery performance, measure five areas: access, quality, utilization, health behaviors, and health status. Annual survey results that appear in the report card include three measures of beneficiary satisfaction: access to appointments, access to system resources, and quality. According to DOD officials, the report card is one way to convert certain annual survey results to a catchment area score. Annual Quality Management Reports are assessments of quality across the system and also use the annual survey results.
DOD’s summary of its 1994-95 and 1996 annual survey results is broken out by different beneficiary types. One set of results consists of responses from active duty family members and a second, retirees and their family members. DOD officials told us that the summary they provided us does not include active duty personnel responses because the summary’s focus was on beneficiaries with a choice in where they obtain health care services, a choice that active duty personnel do not have. The summary data that DOD provided also distinguish between regions with TRICARE and those without. Regions with TRICARE are defined as those that had had TRICARE in place for a sufficiently long period at the time of the 1996 survey.
section_title: Monthly Military Hospital Care Survey
paragraphs: The Customer Satisfaction Survey (referred to in this report as the MTF outpatient survey) measures patient satisfaction with the effectiveness and efficiency of a recent, specified MTF outpatient visit. According to Health Affairs officials, this survey is intended to provide MTF Commanders and headquarters levels with quick, frequent, civilian-benchmarked feedback on the satisfaction of beneficiaries with their visits to MTF outpatient clinics. The survey asks about the patients’ satisfaction with their experience both in obtaining the appointment and during the appointment. According to DOD officials, this systemwide survey will replace most of the ad hoc surveys currently being done locally at MTFs. DOD officials said that a mail survey of this type is more reliable than surveys handed out to patients in the MTF clinics.
DOD contracted with a health services research organization to design and conduct the MTF outpatient survey. DOD’s contractor mails out surveys each month to patients who received outpatient care at clinics that have more than 200 outpatient visits per month. Over the course of each year, the survey will be mailed to 200 patients at each of about 2,100 clinics. The actual number of surveys mailed for April 1997 appointments was 52,642. Each month, MTFs forward patient appointment data to the contractor, who prepares a random sample of names and mails questionnaires directly to the patients, 30 to 50 days after the appointment. The questionnaire is customized to the date, doctor, and clinic of the appointment; asks 17 multiple choice questions about the visit; and allows for written comments. The contractor sends these written comments directly to the MTF Commander, without analysis by the contractor. Patients mail the completed questionnaires directly to the contractor, who produces reports of that month’s results as well as each clinic’s average results for the past 3 months.
Health Affairs distributes a number of different reports of the results of the monthly outpatient surveys. The contractor reports survey results at both MTF and individual clinic levels to MTFs on a monthly basis. These reports provide a “rolling” picture of the past 3 months’ data. The clinic-level report compares each clinic with itself during the previous reporting period as well as with other clinics within the MTF, peer clinics at other MTFs, and civilian HMOs. The MTF-level report compares each MTF with itself during the previous reporting period as well as with other MTFs within the same service, MHS-wide averages, and civilian HMOs. The contractor also prepares quarterly summary—“roll-up”—reports for lead agents, Surgeons General, other service command entities, and Health Affairs within 45 to 60 days of the end of each quarter. All of these reports are standardized and one page long; they report on customer satisfaction with access, quality, and staff interaction.
Figures II.1, II.2, and II.3 show each region’s results and comparison scores for civilian HMOs in the same geographic areas. April/May/June 1997 was the first 3-month period for which survey results were available. During this period, the Central, Heartland, Mid-Atlantic, Northeast, and Pacific-Alaska regions did not yet have TRICARE.
Figure II.1: Monthly MTF Outpatient Visit Survey Results for Satisfaction With Access Compared With Civilian HMO Benchmarks “Satisfaction with access” focuses on individuals’ satisfaction with referral for specialty care, access to medical care, office wait time, time to return phone calls, ease of making phone appointments, and appointment wait time.
subsections: section_title: Figure II.2: Monthly MTF Outpatient Visit Survey Results for Satisfaction With Quality Compared With Civilian HMO Benchmarks
paragraphs: “Satisfaction with quality” focuses on individuals’ satisfaction with overall quality of care received, how well care met needs, thoroughness of treatment, how much the individual was helped, and explanations of procedures and tests.
section_title: Figure II.3: Monthly MTF Outpatient Visit Survey Results for Satisfaction With Staff Interaction Compared With Civilian HMO Benchmarks
paragraphs: MTF Visit Satisfaction With Staff Interaction “Satisfaction with staff interaction” focuses on individuals’ satisfaction with personal interest in the patient, advice on ways to avoid illness/stay healthy, amount of time with doctor and staff, attention to what patients said, and friendliness and courtesy of staff.
section_title: TRICARE Prime Enrollee Satisfaction Survey
paragraphs: Health Affairs’ TRICARE Marketing Office commissioned a telephone survey of TRICARE Prime enrollees who were enrolled in the program on September 30, 1996. The survey consisted of 7,728 interviews conducted between October 18 and December 8, 1996, and covered five TRICARE regions: Golden Gate, Northwest, Pacific, Southern California, and Southwest. The survey addressed a number of issues related to enrollees’ understanding of TRICARE Prime, satisfaction, and reenrollment intentions. TRICARE Prime-specific questions from this survey have been incorporated into the ongoing annual surveys.
section_title: Other DOD Efforts to Solicit Beneficiary Feedback
paragraphs: Health Affairs also conducts other surveys to solicit beneficiary feedback on various topics unrelated to satisfaction with health care:
The DOD Survey of Health Related Behaviors Among Military Personnel is carried out about every 3 years to collect worldwide data from active duty personnel on drug and alcohol abuse and other health-related behaviors.
The Health Enrollment Assessment Review, a questionnaire completed by patients as they enroll in TRICARE Prime, is used to identify high-volume care users and their chronic conditions, assess the need for preventive services, and motivate behavioral change.
The MHS User Survey is conducted twice each year to collect data on the health care sources of DOD’s U.S. beneficiaries for use in developing capitation budgets.
DOD has also used focus groups to obtain beneficiary feedback on TRICARE’s success. From October to December 1995, DOD hosted a series of focus groups in the Southwest and Northwest regions to test beneficiaries’ knowledge of TRICARE at the time it was introduced in these regions and, thus, the success of its beneficiary education and marketing efforts. DOD officials told us the results of these focus groups helped establish a baseline of beneficiary perceptions of and attitudes toward the program to help in designing future TRICARE marketing efforts.
In November 1996, Health Affairs issued a policy designed to standardize surveys across the MHS, ensure that all survey information is generalizable, allow comparisons with civilian plans, and minimize the time and paperwork burden on beneficiaries. In instituting this policy, Health Affairs intended to avoid surveys that produce invalid results and results that cannot be compared across MHS or with those of civilian health care plan surveys. According to the policy, entities under MHS authority—MTFs, offices of service Surgeons General, and managed care support contractors—must obtain approval from Health Affairs before conducting their own surveys. According to Health Affairs officials, however, MTFs and other entities can continue to gather information from beneficiaries as long as they use open-ended questions and do not attempt to generalize the results. In fact, Health Affairs officials told us that a feedback or complaint system that allows people to describe their concerns in their own words is a useful tool for MTFs to use to identify particular areas of concern to beneficiaries.
section_title: Beneficiary-Initiated Feedback
paragraphs: Beneficiaries make complaints and give compliments directly to many offices throughout the MHS, using several different methods. Beneficiaries contact Health Affairs, TSO, the Surgeons General, Inspectors General, lead agents, and MTFs. And the managed care support contractors receive such feedback in their headquarters offices, regional offices, and local contractor offices. Beneficiaries also express concerns to associations representing beneficiaries’ interests.
Beneficiaries communicate their concerns in a variety of ways. For example, beneficiaries communicate orally through phone calls and in person, as well as in written form through letters, electronic mail messages, faxes, and filling out comment forms at MTF clinics. One special category of letters received within MHS is priority correspondence—letters regarding beneficiary concerns referred from the White House, the Congress, the Secretary of Defense, or the three service Secretaries. DOD requires managed care support contractors to have a toll-free phone line for beneficiaries, and much of the feedback that the contractors receive comes in over these lines.
Officials throughout DOD told us that they consider it important that complaints be resolved at as low a level as possible. They said that people who register dissatisfaction should not be “given the runaround” in the process of trying to find someone to listen to and deal with their complaint. This emphasis is consistent with the National Performance Review report on the importance of empowering front-line employees to provide “on-the-spot, just-in-time resolution of  problems.”
subsections: section_title: Military Treatment Facilities
paragraphs: Each MTF we visited had procedures in place enabling beneficiaries to comment directly to MTF staff while at the facility. Much of this feedback is in the form of oral comments made directly to MTF staff members or through comment cards or forms beneficiaries fill out. MTF officials told us that they also receive comments through phone calls, letters, and electronic mail.
The MTFs differed in their approaches to handling beneficiary comments. Some MTFs had designated personnel throughout the facility who served as patient representatives or patient advocates. These staff were tasked with receiving beneficiary comments about their own clinic or department. MTFs with patient representatives at this level also had a senior patient representative whose job was to be available to any beneficiaries with comments, whether concerning a particular facility area, the whole facility, or military health care in general. Other facilities did not have formally designated patient representatives at clinics or departments but, instead, had a single patient representative office where beneficiaries could go to make comments.
Procedures for documenting beneficiary feedback also varied among the MTFs visited. For example, some MTFs entered everything the patient representatives received into a central patient feedback database, and some also kept hard copy documentation of the comments that came in. Another MTF had a system that required oral comments to be documented in writing. Staff kept hard copies of both those comments and the ones that came in through comment cards but did not enter the comments into a database. Another MTF did little or no documentation of oral or written beneficiary feedback. The head patient representative at that facility said that he did not have enough time to both handle patient concerns and prepare documentation, so he opted to spend time with patients instead of doing paperwork.
Also, wide differences existed in how much the MTFs analyzed beneficiary feedback for trends. For example, some used the categorized patient feedback in their central database to prepare regular feedback trend reports. They analyzed how the number of complaints per type changed over time and which hospital areas were generating more complaints. Other MTFs did little or no formal trend analysis of beneficiary comments, although staff members at these facilities told us that they relied on their experience with feedback at the facility over time to notice trends.
We also found variation in how the feedback tracking results were reported to MTF management or to others in the facility. For example, some MTFs distributed formal feedback reports on trends to senior MTF management, as well as reports about department-level feedback to supervisory staff in various areas of the facility. At another facility, however, internal reporting of patient feedback consisted of oral input from the head patient representative to a senior management committee, with no supporting documentation.
MTF patient representatives told us that these systems constitute the formal structure that is in place to receive feedback, but that other avenues exist. For example, they said that beneficiaries can speak to staff members throughout the MTF if they have concerns and that many do. People can speak with their doctor or other staff members in the various clinics, or they can go to different parts of the MTF’s administrative structure, such as the managed care office or the MTF commander’s office. Even at MTFs with extensive feedback documentation and trend analysis systems, staff members noted that some of the feedback that comes in to staff other than patient representatives does not make it into the MTFs’ systems. For example, one officer in an MTF command section told us that he hears beneficiary complaints and handles them but does not typically report what he hears to the central MTF patient representative office that maintains a database of patient complaints.
MTF officials told us that they do not systematically report most beneficiary feedback to Health Affairs or the service Surgeons General. Officials at MTFs and other MHS offices told us that MTF staff are expected to resolve problems that arise, whether identified through beneficiary complaints or not. Health Affairs and the service Surgeons General expect to be brought in only to handle issues that the MTF cannot. While such issues do get referred to the higher levels, the officials told us that information about problems solved locally normally do not. The exception was regular reporting of contractor-related issues to lead agents by MTFs.
One exception to the lack of systematic reporting of beneficiary feedback is found in the Southern California Region. MTFs in that region are part of a program led by the lead agent to systematically report to the lead agent certain types of beneficiary comments. Lead agent officials told us that MTFs in the region have been asked to send to the lead agent the beneficiary complaints made to the MTF concerning the managed care support contractor. For example, if a beneficiary tells the patient representative about an enrollment card problem or a problem getting contractor network care, the MTF will send a copy of the complaint to the lead agent, where it will be centrally tracked, as well as notify the managed care support contractor of the problem. The regional managed care support contractor has also been asked to do the same for MTF-related complaints made to it. Lead agent officials told us that they hope to expand this project to include all regional complaints in the future.
Further, MTFs have systems in place for documenting and reporting clinical health care quality issues, some of which come to light through patient complaints. To maintain JCAHO accreditation, MTFs must have systems in place to track clinical care quality issues. MTF officials told us that such complaints, along with other clinical quality issues identified at the facility, are documented and analyzed for trends and become the subject of detailed review by special committees as well as by MTF risk management and legal office staff.
section_title: Lead Agents
paragraphs: Some beneficiary concerns come directly to the lead agents through letters or phone calls, others come through oral or written reports from regional MTF staffs and the contractors, and still others are referred to the lead agent by other offices.
The three lead agents we visited had issues tracking systems that tracked, among other things, concerns that came to light through complaints from beneficiaries. The Southeast Region lead agent maintained a central log of complaints that came directly to the lead agent as well as complaints forwarded to the lead agent by other DOD offices (including priority correspondence complaints); complaints about MTFs forwarded by the region’s managed care support contractor; and certain complaints received by MTFs in the region. Southeast Region officials told us that beneficiary complaints about managed care support contractor functions were frequently the subject of discussion during regular telephone meetings between contracting officer’s technical representatives (COTR) at the region’s MTFs and lead agent staff. Lead agent officials in the Southeast Region also told us that they used the system to track issues to ensure they were being properly addressed and resolved, but that they did not organize the issues by category or analyze for trends over time.
In the Southern California Region, the lead agent had implemented a system specifically to track complaints. The system tracked complaints (1) received by MTFs in the region if they concerned the managed care support contractor, (2) received by the managed care support contractor if they concerned an MTF, and (3) received by the lead agent directly. The lead agent staff tracked and analyzed for trends the complaints in this system by category of complaint. Lead agent staff told us that they want to expand the system to include more types of complaints in the future.
The Southwest Region lead agent asked the COTRs at the MTFs to perform a number of contractor oversight functions and to report the results monthly to the lead agent. Some of the issues that the COTR reports raised were related to beneficiary complaints. The lead agent staff then compiled the issues raised by the various monthly reports of COTRs into a single letter to the region’s managed care support contractor asking for issue-by-issue responses.
Lead agent officials in the three regions we visited told us that they use the complaints that they receive to identify and proactively deal with issues before they become worse, as well as to monitor overall TRICARE performance in their region. Lead agent staff said that when their beneficiary complaint tracking indicates a possible problem, they discuss the issue with the managed care support contractor, MTF staff members, or both to help identify the cause and discuss possible solutions. Lead agent staff also said that by tracking complaints they are better able to identify the root causes of problems in ways that surveys are not, although surveys can, on the other hand, indicate how well DOD is fixing the problems identified through complaints.
Lead agent officials told us they did not systematically report beneficiary feedback-related issues to Health Affairs or the Surgeons General. They said, however, that a number of regularly scheduled video, telephone, and face-to-face meetings take place with Health Affairs, service Surgeons General, and contractor staff and that at these meetings some issues discussed may have emanated from beneficiary comments. But, whether a particular issue is discussed at these meetings is generally the result of a decision made by an individual that the issue warrants the other participants’ attention.
section_title: Other DOD Offices
paragraphs: Some issues communicated to Health Affairs, the service Surgeons General, TSO, and the service Inspectors General come directly from beneficiaries through letters and phone calls. Others are referred through other means, such as priority correspondence, which is referred from congressional and other offices. Some of these complaints are from beneficiaries who have tried to get a problem handled at a lower level, such as an MTF, but were not satisfied. Others are from beneficiaries who simultaneously send their complaint letters to as many places as possible.
Health Affairs, service Surgeons General, and TSO officials told us their organizations have their own tracking systems for beneficiary concerns that come to the attention of their respective offices. Health Affairs officials told us they enter all beneficiary feedback they receive—both directly sent and referred—into a tracking system that notes the receipt date, which staffer was assigned to handle the concern, the response due date, and a short issue description. Officials told us that the system’s purpose is to track response timeliness and not to track or establish trends in issues by category. Reports from the Health Affairs tracking system show that the system’s issue descriptions are not specific enough for tracking or identifying trends in issues by category.
Service Surgeon General office officials described similar systems for tracking the timeliness of the offices’ responses to beneficiary feedback.
Also, staff from the Navy told us that they had begun to track selected beneficiary concerns by type.
TSO also tracks beneficiary issues. According to officials there, a large number of phone calls and letters come into that office and are centrally tracked in a computer database. However, officials also said that the categorization system they use puts issues only into broad categories—such as “claims” or “policy questions”—which limits the usefulness of the system for tracking issues by type.
Beneficiaries can also register their complaints with the Army or Navy Inspector General. These Inspectors General deal mostly with misconduct allegations but, on occasion, they receive health care service-related complaints. Officials at Inspector General offices told us they track beneficiary concerns by nature of issue but report health care issues only on an ad hoc basis. One official at an Inspector General office told us that his office reports an issue to the service Surgeon General only when it appears significant and representative of a systemic issue.
section_title: Managed Care Support Contractors
paragraphs: DOD requires contractors to document and report statistics on the nature and number of beneficiary contacts—including, but not limited to, beneficiary complaints—as well as on the contractors’ response times to beneficiary inquiries. For example, DOD requires monthly contractor reports to TSO on all phone calls received, local contractor offices’ walk-in traffic, and how long contractors take to respond to priority correspondence items. For walk-in activity and phone calls, DOD requires the reason for the person’s visit or call, but not identification of which calls or visits involved complaints from beneficiaries. That is, a reason category called “enrollment” would include calls or visits from beneficiaries who contacted the contractor service center to enroll in TRICARE Prime as well as those who expressed a complaint about some aspect of the enrollment process.
Contractors are also required to report quality of health care issues that they handle—and their actions in response to the issues—to their lead agents. These quality of care issues include both potential quality issues and issues that the contractor determines to have already become quality of care problems. These issues may be reported by hospital staff; identified through review of quality of care indicators, such as incidents of post-operative infections; and raised by beneficiaries through complaints.
In addition to the reports required by managed care support contracts, contractors also gather feedback-related information for their own use. One managed care support contractor’s representatives told us that the contractor categorizes all the complaints it receives, whether over the phone, through the mail, or in person. The contractor also analyzes the data to identify trends and reports the results throughout the organization, including to senior management. Another managed care support contractor’s representative told us his organization similarly tracks complaints received from beneficiaries through calls to the contractor’s toll-free telephone number, as well as complaints raised with the contractor’s field staff when they determine the complaints to be serious enough to warrant entering into the tracking system. The representatives told us that their beneficiary feedback tracking systems are similar to the systems used by their parent companies’ civilian health care operations. The contractors do not systematically report the results of their internal tracking to DOD, although issues that the contractors discover through their own systems may be discussed in ad hoc letters to DOD.
section_title: Comments From the Department of Defense
paragraphs: The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 37050 Washington, DC  20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (202) 512-6061, or TDD (202) 512-2537.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.