Defense Travel System: Estimated Savings Are Questionable and Improvements Are Needed to Ensure Functionality and Increase Utilization
Published: 2006-11-16T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In 1995, the Department of Defense (DOD) began an effort to implement a standard departmentwide travel system. The Defense Travel System (DTS) is envisioned as DOD's standard end-to-end travel system. This testimony is based on GAO's September 2006 related report. Today's testimony highlights GAO's key findings with regard to the following objectives: (1) Were the two key assumptions made in the September 2003 economic analysis reasonable? (2) Was DOD taking action to ensure full utilization of DTS and gathering the data needed to monitor DTS utilization? and (3) Has DOD resolved several functional problems associated with weak system requirements and testing? To address these objectives, GAO (1) reviewed the September 2003 DTS economic analysis, (2) analyzed DTS utilization data, and (3) analyzed DTS flight information.
section_title: What GAO Found
paragraphs: GAO's analysis of the September 2003 DTS economic analysis found that the two key assumptions used to estimate annual net savings were not based on reliable information. Two cost components represent the majority of the over $56 million in estimated net savings--personnel savings and reduced commercial travel office (CTO) fees. In regard to the personnel savings, GAO's analysis found that the $24.2 million of personnel savings related to the Air Force and the Navy were not supported. Air Force and Navy DTS program officials stated that they did not anticipate a reduction in the number of personnel, but rather the shifting of staff from the travel function to other functions. The Naval Cost Analysis Division stated that the Navy will not realize any tangible personnel cost savings from the implementation of DTS. In regard to the CTO fees, the economic analysis assumed that 70 percent of all DTS airline tickets would either require no intervention or minimal intervention from the CTOs, resulting in an estimated annual net savings of $31 million. However, the sole support provided by the DTS program office was an article in a trade industry publication. The article was not based on information related to DTS, but rather on the experience of one private sector company. Furthermore, the economic analysis was not prepared in accordance with guidance prescribed by the Office of Management and Budget and DOD. DOD guidance stated that the life-cycle cost estimates should be verified by an independent party, but this did not occur. The economic analysis did not undertake an assessment of the effects of the uncertainty inherent in the estimates of benefits and costs. Because an economic analysis uses estimates and assumptions, it is critical that the imprecision in both the underlying data and assumptions be understood. Such an assessment is referred to as a sensitivity analysis. DOD acknowledged that DTS is not being used to the fullest extent possible, but lacks comprehensive data to effectively monitor its utilization. DOD's utilization data are based on a model that was developed in calendar year 2003. However, the model has not been completely updated to reflect actual DTS usage. The lack of accurate utilization data hinders management's ability to monitor progress toward the DOD vision of DTS as the standard travel system. GAO also found that the military services have initiated actions that are aimed at increasing the utilization of DTS. Finally, GAO found that DTS still has not addressed the underlying problems associated with weak requirements management and system testing. While DOD has acted to address concerns GAO previously raised, GAO found that DTS's requirements are still ambiguous and conflicting. For example, DTS displaying up to 25 flights for each inquiry is questionable because it is unclear whether this is a valid requirement. Until DOD improves DTS's requirements management practices, the department will not have reasonable assurance that DTS can provide the intended functionality.

section_title: Letter
paragraphs: Thank you for the opportunity to discuss our recent report related to  problems encountered by the Department of Defense (DOD) in its efforts  to successfully implement the Defense Travel System (DTS). As you know,  DOD envisions DTS as the department’s standard end-to-end travel  system. The department estimates that DTS will be fully deployed at all  11,000 intended locations during fiscal year 2007. The September 2003  economic analysis noted that DTS, when fully implemented, would result  in annual net savings of over $56 million. The economic analysis noted that  savings would be realized by the department during fiscal years 2009–2016.  In December 2003, the department’s Chief Information Officer approved a  DTS funding level of approximately $564 million. Of this amount, the  contract for the design, development, and deployment of DTS was for  about $264 million. The remaining costs are associated with areas such as  the operation and maintenance of DTS, operation of the Program  Management Office-Defense Travel System (PMO-DTS), the voucher  payment process, and management and oversight of the numerous  contracted commercial travel offices (CTO).
My testimony today is based on our September 2006 report, which  followed up on our September 2005 testimony and January 2006 report.  One of the major findings in our previous work was that DOD did not have  reasonable assurance that flight information was properly displayed for  DOD travelers because the department failed to properly test the system  interfaces through which the data are accessed for display. We further  noted that the continued use of the existing legacy travel systems at  locations where DTS has been deployed results in underutilization of DTS  and reduces the envisioned savings.
Today, I will highlight three key findings from our September 2006 report:    Two key assumptions related to the estimated cost savings in the  September 2003 DTS economic analysis were not reasonable. DOD  strongly objected to this finding, and I will discuss why we continue to  believe that our finding is accurate.
The department did not have quantitative metrics to measure the extent to  which DTS is actually being used.
DOD still has not addressed several functional problems associated with  weak requirements management and system testing.
Finally, I will discuss our recommendations to improve the department’s  management and oversight of DTS.
Our work focused on the validity of the assumptions that were the  principal drivers of DOD’s net annual estimated savings of over   $56 million. We determined that the data were sufficiently reliable for this  purpose. We did not review the accuracy and reliability of the specific  dollar amounts shown in the September 2003 economic analysis. To  address our objectives we also (1) reviewed the September 2003 economic  analysis and met with cognizant officials, (2) analyzed DTS utilization data  and obtained an overview of the method and data used by DTS program  officials to report the rate of DTS utilization for the various DOD  components, and (3) analyzed DTS flight information. Our work was  performed from October 2005 through July 2006 in accordance with U.S.  generally accepted government auditing standards. Further details on our  scope and methodology are included in our September 2006 report.
subsections: section_title: Validity of DTS Economic Analysis Questionable
paragraphs: In September 2003, DOD finalized its economic analysis for DTS in  preparation for a milestone decision review. The highlights of the  economic analysis are shown in table 1. In December 2003, the DOD Chief  Information Officer granted approval for DTS to proceed with full  implementation throughout the department.
Our analysis of the September 2003 DTS economic analysis found that two  key assumptions used to estimate cost savings were not based on reliable  information. Consequently, the economic analysis did not serve to help  ensure that the funds invested in DTS were used in an efficient and  effective manner. Two primary areas—personnel savings and reduced  CTO fees—represented the majority of the over $56 million of estimated  annual net savings DTS was expected to realize. However, the estimates  used to generate these savings were unreliable. Further, DOD did not  effectively implement the policies relating to developing economic  analyses for programs such as DTS. Effective implementation of these  policies should have highlighted the problems that we found and allowed  for appropriate adjustments so that the economic analysis could have  served as a useful management tool in making funding decisions related to  DTS—which is the primary purpose of this analysis. While the  department’s system acquisition criteria do not require that a new  economic analysis be prepared, the department’s business system  investment management structure provides an opportunity for DOD  management to assess whether DTS is meeting its planned cost, schedule,  and functionality goals.
subsections: section_title: Personnel Savings Are Unrealistic
paragraphs: The economic analysis estimated that the annual personnel savings was  over $54 million, as shown in table 2.
As shown in table 2, approximately 45 percent of the estimated savings, or  $24.2 million, was attributable to the Air Force and Navy. The assumption  behind the personnel savings computation was that there would be less  manual intervention in the processing of travel vouchers for payment, and  therefore fewer staff would be needed. However, based on our discussions  with Air Force and Navy DTS program officials, it is questionable as to  how the estimated savings will be achieved. Air Force and Navy DTS  program officials stated that they did not anticipate a reduction in the  number of personnel with the full implementation of DTS, but rather the  shifting of staff to other functions. According to DOD officials responsible  for reviewing economic analyses, while shifting personnel to other  functions is considered a benefit, it should be considered an intangible  benefit rather than tangible dollar savings since the shifting of personnel  does not result in a reduction of DOD expenditures. Also, as part of the  Navy’s overall evaluation of the economic analysis, program officials  stated that “the Navy has not identified, and conceivably will not  recommend, any personnel billets for reduction.” Finally, the Naval Cost  Analysis Division (NCAD) October 2003 report on the economic analysis  noted that it could not validate approximately 40 percent of the Navy’s  total costs, including personnel costs, in the DTS life-cycle cost estimates  because credible supporting documentation was lacking. The report also  noted that the PMO-DTS used unsound methodologies in preparing the  DTS economic analysis.
The extent of personnel savings for the Army and defense agencies, which  are reported as $16 million and $6.3 million respectively, is also unclear.  The Army and many defense agencies use the Defense Finance and  Accounting Service (DFAS) to process their travel vouchers, so the  personnel savings for the Army and the defense agencies were primarily  related to reductions in DFAS’s costs. In discussions with DFAS officials,  they were unable to estimate the actual personnel savings that would  result since they did not know (1) the number of personnel, like those at  the Air Force and Navy, that would simply be transferred to other DFAS  functions or (2) the number of personnel that could be used to avoid  additional hiring. For example, DFAS expects that some of the individuals  assigned to support the travel function could be moved to support its  ePayroll program. Since these positions would need to be filled regardless  of whether the travel function is reduced, transferring personnel from  travel to ePayroll would reduce DOD’s overall costs since DFAS would not  have to hire additional individuals.
DOD strongly objected to our finding that the personnel savings are  unrealistic. In its written comments, the department stated that it is facing  an enormous challenge and continues to identify efficiencies and eliminate  redundancies to help leverage available funds. We fully recognize that the  department is attempting to improve the efficiency and effectiveness of its  business operations. The Comptroller General of the United States  testified in August 2006 that increased commitment by the department to  address DOD’s numerous challenges represents an improvement over past  efforts.
The fact remains, however, that the results of an economic analysis are  intended to help management decide if future investments in a given  endeavor are worthwhile. In order to provide management with this  information, it is imperative that the underlying assumptions in an  economic analysis be supported by valid assumptions. The September  2003 economic analysis noted that personnel savings of $54.1 million  would be realized by the department annually for fiscal years 2009 through  2016. However, based on our review and analysis of documentation and  discussion with department personnel, we found that the underlying  assumptions in support of the $54.1 million were not valid, particularly in  regard to the amounts estimated for the Navy and Air Force. For example,  we agree with the statements of DOD officials who indicated that the  shifting of personnel to other functions cannot be counted towards  tangible dollar savings, since such actions do not result in a reduction of  DOD expenditures. Moreover, the department did not provide any new  data or related documentation in its comments that were counter to our  finding. As a result of these factors, we continue to believe that the  estimated annual personnel savings of $54.1 million are unrealistic.
section_title: Savings Associated with Reduction of CTO Fees Are Unknown
paragraphs: According to the September 2003 economic analysis, DOD expected to  realize annual net savings of $31 million through reduced fees paid to the  CTOs because the successful implementation of DTS would enable the  majority of airline tickets to be acquired with either no or minimal  intervention by the CTOs. These are commonly referred to as “no touch”  transactions. However, DOD did not have a sufficient basis to estimate the  number of transactions that would be considered “no touch” since the   (1) estimated percentage of transactions that can be processed using the  “no touch” was not supported and (2) analysis did not properly consider  the effects of components that use management fees, rather than  transaction fees, to compensate the CTOs for services provided. The  weaknesses we identified with the estimating process raise serious  questions as to whether DOD will realize substantial portions of the  estimated annual net savings of $31 million.
subsections: section_title: “No Touch” Transaction Volume Estimates Are Not Supported
paragraphs: DOD arrived at the $31 million of annual savings in CTO fees by estimating  that 70 percent of all DTS airline tickets would be considered “no touch”  and then multiplying these tickets by the savings per ticket in CTO fees.  However, a fundamental flaw in this analysis was that the 70 percent  assumption had no solid basis. We requested, but the PMO-DTS could not  provide, any analysis of travel data to support the assertion. Rather, the  sole support provided by the PMO-DTS was an article in a travel industry  trade publication. The article was not based on information related to  DTS, but rather on the experience of one private sector company.
The economic analysis assumed that DOD could save about $13.50 per “no  touch” ticket. Since that analysis, DOD has awarded one contract that  specifically prices transactions using the same model as that envisioned by  the economic analysis. This contract applies to the Defense Travel Region  6 travel area. During calendar year 2005, the difference in fees for “no  touch” transactions and the transactions supported by the current process  averaged between $10 and $12, depending on when the fees were incurred  because the contract rates changed during 2005. In analyzing travel  voucher data for Region 6 for calendar year 2005, we found that the  reported “no touch” rate was, at best 47 percent—far less than the 70  percent envisioned in the economic analysis.
PMO-DTS program officials stated they are uncertain as to why the  anticipated 70 percent “no touch” was not being achieved. According to  PMO-DTS program officials, this could be attributed, in part, to the DOD  travelers being uncomfortable with the system and with making  reservations without using a CTO. Although this may be one reason, other  factors may also affect the expected “no touch” fee. For example, we were  informed that determining the airline availability and making the  associated reservation can be accomplished, in most cases, rather easily.  However, obtaining information related to hotels and rental cars and  making the associated reservation can be more problematic because of the  limitations in the data that DTS is able to obtain from its commercial  sources. Accordingly, while a traveler may be able to make a “no touch”  reservation for the airline portion of the trip, the individual may need to  contact the CTO in order to make hotel or rental car reservations. When  this occurs, rather than paying a “no touch” fee to the CTO, DOD ends up  paying a higher fee, which eliminates the savings estimated in the  economic analysis.
The economic analysis assumed that (1) DOD would be able to modify the  existing CTO contracts to achieve a substantial reduction in fees paid to a  CTO when DTS was fully implemented across the department and (2) all  services would use the fee structure called for in the new CTO contracts.  The first part of the assumption is supported by results of the CTO  contract for DOD Region 6 travel. The fees for the DTS “no touch”  transactions were at least $10 less than if a CTO was involved in the  transactions. However, to date, the department has experienced difficulty  in awarding new contracts with the lower fee structure. On May 10, 2006,  the department announced the cancellation of the solicitation for a new  contract. According to the department, it decided that the solicitation  needed to be rewritten based on feedback from travel industry  representatives at a March 28, 2006, conference. The department  acknowledged that the “DTS office realized its solicitation didn’t reflect  what travel agency services it actually needed.” The department would  not say how the solicitation would be refined, citing the sensitivity of the  procurement process. The department also noted that the new solicitation  would be released soon, but provided no specific date.
section_title: Navy Impact of CTO Management Fees Not Adequately Considered
paragraphs: The economic analysis assumed that the Navy would save about $7.5  million, almost 25 percent, of the total savings related to CTO fees once  DTS is fully deployed. The economic analysis averaged the CTO fees paid  by the Army, the Air Force, and the Marine Corps—which amounted to  about $18.71 per transaction—to compute the savings in Navy CTO fees.  Using these data, the assumption was made in the economic analysis that  a fee of $5.25 would be assessed for each ticket, resulting in an average  savings of $13.46 per ticket for the Navy ($18.71 minus $5.25). While this  approach may be valid for the organizations that pay individual CTO fees,  it may not be representative for organizations such as the Navy that pay a  management fee. The management fee charged the Navy is the same  regardless of the involvement of the CTO—therefore, the reduced “no  touch” fee would not apply.
We were informed by Navy DTS program officials that they were  considering continuing the use of management fees after DTS is fully  implemented. According to Navy DTS program officials, they paid about  $14.5 million during fiscal year 2005 for CTO management fees, almost $19  per ticket for approximately 762,700 tickets issued. Accordingly, even if  the department arrives at a new CTO contract containing the new fee  structure or fees similar to those of Region 6, the estimated savings related  to CTO fees for the Navy will not be realized if the Navy continues to use  the management fee concept.
section_title: Effective Implementation of Existing Policies Should Have Identified Problems with the Economic Analysis
paragraphs: Effective implementation of DOD guidance would have detected the types  of problems discussed above and resulted in an economic analysis that  would have accomplished the stated objective of the process—to help  ensure that the funds invested in DTS were used efficiently and effectively.  DOD policy and OMB guidance require that an economic analysis be  based on facts and data and be explicit about the underlying assumptions  used to arrive at estimates of future benefits and costs. Since an economic  analysis deals with costs and benefits occurring in the future, assumptions  must be made to account for uncertainties. DOD policy recognizes this and  provides a systematic approach to the problem of choosing the best  method of allocating scarce resources to achieve a given objective.
A sound economic analysis recognizes that there are alternative ways to  meet a given objective and that each alternative requires certain resources  and produces certain results. The purpose of the economic analysis is to  give the decision maker insight into economic factors bearing on  accomplishing the objectives. Therefore, it is important to identify factors,  such as cost and performance risks and drivers, that can be used to  establish and defend priorities and resource allocations. The DTS  economic analysis did not comply with the DOD policy, and the  weaknesses we found should have been detected had the DOD policy been  effectively implemented. The PMO-DTS had adequate warning signs of the  potential problems associated with not following the OMB and DOD  guidance for developing an effective economic analysis. For example, as  noted earlier, the Air Force and Navy provided comments when the  economic analysis was being developed that the expected benefits being  claimed were unrealistic. Just removing the benefits associated with  personnel savings from the Air Force and Navy would have reduced the  overall estimated program cost savings by almost 45 percent. This would  have put increased pressure on the credibility of using a 70 percent “no  touch” utilization rate.
Specific examples of failures to effectively implement the DOD policy on  conducting economic analyses include the (1) DTS life-cycle cost  estimates portion of the economic analysis was not independently  validated as specified in DOD’s guidance and (2) September 2003 DTS  economic analysis did not undertake an assessment of the effects of the  uncertainty inherent in the estimates of benefits and costs, as required by  DOD and OMB guidance. Because an economic analysis uses estimates  and assumptions, it is critical that a sensitivity analysis be performed to  understand the effects of the imprecision in both underlying data and  modeling assumptions.
section_title: DTS Remains Underutilized by the Military Services
paragraphs: Our September 2005 testimony and January 2006 report noted the  challenge facing the department in attaining the anticipated DTS  utilization. While DOD has acknowledged the underutilization, we found  that, across DOD, the department does not have reasonable quantitative  metrics to measure the extent to which DTS is actually being used.  Presently, the reported DTS utilization is based on a DTS Voucher Analysis  Model that was developed in calendar year 2003 using estimated data, but  over the years has not been completely updated with actual data. While  the military services have initiated actions to help increase the utilization  of DTS, they pointed out that ineffective DTS training is a contributing  factor to the lower than expected usage rate by the military services.
subsections: section_title: Metrics to Measure DTS Utilization Are Inadequate
paragraphs: The DTS Voucher Analysis Model was prepared in calendar year 2003 and  based on airline ticket and voucher count data that were reported by the  military services and defense agencies, but the data were not verified or  validated. Furthermore, PMO-DTS officials acknowledged that the model  has not been completely updated with actual data as DTS continues to be  implemented at the 11,000 sites. We found that the Air Force is the only  military service that submits monthly metrics to the PMO-DTS officials for  their use in updating the DTS Voucher Analysis Model. Rather than  reporting utilization based on individual site system utilization data, the  PMO-DTS continues to rely on outdated information in the reporting of  DTS utilization to DOD management and the Congress. We have  previously reported that best business practices indicate that a key factor  of project management and oversight is the ability to effectively monitor  and evaluate a project’s actual performance against what was planned.
In order to perform this critical task, best business practices require the  adoption of quantitative metrics to help measure the effectiveness of a  business system implementation and to continually measure and monitor  results, such as system utilization. This lack of accurate and pertinent  utilization data hinders management’s ability to monitor its progress  toward the DOD vision of DTS as the standard travel system, as well as to  provide consistent and accurate data to Congress. With the shift of the  DTS program to the Business Transformation Agency (BTA), which now  makes DTS an enterprisewide endeavor, improved metrics and training  are essential if DTS is to be DOD’s standard, integrated, end-to-end travel  system for business travel.
DTS’s reported utilization rates for the period October 2005 through April  2006 averaged 53 percent for Army, 30 percent for Navy, and 39 percent  for Air Force. Because the PMO-DTS was not able to identify the total  number of travel vouchers that should have been processed through DTS  (total universe of travel vouchers), these utilization rates may be over- or  understated. PMO-DTS program officials confirmed that the reported  utilization data were not based on complete data because the department  did not have comprehensive information to identify the universe or the  total number of travel vouchers that should be processed through DTS.  PMO-DTS program and DTS military service officials agreed that the  actual DTS utilization rate should be calculated by comparing actual  vouchers being processed in DTS to the total universe of vouchers that  should be processed in DTS. The universe would exclude those travel  vouchers that cannot be processed through DTS, such as those related to  permanent change of station travel.
The Air Force was the only military service that attempted to obtain data  on (1) the actual travel vouchers processed through DTS and (2) those  travel vouchers that were eligible to be processed through DTS, but were  not. These data were site-specific. For example, during the month of  December 2005, the PMO-DTS reported that at Wright-Patterson Air Force  Base, 2,880 travel vouchers were processed by DTS, and the Air Force  reported that another 2,307 vouchers were processed through the legacy  system—the Reserve Travel System (RTS). Of those processed through  RTS, Air Force DTS program officials stated that 338 travel vouchers  should have been processed through DTS. DTS Air Force program officials  further stated that they submitted to the PMO-DTS the number of travel  vouchers processed through RTS each month. These data are used by the  PMO-DTS to update the DTS Voucher Analysis Model. However, neither  the Air Force nor the PMO-DTS have verified the accuracy and reliability  of the data. Therefore, the accuracy of the utilization rates reported for the  Air Force by the PMO-DTS is not known.
Because Army and Navy DTS program officials did not have the  information to identify the travel transactions that should have been  processed through DTS, the Army and Navy did not have a basis for  evaluating DTS utilization at their respective military locations and  activities. Furthermore, Navy DTS program officials indicated that the  utilization data that the PMO-DTS program officials reported for the Navy  were not accurate. According to Navy DTS program officials, the Navy’s  primary source of utilization data was the monthly metrics reports  provided by the PMO-DTS, but Navy DTS program officials questioned the  accuracy of the Navy utilization reports provided by the PMO-DTS.
section_title: DOD Has Taken Steps to Improve DTS Utilization, but Further Action Is Needed
paragraphs: Although the military services have issued various memorandums aimed at  increasing the utilization of DTS, the military service DTS program  officials all pointed to ineffective training as a primary cause of DTS not  being utilized to a far greater extent. The following examples highlight the  concerns raised by the military service officials:    Army DTS program officials emphasized that the DTS system is complex  and the design presents usability challenges for users—especially for first- time or infrequent users. They added that a major concern is that there is  no PMO-DTS training for existing DTS users as new functionality is added  to DTS. These officials stated that the PMO-DTS does not do a good job of  informing users about functionality changes made to the system. We  inquired if the Help Desk was able to resolve the users’ problems, and the  Army DTS officials simply stated “no.” The Army officials further pointed  out that it would be beneficial if the PMO-DTS improved the electronic  training on the DTS Web site and made the training documentation easier  to understand. Also, improved training would help infrequent users adapt  to system changes. The Army officials noted that without some of these  improvements to resolve usability concerns, DTS will continue to be  extremely frustrating and cumbersome for travelers.
Navy DTS program officials stated that DTS lacks adequate user/traveler  training. The train-the-trainer concept of training system administrators  who could then effectively train all their travelers has been largely  unsuccessful. According to Navy officials, this has resulted in many  travelers and users attempting to use DTS with no or insufficient training.  The effect has frustrated users at each step of the travel process and has  discouraged use of DTS.
Air Force officials stated that new DTS system releases are implemented  with known problems, but the sites are not informed of the problems.  Workarounds are not provided until after the sites begin encountering  problems. Air Force DTS program officials stated that DTS releases did  not appear to be well tested prior to implementation. Air Force officials  also stated that there was insufficient training on new functionality. PMO- DTS and DTS contractor program officials believed that conference calls  to discuss new functionality with the sites were acceptable training, but  Air Force officials did not agree. The Air Force finance office was  expected to fully comprehend the information received from those  conference calls and provide training on the new functionality to  users/approvers, but these officials stated that this was an unrealistic  expectation.
As discussed in our September 2005 testimony and January 2006 report,  the unnecessary continued use of the legacy travel systems results in the  inefficient use of funds because the department is paying to operate and  maintain duplicative systems that perform the same function—travel.
section_title: Previously Reported DTS Requirements Management and Testing Deficiencies Have Not Been Resolved
paragraphs: Our September 2005 testimony and January 2006 report noted problems  with DTS’s ability to properly display flight information and traced those  problems to inadequate requirements management and testing. DOD  stated that it had addressed those deficiencies, and in February 2006, we  again tested the system to determine whether the stated weaknesses had  been addressed. We found that similar problems continue to exist. Once  again, these problems can be traced to ineffective requirements  management and testing processes. Properly defined requirements are a  key element in systems that meet their cost, schedule, and performance  goals since the requirements define the (1) functionality that is expected  to be provided by the system and (2) quantitative measures by which to  determine through testing whether that functionality is operating as  expected.
We briefed PMO-DTS officials on the results of our tests and in May 2006  the officials agreed that our continued concerns about the proper display  of flight information were valid. PMO-DTS officials stated that the DTS  technology refresh, which was to be completed in September 2006, should  address some of our concerns. While these actions are a positive step  forward, they do not address the fundamental problem that DTS’s  requirements are still ambiguous and conflicting—a primary cause of the  previous problems. Until a viable requirements management process is  developed and effectively implemented, the department (1) cannot  develop an effective testing process and (2) will not have reasonable  assurance the project risks have been reduced to acceptable levels.
subsections: section_title: Providing Complete Flight Information Has Been a Continuing Problem
paragraphs: In our earlier testimony and report, we noted that DOD did not have  reasonable assurance that the flights displayed met the stated DOD  requirements. Although DOD stated in each case that our concerns had  been addressed, subsequent tests found that the problems had not been  corrected. Requirements represent the blueprint that system developers  and program managers use to design, develop, and acquire a system.
Requirements should be consistent with one another, verifiable, and  directly traceable to higher-level business or functional requirements. It is  critical that requirements be carefully defined and that they flow directly  from the organization’s concept of operations (how the organization’s day- to-day operations are or will be carried out to meet mission needs).  Improperly defined or incomplete requirements have been commonly  identified as a cause of system failure and systems that do not meet their  cost, schedule, or performance goals.
Requirements represent the foundation on which the system should be  developed and implemented. As we have noted in previous reports,  because requirements provide the foundation for system testing,  significant defects in the requirements management process preclude an  entity from implementing a disciplined testing process. That is,  requirements must be complete, clear, and well documented to design and  implement an effective testing program. Absent this, an organization is  taking a significant risk that its testing efforts will not detect significant  defects until after the system is placed into production. Our February 2006  analysis of selected flight information disclosed that DOD still did not have  reasonable assurance that DTS displayed flights in accordance with its  stated requirements. We analyzed 15 U.S. General Services Administration  (GSA) city pairs, which should have translated into 246 GSA city pair  flights for the departure times selected. However, we identified 87 flights  that did not appear on one or more of the required listings based on the  DTS requirements. For instance, our analysis identified 44 flights  appearing on other DTS listings or airline sites that did not appear on the  9:00 am DTS listing even though those flights (1) met the 12-hour flight  window and (2) were considered GSA city pair flights—two of the key  DTS requirements the system was expected to meet.
After briefing PMO officials on the results of our analysis in February 2006,  the PMO-DTS employed the services of a contractor to review DTS to  determine the specific cause of the problems and recommend solutions. In  a March 2006 briefing, the PMO-DTS acknowledged the existence of the  problems, and identified two primary causes. First, part of the problem  was attributed to the methodology used by DTS to obtain flights from the  Global Distribution System (GDS). The PMO-DTS stated that DTS was  programmed to obtain a “limited” amount of data from GDS in order to  reduce the costs associated with accessing GDS. This helps to explain why  flight queries we reviewed did not produce the expected results. To  resolve this particular problem, the PMO-DTS proposed increasing the  amount of data obtained from GDS. Second, the PMO-DTS acknowledged  that the system testing performed by the contractor responsible for  developing and operating DTS was inadequate and, therefore, there was  no assurance that DTS would provide the data in conformance with the  stated requirements. This weakness was not new, but rather reconfirms  the concerns discussed in our September 2005 testimony and January 2006  report related to the testing of DTS.
section_title: DOD’s Planned Corrective Actions Will Not Address Fundamental Requirements Management Problems
paragraphs: While DOD’s planned actions, including a recent technology upgrade,  should address several of the specific weaknesses we identified related to  flight displays, they fall short of addressing the fundamental problems that  caused those weaknesses—inadequate requirements management. DTS’s  requirements continue to be ambiguous. For example, DOD has retained a  requirement to display 25 flights for each inquiry. However, it has not  determined (1) whether the rationale for that requirement is valid and   (2) under what conditions flights that are not part of the GSA city pair  program should be displayed. For example, we found that several DTS  flights displayed to the user “overlap” other flights. Properly validating  the requirements would allow DOD to obtain reasonable assurance that its  requirements properly define the functionality needed and the business  rules necessary to properly implement that functionality. As previously  noted, requirements that are unambiguous and consistent are fundamental  to providing reasonable assurance that a system will provide the desired  functionality. Until DOD improves DTS requirement management  practices, it will not have this assurance.
section_title: Recommendations to Improve DTS Management and Oversight
paragraphs: Our recent report included four recommendations to improve the  department’s management and oversight of DTS. We recommended that  DOD (1) evaluate the cost effectiveness of the Navy continuing with the  CTO management fee structure versus adopting the revised CTO fee  structure, once the new contracts have been awarded, (2) develop a  process by which the military services develop and use quantitative data  from DTS and their individual legacy systems to clearly identify the total  universe of DTS-eligible transactions on a monthly basis, (3) require the  PMO-DTS to provide periodic reports on the utilization of DTS, once  accurate data are available, and (4) resolve inconsistencies in DTS  requirements by properly defining the functionality needed and business  rules necessary to properly implement the needed functionality. DOD  concurred with three and partially concurred with one of the  recommendations. In regard to the recommendations with which the  department concurred, it briefly outlined the actions it planned to take in  addressing two of the three recommendations. For example, the  department noted the difficulties in obtaining accurate utilization data  from the existing legacy systems, but stated that the Office of the Under  Secretary of Defense (Personnel and Readiness) and BTA will evaluate  methods for reporting actual DTS utilization.
Additionally, DOD noted that the Defense Travel Management Office  developed and implemented a requirements change management process  on May 1, 2006. In commenting on the report, the department stated that  this process is intended to define requirements and track the entire life  cycle of the requirements development process. While we fully support the  department’s efforts to improve its management oversight of DTS’s  requirements, we continue to believe that the department needs to have in  place a process that provides DOD reasonable assurance that   (1) requirements are properly documented and (2) requirements are  adequately tested as recommended in our January 2006 report. This  process should apply to all existing requirements as well as any new  requirements. As discussed in this report, we reviewed in May 2006 some  of the requirements that were to have followed the new requirements  management process and found problems similar to those noted in our  January 2006 report. Although we did not specifically review the new  process, if it does not include an evaluation of existing requirements, the  department may continue to experience problems similar to those we  previously identified.
DOD partially concurred with our recommendation to evaluate the cost  effectiveness of the Navy continuing with the CTO management fee  structure. However, DOD’s response indicated that the Defense Travel  Management Office is currently procuring commercial travel services for  DOD worldwide in a manner that will ensure evaluation of cost  effectiveness for all services. If DOD proceeds with the actions outlined in  its comments, it will meet the intent of our recommendation.
Effective implementation of these recommendations as well as those  included in our January 2006 report will go a long way towards improving  DTS functionality and increasing utilization. Furthermore, the shift of DTS  to the BTA, which makes DTS an enterprisewide endeavor, should help in  making DTS the standard integrated, end-to-end travel system for business  travel. Management oversight is essential for this to become a reality. As I  stated previously, in written comments on a draft of our report, the Under  Secretary of Defense (Personnel and Readiness), strongly objected to our  finding that the estimated personnel savings included in the economic  analysis are unrealistic. Because none of the military services could  validate an actual reduction in the number of personnel as a result of DTS  implementation, and DOD’s comments did not include any additional  support or documentation for its position, we continue to believe that the  estimated annual personnel savings of $54.1 million are unrealistic.  Although the department’s criteria do not require that a new economic  analysis be prepared, the fiscal year 2005 defense authorization act  requires the periodic review, but not less than annually, of every defense  business system investment. If effectively implemented, this annual review  process provides an excellent opportunity for DOD management to assess  whether DTS is meeting its planned cost, schedule, and functionality goals.  Going forward, such a review could serve as a useful management tool in  making funding and other management decisions related to DTS.
In conclusion, overhauling the department’s antiquated travel management  practices and systems has been a daunting challenge for DOD. While it  was widely recognized that this was a task that needed to be accomplished  and savings could result, the underlying assumptions in support of those  savings are not based on reliable data and therefore it is questionable  whether the anticipated savings will materialize. Even though the overall  savings are questionable, the successful implementation of DTS is critical  to reducing the number of stovepiped, duplicative travel systems  throughout the department. We have reported on numerous occasions that  reducing the number of business systems within DOD can translate into  savings that can be used for other mission needs. As noted above,  management oversight will be an important factor in DTS achieving its  intended goals. Equally important, however, will be the department’s  ability to resolve the long-standing difficulties that DTS has encountered  with its requirements management and system testing. Until these issues  are resolved, more complete utilization of DTS will be problematic.
Mr. Chairman, this concludes my prepared statement. We would be happy  to answer any questions that you or other members of the Subcommittee  may have at this time.
section_title: Contacts and Acknowledgments
paragraphs: For further information about this testimony, please contact McCoy  Williams at (202) 512-9095 or williamsm1@gao.gov, or Keith A. Rhodes at  (202) 512-6412 or rhodesk@gao.gov.
Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. In addition to the  above contacts, the following individuals made key contributions to this  testimony: Darby Smith, Assistant Director; J. Christopher Martin, Senior- Level Technologist; F. Abe Dymond, Assistant General Counsel; Beatrice  Alff; Harold Brumm, Jr.; Francine DelVecchio; and Tarunkant Mithani.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. It may be reproduced and distributed in its entirety without further  permission from GAO. However, because this work may contain copyrighted images or  other material, permission from the copyright holder may be necessary if you wish to  reproduce this material separately.