Federal Bureau of Investigation: Weak Controls over Trilogy Project Led to Payment of Questionable Contractor Costs and Missing Assets
Published: 2006-05-02T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Trilogy project--initiated in 2001--is the Federal Bureau of Investigation's (FBI) largest information technology (IT) upgrade to date. While ultimately successful in providing updated IT infrastructure and systems, Trilogy was not a success with regard to upgrading FBI's investigative applications. Further, the project was plagued with missed milestones and escalating costs, which eventually totaled nearly $537 million. This testimony focuses on (1) the internal controls over payments to contractors, (2) payments of questionable contractor costs, and (3) FBI's accountability for assets purchased with Trilogy project funds.
section_title: What GAO Found
paragraphs: FBI's review and approval process for Trilogy contractor invoices, which included a review role for GSA as contracting agency, did not provide an adequate basis for verifying that goods and services billed were actually received and that the amounts billed were appropriate, leaving FBI highly vulnerable to payments of unallowable costs. This vulnerability is demonstrated by FBI's payment of about $10.1 million in questionable contractor costs we identified using data mining, document analysis, and other forensic auditing techniques. These costs included first-class travel and other excessive airfare costs, incorrect charges for overtime hours, potentially overcharged labor rates, and charges for which the contractors could not provide adequate supporting documentation to substantiate the costs purportedly incurred. FBI also failed to establish controls to maintain accountability over equipment purchased for the Trilogy project. These control lapses resulted in more than 1,200 missing pieces of equipment valued at approximately $7.6 million that GAO identified as part of its review. Given the poor control environment and the fact that GAO reviewed only selected FBI payments to Trilogy contractors, other questionable contractor costs may have been paid that have not been identified. If these control weaknesses go uncorrected, future contracts, including those related to Sentinel--FBI's new electronic information management system initiative--will be greatly exposed to improper payments. In addition, the lack of accountability for Trilogy equipment calls into question FBI's ability to adequately safeguard its existing assets as well as those it may acquire in the future.

section_title: Letter
paragraphs: Thank you for the opportunity to discuss the results of our audit of the  Federal Bureau of Investigation’s (FBI) internal controls over contract  payments related to the Trilogy project and safeguarding assets purchased  with Trilogy funds. Our recently issued report, developed at the request of  this committee, identifies weaknesses in FBI’s ability to establish and  implement controls that reasonably ensure, among other things, that  goods and services billed were actually received and that the amounts  billed were appropriate. Further, our report also discusses how FBI failed  to establish controls to maintain accountability over equipment purchased  for the Trilogy project. These weaknesses resulted in payment of millions  of dollars in questionable contractor costs and missing assets. It is  imperative that FBI correct these weaknesses in order to avoid similar  outcomes for its Sentinel and other information technology (IT) projects.
Before I get into our audit findings, let me first provide some brief  background on the Trilogy project. For several years, FBI’s IT systems  were considered archaic and inadequate for efficiently and effectively  investigating criminal and other cases. Initiated in mid-2001, Trilogy— FBI’s largest IT upgrade to date—was intended to modernize FBI’s IT  infrastructure and systems and provide needed applications to help FBI  agents, analysts, and others do their jobs. The Trilogy project consisted of  two primary efforts—upgrades to FBI’s IT infrastructure and development  of an investigative application system to more efficiently access case files,  which became known as the Virtual Case File (VCF) system. FBI entered  into an interagency agreement with the General Services Administration  (GSA), which served as the contracting agency to acquire the services of  two primary contractors to carry out the Trilogy project. DynCorp—now  Computer Services Corporation (CSC)—was responsible for the IT  infrastructure upgrade, while Science Applications International  Corporation (SAIC) was responsible for development of the VCF system.  In addition, FBI contracted with Mitretek to assist in the administration  and oversight of the project.
Although the original scheduled completion date for the overall Trilogy  project was June 2004, after September 11, 2001, FBI instituted an  accelerated deployment plan. The targeted completion date for the portion  of Trilogy related to FBI’s IT infrastructure was accelerated from May 2004  to July 2002. However, after several delays the upgrade was completed in  April 2004, only a month before the “pre-accelerated” due date.
While the scheduled completion date for the VCF system was originally  June 2004, the due date for the first VCF deliverable was accelerated to  December 2003. However, in July 2004, the VCF portion of the Trilogy  project was scaled back after the completion of the first phase of the  project was determined to be infeasible and cost prohibitive as originally  envisioned. The scaled back VCF effort was recast as a pilot that ended in  March 2005, and was to be used by FBI to help develop requirements for a  successor information management system initiative, referred to as  Sentinel. The overall cost of the Trilogy project, originally estimated at  approximately $380 million, ultimately escalated to approximately   $537 million.
The Department of Justice Office of Inspector General has reported on  numerous issues that contributed to the cost increases and delays,  including poorly defined and slowly evolving design requirements,  contracting weaknesses, unrealistic task scheduling, and lack of  management continuity and oversight for tracking and overseeing costs  effectively. We also earlier reported on weaknesses in FBI’s IT systems  development and management capabilities, including contractor  oversight. Because of these issues, you asked us to audit the costs of the  Trilogy project, the majority of which represented the purchase of goods  and services from contractors. Our objectives were to determine whether  (1) FBI’s internal controls provided reasonable assurance that payment of  unallowable contractor costs would not be made or would be detected in  the normal course of business, (2) FBI’s payments to contractors were  properly supported as a valid use of government funds, and (3) FBI  maintained proper accountability for assets purchased with Trilogy  project funds.
We performed our work in accordance with generally accepted  government auditing standards in Washington, D.C., and at two FBI field  sites and various other GSA and contractor locations in Virginia. The  complete scope and methodology of our review is discussed in appendix II  of our report.
Today, I will summarize the results of our work with respect to   (1) weaknesses in FBI’s internal controls that made it highly vulnerable to  payment of unallowable or questionable contractor costs with Trilogy  funds, (2) certain payments for questionable contractor costs that we  identified, and (3) FBI’s inadequate accountability for assets purchased  with Trilogy project funds.
subsections: section_title: Insufficient Invoice Review and Approval Process Increased FBI’s Vulnerability to Payment of Unallowable Contractor Costs
paragraphs: FBI’s review and approval process for Trilogy contractor invoices, which  was carried out by a review team consisting of officials from FBI, GSA,  and Mitretek, did not provide an adequate basis for verifying that goods  and services billed were actually received by FBI or that payments were  for allowable costs. This occurred in part because responsibility for the  review and approval of invoices was not clearly defined or documented. In  addition, contractor invoices frequently lacked detailed information  required by the contracts and other additional information that would be  needed to facilitate an adequate review process. Despite this, invoices  were paid without requesting additional supporting documentation  necessary to determine the validity of the charges. These weaknesses in  the review and approval process made FBI highly vulnerable to payment  of unallowable or questionable contractor costs.
While the invoice review and approval process differed for each  contractor and type of invoice charge, in general the process carried out  by the review team lacked key procedures to reasonably ensure that goods  and services billed were actually received by FBI or that the amounts  billed and paid were for allowable costs. For example, the review team did  not have a systematic process for verifying that the individuals listed on  labor invoices actually worked the number of hours billed or that the job  classification and related billing rates were appropriate. Further, there was  no documented assessment of whether overall hours billed for a particular  activity were in line with expectations. In addition, the review team paid  contractor invoices for subcontractor labor charges without any attempt  to assess the validity of the charges. The GSA official responsible for  paying the invoices stated that the review team relied on the contractors to  properly bill for costs related to subcontractors and to validate the  subcontractor invoices. However, the review team had no process in place  to assess whether the contractors were properly validating their  subcontractor labor charges or to assess the allowability of those charges.
The insufficient invoice review and approval process was at least in part  the result of a lack of clarity in the interagency agreement between FBI  and GSA as well as in FBI’s oversight contract with Mitretek. We have  identified the management of interagency contracting as a high-risk area,  in part because it is not always clear with whom the responsibility lies for  critical management functions in the interagency contracting process,  including contract oversight. For example, the terms and conditions of the  interagency agreement with GSA only vaguely described GSA’s role in  contract administration. In particular, the agreement did not specify the  invoice review and approval steps to be performed or who would perform  them. Likewise, the Mitretek contract provided a general description of  Mitretek’s oversight duties, but did not specifically mention its  responsibilities related to the invoice review and approval process.  Additionally, the lack of clarity in roles and responsibilities was evident in  our interviews with the review team, where each party indicated that  another party was responsible for a more detailed review.
The failure to establish an effective review process was compounded by  the fact that not all invoices provided the type of detailed information  required by the contracts and other information that would be needed to  validate the invoice charges. For example:    CSC labor invoices did not include information related to individual labor  rates or indicate which overhead rates were applicable to each  employee—information needed to verify mathematical accuracy and to  determine that the components of the labor charges were valid.
CSC invoices provided a summary of travel charges by category (e.g.,  airfare and lodging), but did not provide required information related to an  individual traveler’s trip costs. The travel invoices also did not provide  cost detail by travel authorization number. Therefore, there was no way to  determine that the trips billed were approved in advance or that costs  incurred were proper and reasonable based on the location and length of  travel.
CSC and SAIC invoices for the other direct costs (ODC) provided a  summary of charges by category (e.g., shipping and office supplies);  however, CSC did not provide required cost detail by transaction. In some  cases, the category of charges was not even identified. For example, as  shown in figure 1, on the ODC invoice, a category entitled “Other Direct  Costs” made up $1.907 million of the $1.951 million invoice current billing  total. No additional information was provided on the invoice to explain  what made up these costs.
Even though contractor invoices, particularly those from CSC, frequently  lacked key information needed for reviewing charges, we found through  inquiries with the review team and the contractors that invoices were  generally paid without requesting additional supporting documentation.
We further found that invoices for equipment did not individually identify  each asset being billed by bar code, serial number, or some other identifier  that would allow verification of assets billed to assets received. This  severely impeded FBI’s ability to determine whether it had actually  received the assets included on invoices and to subsequently track  individual accountable assets on an item-by-item basis.
section_title: Some Payments Made to Contractors Were for Questionable Costs
paragraphs: Because of the lack of fundamental internal controls over the process used  to pay Trilogy invoices, FBI was highly vulnerable to payment of  unallowable contractor charges. In order to assess the effect of these  vulnerabilities, we used forensic auditing techniques to select certain  contractor costs for review. We identified about $10.1 million in  questionable contractor costs paid by FBI. These costs included payments  for first-class travel and other excessive airfare costs, incorrect billings for  overtime hours worked, potentially overcharged labor rates, and other  questionable costs. Given FBI’s poor control environment over invoice  payments and the fact that we reviewed only selected FBI payments to  Trilogy contractors, other questionable costs may have been paid that  have not been identified.
subsections: section_title: First-class Travel and Other Excessive Airfare Costs
paragraphs: During our review of CSC’s supporting documentation for selected travel  charges, we found 19 first-class airline tickets costing a total of $20,025.  The CSC contract called for travel to be reimbursed to the extent  allowable under the Joint Travel Regulations, which state that travelers  must use basic economy or coach class unless the use of first-class travel  is properly authorized and justified. Because the documentation provided  by CSC for these first-class tickets we identified did not contain the  required authorizations or justifications, we consider the cost of this travel  in excess of coach-class fares as potentially unallowable.
Also during our review of travel charges, we noted several instances of  unusually expensive coach-class tickets, which we also considered to be  questionable. Upon further inquiry with several airlines, we determined  that most of these were for “full fare” coach-class tickets. We noted that  the airlines used most often by the contractors indicated that it is possible  to obtain a free upgrade to first class with the purchase of the more  expensive full-fare coach ticket. In fact, we found that in some instances,  the current price of a full-fare coach ticket was higher than the current  price of a first-class ticket. We noted 62 full-fare coach tickets billed by  CSC for $85,336. In contrast, we estimated that basic coach-class fares  would have cost $41,978. SAIC and Mitretek also billed FBI for excessive  airfare costs, but to a lesser degree. In total, we identified 75 unusually  expensive tickets costing $100,847, which exceeded our estimate of basic  coach-class fares by approximately $49,848. Table 1 provides examples of  the first-class and excessive airfare travel costs we identified.
section_title: Excess Overtime Charges
paragraphs: Our review also showed that FBI may have paid SAIC for incorrectly billed  overtime charges. The task order for SAIC work stated that the  government would not object to SAIC employees working hours in excess  of 40 per week if necessary. In March 2003, SAIC implemented a policy  that FBI agreed to, which decreased the amount of hours that would be  billed to FBI. This policy stated that contractor staff would be  compensated for hours worked that exceeded 90 hours in a 2-week pay  period, and established a ceiling of 120 hours per pay period. We found,  however, that SAIC employees frequently charged for all hours worked  beyond 80 in a pay period and noted some instances where employees  charged hours beyond the 120-hour ceiling. The costs of these hours were  billed to and paid by FBI. SAIC management acknowledged that billings  were not consistent with the March 2003 policy and indicated that it would  research the issue further to determine whether corrections are  necessary. Based on our review of the labor charges, FBI may have  overpaid for more than 4,000 hours. Using average, fully burdened labor  rates for employees who billed incorrectly, we estimated that FBI may  have overpaid these overtime costs by as much as $400,000.
Questionable Labor Rates  We also found that CSC/DynCorp may have charged labor rates that  exceeded ceiling rates that GSA asserts were established pursuant to a  DynCorp task order. In short, GSA and CSC disagree on whether ceiling  rates for a CSC/DynCorp subcontractor, DynCorp Information Systems  (DynIS), were ever established. When DynCorp entered into the  contractual agreement with GSA, it agreed to ceiling rates for various  labor categories and agreed to negotiate subcontractor ceiling rates  separately for each task order. The May 2001 DynCorp task order award  document stated that ceilings were in place on all DynIS labor category  and indirect rates, subject to negotiation pending the results of a Defense  Contract Audit Agency audit. GSA officials told us they believed that  DynIS labor category rates in DynCorp’s Trilogy proposal represented  established ceilings, and that they negotiated DynIS labor category ceiling  rates with DynCorp. However, CSC stated that it never negotiated labor  category ceiling rates with GSA.
Based on our review of DynCorp’s labor invoices, we noted that several of  DynIS’s rates charged exceeded the labor rates that GSA contended were  ceiling rates. For example, CSC/DynCorp billed over 14,000 hours for work  performed by senior IT analysts during 2001 on the Trilogy project based  on an average hourly rate of $106.14. However, if ceiling rates were  established, the DynCorp proposal indicated that the Trilogy project  would be charged a maximum of $68.73 per hour for a senior IT analyst  working in the field or $96.24 per hour for a senior IT analyst working at  headquarters during 2001. If ceiling rates were established, we estimated  that FBI overpaid CSC/DynCorp by approximately $2.1 million for DynIS  labor costs.
Other Questionable Costs  We also identified about $7.5 million in other payments to contractors that  were for questionable costs. In most cases, these costs were not supported  by sufficient documentation to enable an objective third party to  determine if each payment was a valid use of government funds. For  example, CSC did not provide us adequate supporting documentation for  almost $2 million of subcontractor labor charges and about $5.5 million of  ODC charges we selected to review.
Because $4.7 million of these inadequately supported ODC costs were for  training charges from one subcontractor, CACI Inc. – Federal (CACI), we  subsequently requested supporting documentation from the subcontractor  for selected charges for training costs totaling about $3.5 million. We  found that CACI could not adequately support charges to FBI totaling  almost $3 million that CACI paid to one event planning company (another  subcontractor). CACI stated that supporting documentation was not  applicable because its agreement with the event planner was “fixed  priced.” However, CACI’s assertion was not supported by the terms of the  purchase order and related statement of work that specifically required  documentation to support costs claimed by the event planner and to  charge only for services rendered.
CSC was also unable to provide us adequate supporting documentation for  $762,262 in equipment disposal costs billed by two subcontractors. The  documentation provided consisted of a spreadsheet that summarized costs  of the subcontractors, but did not include receipts or other support to  prove that these costs were actually incurred.
Our review of SAIC’s subcontractor labor charges found that FBI was  billed twice for the same subcontractor invoice totaling $26,335. SAIC  officials agreed that they double billed and stated that they would make a  correction.
section_title: Major Lapses in Accountability Resulted in Millions of Dollars of Missing Trilogy Equipment
paragraphs: Our audit also disclosed that FBI did not adequately maintain  accountability for equipment purchased for the Trilogy project. FBI relied  extensively on contractors to account for Trilogy assets while they were  being purchased, warehoused, and installed. However, FBI did not  establish controls to verify the accuracy and completeness of contractor  records it was relying on. Moreover, once FBI took possession of the  Trilogy equipment, it did not establish adequate physical control over the  assets. Consequently, we found that FBI could not locate over 1,200 assets  purchased with Trilogy funds, which we valued at approximately   $7.6 million. Because of the significant weaknesses we identified in FBI’s  property controls, the actual amount of missing equipment could be even  higher.
FBI relied on contractors to maintain records related to the purchasing,  warehousing, and installation of about 62 percent of the equipment  purchased for the Trilogy project. FBI’s primary contractor responsible  for delivering computer equipment to FBI sites was CSC. FBI officials told  us they met regularly with CSC and its subcontractors to discuss FBI’s  equipment needs and a deployment strategy for the delivery of equipment.  Based on these meetings, CSC instructed its subcontractors to purchase  equipment, which was subsequently shipped to and put under the control  of those same subcontractors. Once equipment arrived at the  subcontractors’ warehouses, the subcontractors were responsible for  affixing bar codes on accountable items—all items valued above $1,000  and certain others considered sensitive that are required by FBI policy to  be tracked individually. In addition, FBI directly purchased about   $19.1 million of equipment for the Trilogy project that was shipped directly  to either CSC or CSC subcontractors.
When equipment was shipped from a subcontractor warehouse to an FBI  site, the subcontractor prepared a bill of lading that listed all items  shipped. However, there was no requirement for FBI officials to verify that  the items were actually received. The subcontractors also prepared a “Site  Acceptance Listing” of equipment that had been installed at each FBI site.  While an FBI official signed this listing, based on our inquiries at two field  offices, we found the officials may not have always verified the accuracy  and completeness of these lists. FBI did not prepare its own independent  lists of ordered, purchased, or paid-for assets and did not perform an  overall reconciliation of total assets ordered and paid for to those  received. Such a reconciliation would have been made difficult by the fact  that invoices FBI received from CSC did not include item-specific  information—such as bar codes, serial numbers, or shipping location.  However, failure to perform such a reconciliation left FBI with no  assurance that it had received all of the assets it paid for.
In addition, equipment that was delivered to FBI sites was not entered into  FBI’s Property Management Application (PMA) in a timely manner,  increasing the risk that assets could be lost or stolen without detection.  We found that 71.6 percent of the CSC-purchased equipment that was  recorded in PMA, representing 84 percent of the total dollar value, was  entered more than 30 days after receipt, and nearly 17 percent of the  equipment, representing 37 percent of the dollar value, was entered more  than a year after receipt. When assets are not timely recorded in the  property system, there is no systematic means of identifying where they  are located or when they are removed, transferred, or disposed of and no  record of their existence when physical inventories are performed. This  severely limits the effectiveness of the physical inventory in detecting  missing assets and in triggering investigation efforts as to the causes.
FBI also could not accurately identify all accountable assets because of  improper controls related to its bar codes—a key tool for maintaining  accountability and control over individual assets. FBI relied on  contractors to affix the bar codes, yet did not track the bar code numbers  given to contractors, the bar code numbers they used, or the bar code  numbers returned. Moreover, FBI provided incorrect instructions to  contractors, initially directing them to bar code certain types of lower cost  equipment that did not need to be tracked. FBI’s loss of control over its  bar codes and failure to timely enter assets into its property tracking  system seriously hampered its ability to maintain accountability for its  Trilogy equipment. Accountability for equipment was further undermined  by FBI’s failure to perform sufficient physical inventory procedures to  ensure that all assets purchased with Trilogy funds were actually located  during the physical inventory.
Given the serious nature of these control weaknesses, we performed  additional test work to determine whether all accountable assets  purchased with Trilogy funds could be accounted for and found that FBI  was unable to locate 1,404 of these assets. These were items such as  desktop computers, laptops, printers, and servers. In written comments on  a draft of our report, FBI told us that it had accounted for more than 1,000  of these items. During our agency comment period, FBI stated that it had  found 237 items we previously identified as missing and provided us  evidence, not made available during our audit, to sufficiently account for  199 of these items. We adjusted the missing assets listing in our report to  reflect 1,205 (1,404 – 199) assets as still missing. FBI later informed us that  the approximately 800 remaining items noted in its official agency  response included (1) accountable assets not recorded in PMA because  they were either incorrectly identified as nonaccountable assets or  mistakenly omitted, (2) defective accountable assets that were never  recorded in PMA and subsequently replaced, and (3) nonaccountable  assets or components of accountable assets that were incorrectly bar  coded.
We considered these same issues during our audit and attempted to  determine their impact. For example, as stated in our report, FBI told us  that components of some nonaccountable assets that were part of a larger  accountable item may have been mistakenly bar coded. Using FBI  guidance on accountable property, we determined that 103, or about 11  percent, of the 926 missing assets purchased by CSC may have represented  nonaccountable components. Because FBI could not provide us with the  location information, we could not definitively determine whether the  items were accountable assets. During the course of our audit, FBI was  not able to provide us with any evidence to support its other statements  regarding the reasons the assets could not be located.
While we are encouraged by FBI’s current efforts to account for these  assets, its ability to definitively determine their existence has been  compromised by the numerous control weaknesses identified in our  report. Further, the fact that assets have not been properly accounted for  to date means that they have been at risk of loss or misappropriation  without detection since being delivered to FBI—in some cases, for several  years.
section_title: Concluding Comments
paragraphs: FBI’s Trilogy IT project spanned 4 years and the reported costs exceeded  $500 million. Our review disclosed that there were serious internal control  weaknesses in the process used by FBI and GSA to approve contractor  charges related to Trilogy, which made up the majority of the total  reported project cost. While our review focused specifically on the Trilogy  program, the significance of the issues identified during our review may  indicate more systemic contract and financial management problems at  FBI and GSA, in particular when using cost-reimbursable type contracts  and interagency contracting vehicles. These weaknesses resulted in the  payment of millions of dollars of questionable contractor costs, which may  have unnecessarily increased the overall cost of the project. Unless FBI  strengthens its controls over contractor payments, its ability to properly  control the costs of future projects involving contractors, including its new  Sentinel project, will be seriously compromised. Further, weaknesses in  FBI’s controls over the equipment acquired for Trilogy resulted in millions  of dollars in missing equipment and call into question FBI’s ability to  adequately safeguard its equipment, as well as confidential and sensitive  information that could be accessed through that equipment from  unauthorized use.
Our companion report includes 15 recommendations to help improve  FBI’s and GSA’s controls over their invoice review and approval processes  and to address questionable billing issues we identified. It also includes 12  recommendations to help improve FBI’s accountability for assets. FBI  concurred with our recommendations and outlined actions under way and  further planned actions to address the weaknesses we identified. FBI also  provided additional information related to Trilogy assets we identified as  missing. While GSA accepted our recommendations, it did not believe that  one of them was needed, and described some of the improvements to its  internal controls and other business process changes already  implemented. GSA also expressed concern with some of our observations  and conclusions related to the invoice review and approval process and  our analysis of airfare costs. We continue to believe that our report is  accurate and that all recommendations should be implemented.
We understand that FBI has outlined actions to implement our  recommendations. While we are encouraged by these efforts, let me just  emphasize the importance of continually monitoring the implementation  of corrective actions to ensure that they are effective in helping to avoid  the types of control lapses that we identified throughout the Trilogy  project. Without such vigilant monitoring, Sentinel and other efforts will  be greatly exposed to similar questionable or inappropriate payments and  lack of accountability over assets.
Mr. Chairman and members of the committee, this concludes my prepared  statement. I would be pleased to answer any questions that you may have.
section_title: Contact and Acknowledgments
paragraphs: For more information regarding this testimony, please contact Linda M.  Calbom at (202) 512-9508 or calboml@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this testimony. Individuals making key contributions to this  testimony included Steven Haughton (Assistant Director), Ed Brown,  Marcia Carlsen (Assistant Director), Lisa Crye, and Matt Wood. Numerous  other individuals contributed to our audit and are listed in our companion  report.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. It may be reproduced and distributed in its entirety without further  permission from GAO. However, because this work may contain copyrighted images or  other material, permission from the copyright holder may be necessary if you wish to  reproduce this material separately.