Contract Management: Service Contract Approach to Aircraft Simulator Training Has Room for Improvement
Released: 2006-09-22T00:00:00

Letter

Background

Air Force and Army cite the increasingly complex training requirements needed to prepare for the ever more lethal battlefield environment as a factor that has led to greater reliance on flight simulators. A flight simulator is a system that tries to realistically replicate, or simulate, the experience of flying an aircraft. Flight simulators range from video games to full-sized cockpit replicas mounted on hydraulic (or electromechanical) actuators and controlled by state-of-the-art computer technology. According to Air Force and Army officials, aircraft simulators are a cost- effective way of helping to develop and refine operational flight skills. Simulators can facilitate training that might be impractical or unsafe if done with actual systems and allow for concentrated pilot practice in selected normal and emergency actions. Simulators also can train operators and maintainers to diagnose and address possible equipment faults, and enhance proficiency despite shortages of equipment, space, ranges, or time.

In the late 1990s, the Air Force and Army were faced with increasingly obsolete simulators and the need to quickly acquire up-to-date pilot and aircrew training. In 1997, the then-Commander of the Air Force’s Air Combat Command proposed an innovative approach of buying training as a service, under which the contractors would own, operate, and maintain the simulator hardware and software. The simulator service contracts are one component of a much broader effort, now known as the DMO program. The DMO goal is to provide state-of-the-art simulator training on- demand at the location of the trainee, with the ultimate vision of networking different sites together to create more realistic flying scenarios. Plans call for each fighter unit eventually to be equipped with high-fidelity simulators. As of the fiscal year 2002 budget, the DMO program was formalized in the Air Force budget with the assignment of a program element line item that combined previous program elements for the various simulator systems. For fiscal year 2006, over $200 million was budgeted for the program.

In the early 2000s, Army use of rotary-wing aviation simulation training was limited because the simulators being used were grossly obsolete and based on late 1970s’ technology. To revamp its helicopter training, the Army in late 2001 began the Flight School XXI program. Following the Air Force’s lead, the Army decided to acquire up-to-date simulator training using a service contract.

Congress and the Office of Management and Budget (OMB) have recently addressed the growing level of procurement of services. For example, Congress included provisions in Section 801 of the National Defense Authorization Act for Fiscal Year 2002 designed to improve management and oversight of procurement of services. To ensure that DOD acquires services by means that are in its best interest and managed in compliance with applicable statutes, regulations, directives, and other requirements, the Act required DOD to establish a service acquisition management structure, comparable to the management structure that applies to the procurement of products. In September 2003, we reported that DOD and the military services had a management structure in place for reviewing individual services acquisitions valued at $500 million or more, but that approach did not provide a departmentwide assessment of how spending for services could be more effective. Also, OMB Circular A-11’s Appendix B, “Budgetary Treatment of Lease-Purchases and Leases of Capital Assets,” was amended in 2005 to require agencies to submit to OMB for review any service contracts that require the contractor to acquire or construct assets valued over $50 million. While these provisions do not apply to the previously-awarded simulator training contracts, future replacement contracts will be covered.

Contract Information

All of the Air Force and Army simulator service contracts are funded with O&M funds. O&M funds are typically used for such things as military force operations, training and education, and depot maintenance. The contracts are requirements contracts, meaning that the government, within available funds, shall order from the contractor all the training services specified for each of the aircraft platforms that are required during the effective performance periods. Additionally, each contract contains language limiting the government’s liability in the event the contract is terminated. For example, the Air Force F-15C contract states that the government reserves the right to terminate the contract for its sole convenience and that such termination prior to the issuance of a funded task order shall result in no payment to the contractor of any amount for any work performed or costs incurred. Table 1 provides additional descriptive information for each contract.

The military services are relying on industry to capitalize the required up- front investment needed to acquire simulator hardware and software, with the understanding that the contractors will amortize this investment by selling training services by the hour. Each contract establishes operating hours and the hourly payment rates for the life of the contracts, with rates structured to provide the contractor with higher income in the initial years of service. In calendar year 2004, for example, if the F-16 contractor provided Shaw AFB with simulator availability that met 95 percent of the required system elements, the hourly rate would be $5,225, whereas in calendar year 2006, it would drop to $709 per hour.

We have previously identified the need to examine the appropriate role for contractors to be among the challenges in meeting the nation’s defense needs in the 21st century. We recently reported that the government’s increasing reliance on contractors for missions previously performed by government employees highlights the need for sound planning and contract execution.

Acquisition Reforms Shaped Contract Structures

The structure of the simulator service contracts was heavily influenced by mid-1990s’ acquisition reform initiatives such as the Federal Acquisition Streamlining Act of 1994 and the Clinger-Cohen Act of 1996. These Acts encouraged agencies to use commercial acquisition procedures as a way to streamline the acquisition process. Differences under commercial versus non-commercial procedures pertain, for example, to the contracting officer’s determination of price reasonableness, the government’s right to inspect and test, and government rights to acquire technical data. Appendix III outlines these and other key differences. The Air Force contracts for simulator training are structured as commercial acquisitions, but the Army’s is not. Army officials told us they could not justify calling the requirement “commercial” because the simulators would be configured to reflect combat helicopters, which do not exist in the commercial market. In August 2005, a DOD Inspector General review of the procurement procedures for the F-16 contract concluded that the simulator service did not meet the definition or intent of a commercial service and recommended that the Air Force not use commercial procedures for the re-competed F-16 contract. The Air Force is using non- commercial procedures for the new contract.

To allow for contractor recoupment of up-front investment, the strategy to acquire simulator services envisioned longer duration contracts. This coincided with practices in commercial industry, where long-term relationships between buyer and seller were becoming common. The Air Force and Army adopted this approach by including award-term incentives in the contracts. This incentive can best be described as a variant of an award-fee incentive, where the contractor is rewarded for excellent performance with an extension of the contract period instead of additional fee. Under the award-term concept, an assessment of the contractor’s performance is presented to the term determining official, who unilaterally determines whether to award an extension or a reduction to the contract ordering period. The potential total years of contract performance under the simulator contracts range from 13 to 19.5 years. Appendix IV contains the details of each contract. Award-term incentives are relatively new in government contracting and are not addressed in the Federal Acquisition Regulation (FAR).

Key Players

Several key players are involved with acquisition and use of simulator training.

For the Air Force: Air Combat Command: The requiring entity—the user of simulator training services—is located at Langley AFB, Virginia. The command trains, equips and maintains combat-ready forces for rapid deployment and employment.

Aeronautical Systems Center: This organization is the acquisition agency for the simulator contracts. Located at Wright-Patterson AFB, Ohio, it manages development, acquisition, modification, and in some cases, sustainment for a wide variety of aircraft and related equipment programs. The center develops attack, bomber, cargo, fighter, trainer, and reconnaissance aircraft for the Air Force.

Air Force fighter units: These are the users of the simulator training, which currently is taking place in 10 fighter units.

Fort Rucker, Alabama: Fort Rucker is the requiring entity for the Army’s helicopter flight simulator services. It is the home of all Army aviation flight training and the location of the initial training for new aviators, known as Flight School XXI. The types of helicopters used in Flight School XXI training are the TH-67 basic training helicopter, Chinook, Blackhawk, Apache, and the Attack Reconnaissance Helicopter. Unlike the Air Force’s multiple sites, the helicopter simulators provided under the service contract are located at only this one training site, not at each operational unit.

Army Program Executive Office for Simulation, Training and Instrumentation: This office’s mission is to provide training, testing, and simulation solutions for soldier readiness. The office is co-located in Orlando, Florida, with the Naval Air Systems Command, which awarded the contract on behalf of the Army.

Service Contract Approach Had Perceived Benefits but Was Not Adequately Supported

Both the Air Force and Army were faced with obsolete simulators due to decisions to not devote sufficient procurement funds to upgrade existing simulator hardware and software. The decision to buy simulator training as a service allowed use of O&M funds, which would alleviate the need to compete for procurement funds. Further, it was envisioned that service contracts would allow for automatic simulator upgrades to match the changing aircraft configurations, because industry would be responsible for acquiring, operating, and maintaining the simulators and keeping them concurrent. However, the decision to embark on a services approach was not supported by a thorough analysis of the costs and benefits, despite a DOD directive providing that the acquisition of simulators is to be based on an evaluation of the benefits and trade-offs of potential alternative training solutions.

Simulators Had Lost Out in Competition for Procurement Funds

The difficulty associated with competition for limited procurement dollars was a key factor in the decision to turn to service contracts for war- fighting training. Frequently, simulators have lost out in this competition and ended up under-funded. In 1997, the Air Force identified simulators for four aircraft—the F-15C, F-16, F-15E, and AWACS—as “obsolete or grossly non-concurrent” due to age, technological obsolescence, and lack of concurrency with operational aircraft. By early 2002, the Army was also faced with non-concurrent helicopter simulators, and field unit commanders were reporting decreased unit readiness. For example, while the goal of the training at Fort Rucker is to produce aviators trained at a proficiency level of two (with level one being the highest), Army officials reported that most of the aviators were leaving school with only a proficiency level of three.

These degraded situations existed despite a DOD directive that provides for the military services to ensure that all development, procurement, operation, and support costs for the acquisition of training simulators were programmed and funded. Recognizing the need to keep simulators current with aircraft configurations, particularly as the use of simulators to substitute for live flying hours was rising, the Air Force issued specific guidance on training devices. For example, Air Force Instruction 36-2248, Operation and Maintenance of Aircrew Training Devices, provides that funding be established for simulator modifications concurrently with modifications to the weapon system. Also, Air Force Instruction 36-2251, Management of Air Force Training Systems, provides that the training system receive the same precedence rating as the prime mission system it supports and the same visibility, funding, and documentation. Nevertheless, Air Force funding decisions had not kept flight training simulators for the four aircraft systems concurrent with aircraft configurations.

Also in the late 1990s, the Air Combat Command had unexpended O&M flying hour funds available due to flight crew deployments and obstacles in scheduling training. Use of these funds for service contracts would alleviate the need to compete for procurement funds in an increasingly tight arena. The competition for procurement dollars was also a factor for the Army, which noted that the funds necessary to maintain and upgrade its helicopter training simulators had “not competed effectively against other Army operational and logistics requirements.”

Service Contracts Intended to Yield More Concurrent Simulators

Air Force acquisition officials conducted market research to determine how civilian airlines acquired flight training. They found “turnkey” training services contracts in place in the commercial airline industry. These officials envisioned that services contracts would provide quicker state-of- the-art pilot and aircrew training and keep up with the rapid pace of technology development by shifting the responsibilities for simulator ownership, operation, and maintenance from the government to the contractor. Further, with the contractor responsible for any development, production, and testing necessary to ready the simulators for use, the Air Force saw that it would be relieved of these multiple acquisition efforts, an important factor given the recently downsized acquisition offices. In addition, a stated benefit of the service contracting approach for simulator training as initially implemented was the streamlining or reduction in government oversight. Since commercial acquisition procedures were used to buy these services, fewer government system reviews were required.

When it decided to take a new approach to solve its helicopter simulator concurrency problems, the Army conducted its own market research, solicited business solutions from industry, and conferred with Air Force DMO officials.

Decision to Use Service Approach Not Supported with Thorough Analysis of Costs and Benefits of Alternatives

Neither the Air Force nor the Army thoroughly analyzed the costs and benefits of alternative approaches before pursuing this new approach. DOD’s August 1986, Directive 1430.13, Training Simulators and Devices, provides that the acquisition of simulators be based on an analysis of the training need, the potential use of existing devices to satisfy that need, and an evaluation of the benefits and trade-offs of potential alternative training solutions. A 1999 report to the Air Force on the DMO program also noted the importance of identifying key business factors before embarking on a major acquisition. As a result of the failure to conduct a thorough review of the various alternatives to solving the problem of non-concurrent simulators, decision makers lacked information on the potential cost and benefit estimates that would be encountered should facts, circumstances, and assumptions change.

The historical documents we reviewed demonstrate that within the Air Force there was uncertainty about the cost-effectiveness of the service contract approach to simulator training. Although the potential for reduced costs through outsourcing certain responsibilities and eliminating government logistics support were cited in some decision documents, other documents indicated that the service contract approach would not cost significantly more or less than the traditional ownership strategy. Air Force officials told us that a comprehensive study of various options for providing simulator training had been commissioned. However, they have been unable to locate it.

In preparing to re-compete the F-16 simulator contract, the DMO program office completed a formal business case analysis in November 2005, in response to a July 2005 congressional request. Air Force officials acknowledged that, if not for the request, the formal business case analysis would not have been completed.

The Army completed two business case analyses prior to contracting for simulator services under the Flight School XXI program, but the analyses lacked sufficient detail to provide a thorough examination of the pros and cons of the new approach. The scope of the analyses was limited to determining (1) what length of service contract would be appropriate to justify the large up-front investment required of the contractor and (2) whether projected funding was sufficient to meet program costs in the event the Army was required to follow the traditional acquisition approach. The Army provided us with decision briefings that set forth various options for simulator training, but the documents ruled out all but the service contract approach without providing supporting analyses of the costs and benefits associated with each alternative. Further, the traditional method, where the government bought the simulators, was eliminated as an option due its perceived inability to meet the Flight School XXI 15-month start-up time frame. This schedule eventually slipped more than 10 months with, according to Army officials, no detrimental effect on student training schedules. The briefings do not address the possibility that the 15-month time frame was flexible.

While Training Has Improved, the Expected Number of Simulator Sites Has Not Been Activated

Air Force and Army officials told us the new simulators are big improvements over what they had previously. However, the Air Force has faced funding uncertainties using O&M funds for the contracts, and subsequent schedule slippages have resulted in fewer simulator sites activated than planned. In particular, the F-16 simulator training contractor, citing the reduced activations, notified the Air Force as early as May 2001, that it was unable to provide simulator services as originally agreed and wished to restructure the contract. Later, the company cited Air Force funding problems and schedule slips as the basis for claims against the Air Force and notified the Air Force that its financial situation under the contract was no longer viable. The Air Force will let the current F-16 simulator training contract expire in June 2007 and is in the process of re-competing the contract, which will likely result in a training gap for pilots and additional costs to the Air Force.

Some Air Force and Army Sites Are Operating and Officials Cite Improved Training Results

At the locations we visited, officials told us they were pleased with the quality of the simulator training, particularly when compared with the level of training they had in the past. Pilots are routinely surveyed about the training they receive, and officials told us that, generally, the results have been very positive. For example, the Director of Operations for the F-16 mission training center at Shaw AFB told us that the simulation hardware and software are outstanding and that the training received by young pilots is great. Initial training began under the Army’s Flight School XXI contract in November 2005. While all planned simulators have not yet been activated, according to Flight School XXI officials the school is now meeting its training goal and producing aviators with a proficiency level of two, an improvement over the old regime.

As of July 2006, the Air Force had 16 training simulator sites operational, as shown in table 2.

Air Force Funding Uncertainties Have Resulted in Schedule Slippages

The use of O&M funds under the service contract approach was intended to overcome the situation the military services had faced in the past, when internal decisions on funding priorities had resulted in inadequate procurement funds being made available for simulators. However, almost from the start of the DMO program, funding has been less than projected. As a result, schedule slippages have occurred for many sites compared with original Air Force requirements set forth in acquisition plans. Army officials told us that, to date, O&M funding for the Flight School XXI program has not been reduced. Army officials committed at the outset to fully fund the contract in accordance with the originally projected funding profile and, to date, the funding level has remained stable.

As early as the 2002 budget planning process, Air Force budget requests did not fully fund planned activations, with a total difference between estimated requirements and funding of $524 million over the future year defense plan, as shown in table 3.

An October 2000 Air Force “roadmap” report stated that this funding scenario would “severely impact the executability of the current contracted efforts, as well as the entire vision.” Further, other Air Force decisions, in reaction to fiscal constraints and programs viewed as higher priority, have led to additional funding differences. The Air Combat Command sought to mitigate the impact of these funding differences by shifting flying hour funds into the DMO program in 2003. Table 4 depicts some key events pertaining to the program’s funding impacts and the command’s attempts to secure additional O&M funds.

Largely as a result of these funding uncertainties, many Air Force mission training centers have been activated significantly behind the planned schedule contained in acquisition management plans. These schedule slippages for AWACs, F-15C, and F-16 are shown in tables 5, 6, and 7, respectively.

Air Force officials told us that since most of the original dates were “notional,” meaning that they were not firm requirements, but rather were intended to provide contractors with information about potential mission training center sites, the timely achievement of the schedules was not required. However, contractor representatives told us that their proposals relied upon the planned site activation schedules contained in the contracts, and delays could directly affect their profitability.

Army’s Simulator Activation Schedule Was Also Delayed

The Army has twice rebaselined the activation schedules for the Flight School XXI simulators—the TH-67 and the advanced aircraft virtual simulators (AAVS)—as shown in figure 1.

In the original contract, the TH-67 basic training helicopter simulators were scheduled to begin operation in December 2004, 15 months after contract award. The Flight School XXI project manager could not provide documentation to support this time frame and, in fact, told us that the flight school could not have been ready for students at that time. The Army subsequently rebaselined the schedule to allow for an 8-month delay. Similarly, the Army revised the AAVS activation schedule—originally set at 18 months after contract award—to allow a 7-month delay. According to the project manager, these delays resulted from a protest of the contract award by a competitor and the contractor’s renegotiation with its subcontractors. The schedule was rebaselined a second time, as shown above, because the contractor was not able to meet the adjusted schedule. The Army agreed to the further slippages in exchange for the contractor’s providing two extra terrain databases as consideration. Despite these schedule changes, the necessary simulators and facilities were ready for the first flight school class in November 2005, in accordance with the final revisions to the contract schedule.

F-16 Simulator Training Gap Anticipated

The risk the government faces if a contractor fails to perform as expected under the service contracts is heightened because the government does not own anything—the hardware, software, and data rights are owned by the contractor. In the traditional approach, the government would own the hardware and any software or data it had acquired rights to. While there would be no guarantees as to the condition of these items if the contractor had failed to perform, the government would at least be able to provide them to the replacement contractor, who could potentially make use of them under a new contract. The situation the Air Force has faced with the F-16 simulator contract is illustrative of the potential for not only a degradation in training, but also increased costs to the government when contract performance does not occur as planned.

From the outset, the Air Force believed that the F-16 simulator contractor’s cost estimate was low, as it was about $70 million less than the government’s estimate. According to Air Force and contractor officials, the reason for the low cost estimate was that the contractor amortized its development costs over all the sites that were planned to be activated rather than the minimum number that were contractually required. When schedule delays occurred and the expected sites were not activated, the contractor reported that it lacked the financial viability to continue work under the contract. In April 2003, the contractor stopped work toward making the simulators concurrent with the aircraft, stating that it considered the tasks beyond the contract scope. Subsequently, it told the Air Force it was not in its best interest to activate additional training sites.

The Air Force will allow the F-16 simulator training contract to expire in June 2007 because, according to DOD, the contractor failed to earn enough award-term points to extend the period of performance. The Air Force plans to re-compete the contract. Two aspects of the original contract, awarding it as a commercial acquisition and including an award- term provision, will not be included in the new contract.

Because of the time needed to re-compete the contract and for the winning contractor to provide initial training capabilities, the Air Force faces a potential training gap of over 2 years, during which even the current degraded level of F-16 simulator training services will not be available to pilots. In an effort to ensure some level of continued training during that period, the Air Force plans to award a contract for interim service capability at three air bases. This interim capability will be available for block 50 aircraft only. For the block 40 aircraft, the Air Combat Command plans to spend approximately $20 million to refurbish old F-16 unit training devices. These devices are limited in training potential compared to the current level of simulation.

Return on the Expenditure of Taxpayer Dollars Is Not Effectively Tracked

The Air Force and the Army are not effectively tracking the return on their expenditure of taxpayer dollars to acquire simulator training services. The extent to which the simulators are being used is either not measured or is measured inconsistently. The government is paying for activities conducted during the simulator development period but lacks insight into what it is actually paying for. Finally, award-term evaluations that were established to encourage excellent contractor performance do not always measure key acquisition outcomes such as simulator availability and concurrency, and can result in additional contract years being awarded for only “satisfactory” performance.

Information on Utilization Rates Is Lacking

The utilization rate is the percentage of available hours the simulators are actually used. The Army is not tracking the extent to which aviators are using the contracted service for Flight School XXI simulators, even though for simulators the Army owns, utilization rates are tracked. Program officials told us that, because the Army is contracting for simulator training to be available, there was no need to track the extent to which the government is using this availability. Without data on utilization rates, the Army has no basis for determining the extent to which it is using the services it is buying.

We found that Air Force installations are collecting information on monthly utilization rates, as provided for in a May 1998 Air Force instruction. However, rates at the locations we examined were often far less than the hours the government purchased. For the three AWACS mission training centers at Tinker AFB, for example, we found that, during the 2-year period ending December 2005, monthly utilization rates were frequently reported at less than 50 percent, as shown in figure 2.

The Air Force Audit Agency has reported that installations had acquired excess simulator capacity and unnecessarily consumed O&M funds that could have been applied to other mission requirements. At Shaw AFB, for example, the agency found that the Air Force had paid to use the simulator 10 hours a day, but only used it about 6 hours per day over a 4-month period. The underutilization was attributed to missions being either not scheduled or cancelled. Deployment requirements and range training were identified as contributing factors. At Spangdahlem AB, the audit agency reported that the Air Force had contracted for excess hours of simulator availability to provide the maximum flexibility for pilot schedules. As a result, the Air Force paid for enough simulator availability to hold 3,952 training events in fiscal year 2005, even though it needed only 1,982 training events to meet training requirements.

Our analysis also found that monthly utilization rate calculations are inconsistent among DMO system sites, even though an Air Force instruction provides guidance on how to calculate and report utilization data. We asked six installation quality assurance representatives how they calculated utilization rates. Four of the six representatives were unaware of the instruction, telling us that they had not received any guidance for calculating simulator use. Several different calculation methods are being used, as described in table 8.

Air Combat Command officials told us the reported utilization rates are used to determine whether or when to activate another training center at a site. They also said they are using utilization rate information to determine how many additional “live” flying hours can be moved to the simulators, in particular to alleviate the burden of high fuel costs for aircraft. Because of the very different methods being used to calculate the rates, however, decisions are being made based on non-comparable information.

In addition, we found that the Air Force’s instruction for calculating monthly simulator utilization rates could result in overstating the rates, thus overstating the return on the expenditures made. The instruction directs that utilization be reported when any or all devices at a given location are used. Thus, the Air Force can pay to have four simulators available at a site, use only one of the four during a training period, and still report that simulator utilization was 100 percent as opposed to 25 percent of the paid availability.

Payments for Tasks during Development Period Are Not Transparent

Under the services approach, contractors commit to major investment at the front end, with the return on their investment to come from hourly fees received for providing simulator service. As an additional way to help the contractor recoup its costs earlier, the government added “preparatory” tasks during the development period prior to the start of service. These tasks are defined in the contracts as discrete events, such as site surveys and training capability assessments, that are ordered and paid for prior to the start of service. Payments for these tasks provide the contractor cash flow between contract award and the planned service start dates and give the government a contractual avenue for contract oversight prior to receiving services. We found that the Air Force and Army have little insight into what they are paying for under the preparatory tasks. Although the invoices reflect only the discrete tasks, such as training capabilities assessments, the wide range of invoice amounts—from $91,000 to more than $6.5 million for similar tasks—and our discussions with contractor officials suggest that the government is actually making milestone payments to the contractors for a portion of their up-front costs to acquire and develop the simulators.

The original service contract concept for the F-15C, the first simulator contract awarded, had no provision for the contractor to recoup any costs during the development period, which usually lasts more than a year. Figure 3 shows the development period before the start of simulator services and the original hourly rate structure under the F-15C contract.

This original approach, according to Air Force and contractor officials, contributed to schedule and certification delays with the F-15C. Air Force officials told us that they had no contractual avenue to obtain insight into the contractor’s performance during the development period and thus were not aware that the contractor had encountered delays in obtaining information from other programs and in determining the complexity of some simulation elements. As a result, full service was not implemented on schedule and certification of simulation service was delayed until after the start of initial service. Further, according to the contractor, it suffered an unrecoverable loss of income during the high-rate, initial service period. Subsequently, based on feedback received from industry, the Air Force changed its approach and incorporated preparatory services into the F-15C contract and all subsequent DMO system contracts to obtain more visibility into contractor activities during the development period. The Army also paid for preparatory services during the development period of the Flight School XXI contract.

Our analysis of the Air Force’s payments for preparatory services found significantly disparate costs for site surveys and training assessments, as reflected in tables 9 and 10, respectively.

We asked Air Force and Army officials what was specifically included in these preparatory services and how they determined what they received in return for payments made. They told us that the contractors determine what is included and needed for each service at each site.

Three of the four contractors we spoke with agreed that funding for preparatory tasks helped defray their development costs. They said that, in effect, they bill for these tasks as milestone payments rather than for the discrete tasks themselves. Thus, they are able to begin defraying hardware and software development costs before the start of services. Officials from the fourth contractor stated that site survey tasks are standard but that there is some leeway in what is to be done for training capability assessments and training capability requirements assessments.

With the upcoming re-competition of the F-16 simulator training contract, the Air Force may pay again for the preparatory service tasks in the new contract’s development period, having already spent nearly $42 million on these tasks in the initial contract. Air Force officials told us they cannot assume that potential offerors would make use of the preparatory work the original contractor has performed.

Award-term Evaluations Do Not Measure Key Acquisition Outcomes

In an effort to measure performance and encourage the contractors to perform in an efficient and effective manner, both the Air Force and Army employ award-term incentives. However, while the award-term evaluation areas include pilot and crew satisfaction, they do not always measure the key acquisition outcomes of system availability and concurrency with aircraft upgrades.

While the Air Force does include system availability as an evaluation area, it is assigned only 25 to 30 percent of the total score. Concurrency is not included as a separate evaluation area. The Army’s evaluation areas, on the other hand, include concurrency but not system availability. While the Army requires the tabulation and submission of such data as operational availability and training service completion rate, these data are not included in the award-term evaluations. In addition, several of the evaluation areas include assessments of such things as responsiveness to government requests for cost and pricing data for proposed work not in the initial contract. We recently recommended that DOD move toward more outcome-based award-fee criteria that would promote accountability for acquisition outcomes, rather than include criteria such as responsiveness to government customers or the quality of proposals submitted.

Table 11 compares the award-term evaluation areas and the weight given to each area.

The Air Force and Army both assign the largest weight to “pilot/crew satisfaction.” However, this measure has limitations, particularly when it is heavily relied on to inform award-term decisions. Air Force officials told us that it is in the pilots’ best interests to assign a high rating to this factor; otherwise, they could be viewed as not having received adequate training and could be asked to retake it. Additionally, pilots are frequently hurried in completing their surveys and dash off check marks without much consideration. Also, the distinction between the levels of satisfaction can be blurry. For the Army, for example, if training and support are adversely impacted for an “extended period,” user satisfaction is to be rated as unsatisfactory. However, if the adverse impact occurs “infrequently or temporarily,” it is considered marginal. Because the terms are not defined, the Army cannot be certain that pilots are providing consistent ratings.

We also found that, under the Air Force’s award-term plan, contractors can earn an additional award-term year for only satisfactory performance because awarded points are rolled over to the next evaluation period. A contractor with only satisfactory performance in each of five rating areas can receive up to 51 points each year; thus, within 2 years, it can accumulate the 100 points needed for a 1-year contract extension. The F-16 simulator training contractor, for example, which recently notified the government that it could not continue to perform under the contract, received overall award-term evaluations of “very good” for the first two rating periods (May 2002 through July 2003) and “satisfactory” in the third and fourth periods (July 2003 through January 2005) and earned one contract year extension. The Army has taken a different approach; under its award-term plan it is very unlikely that the contractor can be awarded contract extensions for “satisfactory” performance because rollover is allowed only when more than 100 points are earned. Thus, the contractor with only satisfactory performance cannot accumulate enough points for an additional contract year.

While service is not yet available, the F-15E simulator training contract, awarded in August 2003, does not include an award-term incentive because, according to the contracting officer, “it doesn’t work.” Contractor officials told us that the subjective nature of the criteria and the manner in which they are applied negate the award term as a performance incentive. Both the Air Force and Army indicated that they are moving away from using award-term incentives on future contracts. The Air Force will not include such an incentive in its re-competition for the F-16 simulator training contract because, according to the DMO director, it has not been found to be a significant motivator to the contractor; experience has shown that withholding payment for poor service is a much more effective tool to induce improved performance. In addition, since a recent statutory provision limits future total contract periods of performance to 10 years, an award-term provision can no longer be used to implement long-term arrangements such as those in place for the existing simulator training contracts.

We recently reported that DOD has little evidence to support its belief that award fees improve contractor performance and acquisition outcomes and, in fact, frequently pays out most of the available award fee to contractors regardless of their performance outcomes. We also found that DOD contracts frequently included rollover provisions, where unearned award fee from one evaluation period was shifted to a subsequent evaluation period or periods, thus providing the contractor an additional opportunity to earn previously unearned fee. We recommended that DOD issue guidance on when rollover of award fee is appropriate. A March 2006 guidance on award fee contracts states, among other things, that use of rollover provisions should be the exception rather than the rule and that the decision to use rollover provisions should be addressed in the acquisition strategy, including a rationale as to why a rollover provision is appropriate.

Conclusion

Because simulator training had lost out in the internal competition for procurement funds, the Air Force and Army turned to service contracts, expecting that O&M funds would be made available to meet requirements. In the case of the Air Force, this expectation has not materialized and planned site activations have been slowed. In addition, although the Air Force and Army plan to continue with the service contract approach for simulator training, neither supported the decision with a thorough analysis of the costs and benefits of alternative approaches to delivering the training. Finally, the heightened risks associated with increased reliance on contractors to deliver simulator training calls for careful attention to contract management and oversight. Effective and well-managed incentives for motivating performance are especially important. Better government visibility into the contractors’ activities, such as preparatory tasks, during the development period is critical so that the government can understand the basis for what are essentially milestone payments during that phase. In addition, unless utilization rates are tracked in a consistent manner, the government will not know whether it is making the best use of what it is buying.

Recommendations for Executive Action

To help ensure that the best approach is used to provide the war-fighter with needed training, we recommend that the Secretary of Defense direct the Secretaries of the Air Force and Army to conduct a thorough analysis of the costs and benefits of using service contracts for simulator training to determine if it is indeed the best approach. The analysis should proactively address potential risks associated with the service contract approach and identify the level of simulator training needed to meet requirements.

To help ensure that the required training is provided to pilots, we recommend that the Secretary of the Air Force reconcile the funding level needed for simulator training with the requirements identified in the evaluation of costs and benefits of the service contract approach and take steps to allocate funds accordingly.

To help ensure that the incentives motivate contractor performance toward achieving desired training outcomes, we recommend that the Secretary of Defense direct the Secretaries of the Air Force and Army to take the following two actions: Determine whether it is in the government’s best interest to retain the award-term incentive under these service contracts.

If the award-term incentive is retained, take appropriate steps to improve the approach by reassessing the areas to be rated and the definitions of performance levels for the various grade categories. For the Air Force, improvements to the approach should include a determination as to whether to continue allowing rollover of award-term points.

To help ensure greater transparency into what the government is paying for preparatory tasks during the development phase, we recommend that the Secretary of Defense direct the Secretaries of the Air Force and Army to take the following two actions: Reassess the pricing of any up-front payments made to the contractors during the development period on future replacement or restructured contracts.

If retained, take appropriate measures to (1) create an appropriate and transparent contract payment mechanism, separate from the preparatory tasks, if development costs are to be reimbursed; and (2) increase visibility into the percentage of upfront development costs contractors are recouping from these preparatory tasks and development payments.

To help ensure that available simulator training for the warfighter is used in the most effective and efficient manner, we recommend that the Secretary of Defense take the following four actions: Direct the Secretaries of the Air Force and Army to determine whether and how simulator utilization can be increased in order to maximize use of taxpayer dollars.

Direct the Secretary of the Army to track and record monthly utilization rates on Flight School XXI contracted simulator training in order to have the data necessary to adjust training requirements and contract provisions, as necessary.

Direct the Secretary of the Air Force to revise Air Force Instruction 36-2248, Operation and Management of Aircrew Training Devices, to ensure that, for the purposes of reporting utilization rates, the usage of individual training simulators is calculated.

Direct the Secretary of the Air Force to ensure that all sites consistently track and report simulator utilization.

Agency Comments and Our Evaluation

In written comments on a draft of this report, DOD concurred with all but one of our recommendations. DOD partially concurred with our recommendation that the Army track and record monthly utilization rates on simulators at Flight School XXI. DOD stated that the service contract approach requires only that the vendor meet the programmed student training load. Nevertheless, DOD stated that the contractor is required to submit utilization data and that the data are available for use in future adjustments to the contracting strategy, requirements, or provisions. Our recommendation was intended to encourage DOD to fully understand its student training requirements and to collect the information to decide whether it needs to adjust requirements or contract provisions regarding simulator availability. Whether the utilization rates pertain to individual simulators or the student training load as a whole, we believe that the Army needs to know the extent to which it is actually using the simulator availability it is buying. DOD also offered two corrections to information in the draft, and we made changes as appropriate. DOD’s comments are included in their entirety in appendix II.

We will send copies of this report to the Secretaries of Defense, the Air Force, and the Army; appropriate congressional committees; and other interested parties. We make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff has questions concerning this report, please contact me at (202) 512-4841 or by e-mail at shamesl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report.

Appendix I: Scope and Methodology

To determine which factors led the Air Force and Army to acquire simulator training as a service contract using operation and maintenance funds, we analyzed historical documents such as acquisition plans, briefings, and decision memorandums. For the Air Force, we interviewed Air Force management, including officials at the Office of the Assistant Secretary of the Air Force, Acquisition; Aeronautical Systems Center (responsible for contracting the simulator training services) and the Air Combat Command (funds and uses the simulator training). For the Army, we interviewed officials at the Office of the Assistant Secretary of the Army for Acquisition, Logistics, and Technology; and Army officials responsible for managing the Army’s Flight School XXI initiative, including officials of the Program Executive Office for Simulation, Training and Instrumentation. We visited Langley Air Force Base, Virginia, to observe F-15C simulator training; Shaw Air Force Base, South Carolina, to observe F-16 simulator training; and Fort Rucker, Alabama, to observe the Flight School XXI helicopter simulator training. Additionally, to evaluate whether the military services adequately justified the new service contract approach, we reviewed the Office of Management and Budget’s Circular A-11, Appendix B, “Budgetary Treatment of Lease-Purchases and Leases of Capital Assets,” and Air Force and Army regulations and guidance regarding business case analyses. We also drew from our prior reviews of Department of Defense systems, in particular our recent review of the Army’s Future Combat System.

To assess whether the new approach has resulted in the planned number of simulator training sites being activated, we evaluated contract documents and information provided by the Air Combat Command and Aeronautical Systems Center to compare planned to actual schedule activations. We gathered and analyzed budget data related to program schedules and interviewed program officials. We analyzed contract documents and other program documents from Flight School XXI and discussed the schedule rebaselining with Army officials. We analyzed the Air Force’s request for proposals for the F-16 simulator training contract re-competition to determine whether key differences in the acquisition approach were incorporated.

To determine if the Air Force and Army are effectively tracking the return on their expenditure of taxpayer dollars, we analyzed simulator utilization data and military service guidance on utilization rates; analyzed contractor performance measurements, annual evaluations, and award-term plans for the simulator training contracts; and compared preparatory service costs charged to the government under each of the four Air Force contracts and the Army contract. We also interviewed contractor representatives and government officials.

Appendix II: Comments from the Department of Defense

Appendix III: Comparison of Contractor Requirements under Commercial and Non- commercial Acquisition Procedures

The following table shows differences, as set forth in the Federal Acquisition Regulation (FAR), for contractor requirements under commercial versus non-commercial acquisition procedures.

A wide selection of contract types is available in order to provide flexibility. (FAR 16.101(a))

Limited contract types are authorized. Agencies shall use firm-fixed-price contracts or fixed-price contracts with economic price adjustment. These contract types may be used in conjunction with an award fee and performance or delivery incentives when the award fee or incentive is based solely on factors other than cost. (FAR 12.207)

To implement the Services Acquisition Reform Act of 2003 (contained in Section 1432 of the National Defense Authorization Act for Fiscal Year 2004, Pub. L. No. 108-136 (2003)), a proposed amendment to FAR would expressly authorize the use of time-and- materials and labor-hour contracts for certain categories of commercial services under specified conditions. (FAR Case 2003-027, 70 Federal Register 56318, Sept. 26, 2005.)

Inspection and testing Government has right to inspect and test. (FAR 46.102 & 46.202-3)

Contracts for commercial items shall rely on contractors’ existing quality assurance systems as a substitute for Government inspection and testing before tender for acceptance unless customary market practices for the commercial item being acquired include in-process inspection. Any in- process inspection by the Government shall be conducted in a manner consistent with commercial practice. (FAR 12.208)

Price must be determined fair and reasonable through various proposal analysis techniques. (FAR 15.404-1)

While price reasonableness must be established, the contracting officer should be aware of customary commercial terms and conditions when pricing commercial items. Commercial item prices are affected by factors that include, but are not limited to, speed of delivery, length and extent of warranty, limitations of seller’s liability, quantities ordered, length of the performance period, and specific performance requirements. (FAR 12.209)

Commercial items are exempt (FAR 15.403-1(b)(3) and (c)(3))

Required for contract award and modifications unless applicable exception, such as adequate competition or prices agreed upon are based on prices set by law or regulation. Threshold for application is $550,000. (FAR 15.403-1 and -4)

The contracting officer must consider the following order of preference when a contractor requests contract financing., (a) Private financing without Government guarantee. (b) Customary contract financing. (c) Loan guarantees. (d) Unusual contract financing. (e) advance payments. (FAR 32.106)

For purchases of commercial items, financing of the contract is normally the contractor’s responsibility. (32.202-1) However, customary market practice for some commercial items may include buyer contract financing. In these circumstances, the contracting officer may offer Government financing in accordance with the policies and procedures in Part 32. (FAR 12.210)

However, government financing provided only to extent actually needed for prompt and efficient performance, considering availability of private financing and probable impact on working capital of predelivery expenditures and productions lead-times. (FAR 32.104)

Government financing of commercial purchases is expected to be different from that used for non- commercial purchases. While the contracting officer may adapt non-commercial techniques and procedures for use in implementing commercial contract financing arrangements, the contracting officer must have a full understanding of effects of the differing contract environments and of what is needed to protect the interests of the Government in commercial contract financing. (FAR 32.202-1(c)) Types of payments for commercial item purchases. (FAR 32.202-2) 1. Commercial advance payment: payment made before any performance of work (not to exceed 15 percent of contract price) 2. Commercial interim payment: payment made after some, but not all, work has been performed 3. Delivery payment: payment made for accepted supplies or services, including partial deliveries (FAR 32.001)

The Government may acquire technical data and rights in technical data for multiple purposes. Agencies shall strike a balance between the government’s need and the contractor’s legitimate proprietary interest. (FAR 27.4)

Generally, the Government shall acquire only the technical data and the rights in that data customarily provided to the public with a commercial item or process. The contracting officer shall presume that data delivered under a contract for commercial items was developed exclusively at private expense. When a contract for commercial items requires delivery of technical data, the contracting officer shall include appropriate provisions and clauses delineating the rights in the technical data in the contract. (FAR 12.211)

The Government may acquire computer software/documentation for multiple purposes. Agencies shall strike a balance between the government’s need and the contractor’s legitimate proprietary interest. (FAR 27.402)

Commercial computer software or commercial computer software documentation shall be acquired under licenses customarily provided to the public to the extent such licenses are consistent with federal law and otherwise satisfy the government’s needs. Generally, offerors and contractors shall not be required to— 1. Furnish technical information related to commercial computer software or commercial computer software documentation that is not customarily provided to the public; or 2. Relinquish to, or otherwise provide, the Government rights to use, modify, reproduce, release, perform, display, or disclose commercial computer software or commercial computer software documentation except as mutually agreed to by the parties. (FAR 12.212(a))

Compliance generally required for contractors in connection with negotiated contracts in excess of $500,000.

Cost Accounting Standards do not apply to contracts for acquisition of commercial items when they are firm-fixed-price or fixed-price with economic price adjustment. (FAR 12.214)

Contractors must disclose and consistently follow their cost accounting practices. (FAR 30.101)

In determining whether a potential awardee is a responsible contractor, per criteria in FAR 9.104-1, contracting officers may require a preaward survey when the information on hand or readily available is not sufficient to make such a determination. (FAR 9.106-1)

If the contemplated contract will involve the acquisition of commercial items, the contracting officer should not request a preaward survey unless circumstances justify its cost. (FAR 9.106-1(a))

When contracting by negotiation, the contracting officer shall insert the clause at FAR 52.215-2, Audit and Records—Negotiation in solicitations and contracts which allows contracting officer examination of costs when cost or pricing data is required or for cost-reimbursement, incentive, time- and-materials, labor-hour, or price redeterminable contracts. (FAR 15.209(b) and 52.215-2)

Commercial item contracts exempted. (FAR 15.209(b)(1)(iii))

On contracts for supplies over $10,000, contractors must adhere to provisions pertaining to minimum wages, maximum hours, child labor, convict labor, safe/sanitary working conditions. (FAR 22.602)

Not applicable. (FAR 12.503(a))

Contractor must warrant that it has not employed or retained anyone, on a contingent fee basis, to obtain this contract.

Not applicable. (FAR 12.503(a)) (FAR 3.404, 52.203-5)

Contractor must agree that it will provide a drug-free workplace (FAR 23.504(a))

Not applicable. (FAR 12.503(a))

Contractor must report on its affirmative actions to employ and advance covered veterans (FAR 22.1302(a))

Law’s limitation on use of appropriated funds for contracts with entities not meeting veterans employment reporting requirements is not applicable. (FAR 12.503(a))

Contracts for services must prohibit contractor activities regarding, and require contractor policies to combat, severe forms of trafficking in persons, the procurement of commercial sex acts, and use of forced labor. (FAR 22.1705)

Not applicable. (FAR 12.503(a))

Contract clause required providing that contractors employing laborers or mechanics are required to compensate them for overtime. (FAR 52.222-4)

Requirements for a certificate and contract clause related to the Act are not applicable. (FAR 12.503(b))

Contract clause requires prime contractors to (1) have in place and follow reasonable procedures designed to prevent and detect violations of the Act; and (2) cooperate fully with any Federal agency investigating a possible violation of the Act. (FAR 3.502-2(i))

Requirements for a clause and certain other requirements related to the Act are not applicable. (FAR 12.503(b))

Contracts must include clause requiring use of U.S- Flag Air Carriers by government contractors when available (FAR 47.405)

Requirement for a clause related to the Act is not applicable. (FAR 12.503(b))

Contracts must include clause precluding contractors from restricting direct subcontractor sales to the Government. (FAR 3.503-2 and 52.203-6(a))

Contractors may restrict subcontractors’ sales to the Government, as long as the Government is treated no differently than other prospective purchaser. (FAR 52.203-6, Alternate I)

Generally, contracting officer permitted to make unilateral changes within the scope of the contract and to require continued contractor performance of the contract as changed. (FAR 43.201)

Changes may be made only by written agreement of the parties (bilateral). (FAR 12.301(b)(3); 52.212- 4(c))

Generally, termination costs for fixed-price contracts limited to total contract price less payments made or to be made under contract plus reasonable costs incurred in performance of work terminated, to include fair and reasonable profit, and reasonable settlement costs. Cost principles and procedures of FAR Part 31 apply to costs. (FAR 49.502(b); 52.249- 2)

Termination costs limited to percentage of contract price reflecting percentage of work performed prior to termination plus reasonable charges resulting from termination. For payments thereunder, contractor not required to comply with cost accounting standards or contract cost principles in FAR Part 31. (FAR 12.301(b)(3); 52.212-4(l))

Appendix IV: Potential Duration of Air Force and Army Simulation Contracts

Aug. 2003 to July 2016 (13 years)

Appendix V: GAO Contact and Staff Acknowledgments

Acknowledgments

In addition to the individual named above, Michele Mackin, Assistant Director; Marie Ahearn; Christine Bonham; Gary Delaney; Carlos Diz; Benjamin Federlein; Victoria Klepacz; and Sanford Reigle made key contributions to this report.