Defense Acquisitions: Status of DOD's Implementation of Independent Management Reviews for Services Acquisitions
Released: 2010-01-28T00:00:00

Letter

DOD Has Implemented Guidance That Generally Addresses Legislative Requirements

To meet the legislative requirements regarding independent management reviews, DOD issued guidance and instructions providing for a peer review process for services acquisitions. DOD’s guidance generally addresses requirements prescribed in the Act to develop a process to evaluate the specified contracting issues, but according to DOD officials, DOD has not yet determined how the department plans to disseminate lessons learned or track recommendations that result from the newly instituted reviews. DOD officials expect to further refine their processes, including developing a more formal means for disseminating lessons learned and tracking recommendations as DOD assesses its initial experiences with peer reviews. Through the first year of implementation, DPAP, which is responsible for conducting reviews of acquisitions over $1 billion, had conducted 29 peer reviews on 18 services acquisitions. Similarly, the military departments, which are responsible for conducting reviews of their acquisitions under $1 billion, issued guidance that provides for peer reviews at various levels within the departments based on dollar values. The military departments could not, however, determine the exact number of peer reviews conducted because of the absence of comprehensive reporting processes. Further, as peer review processes evolve, the military departments are considering ways to disseminate lessons learned and track recommendations.

DPAP Has Established a Peer Review Process for Services Acquisitions

DPAP issued a memorandum in September 2008 establishing a peer review process to fulfill the requirement for an independent management review of contracts for services. The requirement for a peer review process was subsequently incorporated into DOD Instruction 5000.02, Operation of the Defense Acquisition System, in December 2008. The guidance states that these reviews are intended to ensure consistent and appropriate implementation of policy and regulations, improve the quality of contracting processes, and facilitate sharing best practices and lessons learned. According to DOD officials, peer reviews by design are a means of improving individual acquisitions and not necessarily a tool for strategically managing DOD’s services portfolio.

Under DPAP’s guidance, peer reviews supplement its existing process to review and approve services acquisitions. Pursuant to congressional direction, DOD had previously established a management review process that was intended to ensure that DOD services acquisitions are based on clear, performance-based requirements with measurable outcomes and that acquisitions are planned and administered to achieve intended results. In these management reviews, DPAP assesses and approves the acquisition strategies submitted by the military departments or defense agencies for obtaining contractor-provided services estimated to be valued at $1 billion or more. Once the acquisition strategies are approved, DOD contracting offices may continue the acquisition process, including soliciting bids for proposed work and subsequently awarding contracts. DOD may award different contract types to acquire products and services, or issue task orders under existing contracts. In November 2009, we reported that the number of contracts and task orders issued after the acquisition strategies were approved was significant. For example, we reported that nearly 1,900 task orders were issued under the seven professional and management support services acquisitions we reviewed.

DOD generally conducts peer reviews at three key points in the acquisition process prior to contract award—prior to issuance of the solicitation (phase 1), prior to request for final proposal revisions (phase 2), and prior to contract award (phase 3)—and is to conduct periodic post-award reviews (phase 4) (see fig. 1).

In February 2009, DOD issued guidance that clarified the relationship between the management reviews and the peer reviews. For example, the guidance identifies specific issues to assess and the criteria for the reviewers to use during the management reviews or pre-award peer reviews. According to the guidance, some contracting issues identified in the Act, such as contract type and competition, are to be assessed during the management reviews. Conversely, other contracting issues identified in the Act, including requirements definition and the extent of the agency’s reliance on contractors to perform functions closely associated with inherently governmental functions, are to be assessed during pre-award peer reviews. The pre-award peer reviews also are to evaluate several elements of the source selection process that are not specified in the Act, such as the clarity and consistency of the documentation. Further, the guidance established review criteria for post-award reviews that address each of the contracting issues identified in the Act. For example, during post-award reviews, reviewers are to assess the extent to which the contracting office was able to achieve competition for orders and whether it was using appropriate contract types, well-defined requirements, and appropriate cost/pricing methods.

According to DOD officials, in conducting these reviews, DPAP convenes a peer review team consisting of three to five members. Officials said that the teams are generally chaired by a deputy director within DPAP and include participation from senior contracting officials from the military departments and defense agencies as well as legal advisors from the Office of the Secretary of Defense’s General Counsel. The teams review acquisition documents prior to an on-site review and hold discussions with contracting officers over multiple days. Upon completion of the on-site review, peer review teams develop summary memoranda that include observations and recommendations.

The February 2009 guidance indicated that DPAP is to review services acquisitions with an expected value of over $1 billion. In addition, DPAP may review acquisitions under that threshold that it has designated as special interest because of the nature or sensitivity of the services to be acquired. According to DOD officials, DPAP does not have a capability to independently identify acquisitions that will require its review, but rather relies on the military departments and defense agencies to notify DPAP of acquisitions that will exceed the threshold. DPAP officials noted that some reviews were not conducted because the military departments did not notify DPAP that a peer review was necessary.

DPAP officials stated that they are currently focusing on the pre-award peer reviews and are phasing in post-award peer reviews. As of September 30, 2009, DPAP had conducted 29 peer reviews for 18 services acquisitions. Because the peer review process was only implemented in September 2008, no single acquisition has been subject to all phases of the peer review process and no acquisition has been peer reviewed in both the pre- and post-award phases. While most of the reviews have focused on proposed acquisitions for which the initial contract had not yet been awarded, DPAP has also conducted two phase 3 peer reviews for proposed task orders valued at over $1 billion that were to be issued under an existing contract that had previously been reviewed. DPAP has not yet determined if it will establish a policy for conducting peer reviews for all individual task orders over this amount in the future. For the 29 peer reviews of services acquisitions that DPAP conducted, figure 2 shows when each review occurred and the corresponding milestone. For example, DPAP conducted a phase 1 peer review prior to the issuance of the solicitation for 12 of the 18 services acquisitions.

Our review of the summary memoranda of the pre-award peer reviews that DPAP conducted as of September 30, 2009, found that review teams generally documented the evaluation of the use of contracting mechanisms and, to a lesser extent, the use, management, and oversight of subcontractors. DPAP officials noted that other contracting issues may have been discussed during pre-award site visits and not included in the summary memorandum because the peer review team did not identify any concerns that warranted inclusion. Further, we found that review teams made several related recommendations, as illustrated in the following examples: One pre-award peer review team recommended that the contracting office reconsider the number of contracts that it had proposed be awarded under an acquisition. In this case, the contracting office had proposed limiting the number of contracts to three prior to knowing what proposals and business arrangements would be submitted by industry. The peer review team noted that this may unduly restrict flexibility of the military department. Further, the team was unsure if documentation to support the limitation on contract number would be sufficient to withstand a bid protest from an unsuccessful offeror.

Another pre-award peer review recommended that the contracting office increase its use of subcontractors and encourage the prime contractors to establish mentor-protégé relationships with their subcontractors to bring more qualified contractors into an industry.

Our review of the summary memoranda for the three post-award peer reviews conducted by DPAP found that consistent with guidance, the review teams evaluated all the contracting issues identified in the Act. All three summary memoranda listed the required contracting issues and then reported the peer review teams’ observations and recommendations for the contracting offices to consider for the acquisition, as illustrated by the following examples: One post-award peer review team recommended that the contracting officer modify the contract to include provisions requiring the contractor to provide information on pass-through charges for all future task orders issued. At the time of the peer review, the contract did not contain a clause requiring the contractor to provide such information, and therefore the government was unable to determine the extent of pass-through charges and whether they were excessive.

Another post-award team recommended that the contracting office reduce the use of time-and-materials task orders. In this case, the acquisition strategy envisioned that most of the work would be performed through fixed-priced task orders; however, time-and- materials task orders accounted for 62 percent of the value of orders issued under the contract in the first 2 years of performance.

While DPAP’s guidance noted that the recommendations made during peer reviews are advisory in nature, it also states that contracting offices are to document in the contract file the disposition of all pre-award peer review recommendations prior to contract award. The guidance does not address recommendations made during post-award reviews. According to DOD officials, contracting offices generally accept recommendations provided by the peer review teams. DPAP officials said that if the contracting office decides not to accept a peer review team’s recommendation, the contracting officer is expected to document the reason in the contract file and provide a copy to DPAP.

In addition to providing recommendations to address potential issues in proposed acquisitions, the peer review teams have also identified some best practices. For example, in one summary memorandum the team called attention to the contracting office’s post-award performance plan for the acquisition, which specified how the office intended to evaluate and assess contract performance to maintain effective contract surveillance procedures. The team noted that the plan allowed real-time access to detailed cost performance data when combined with regular surveillance. According to officials, DOD, however, has not yet issued guidance establishing procedures to systematically track the recommendations made by peer review teams or disseminate best practices as required by the Act. DOD officials noted that to date, sharing lessons learned from peer reviews has largely occurred through word of mouth or through conferences. For example, at a December 2009 conference for senior DOD contracting officials, DPAP presented an update on its peer review process that included a discussion of lessons learned. To identify methods to better disseminate trends, lessons learned, and best practices identified during peer reviews, in August 2009 DPAP established a subcommittee within the Panel on Contracting Integrity. DPAP officials expect that the subcommittee will report on its findings in 2010. Further, an official stated that DPAP plans to consider ways to track the implementation of recommendations made during peer reviews.

Military Departments’ Peer Review Processes Are Evolving

The September 2008 DPAP guidance required the military departments to establish their own procedures for conducting pre- and post-award peer reviews on acquisitions under $1 billion, but provided the flexibility to the services to tailor the process to best meet their needs. In response, the Air Force issued its guidance in January 2009, the Navy in March 2009, and the Army in April 2009. The military departments’ policies varied in such areas as the frequency and timing of the reviews and the organizational levels delegated responsibility for conducting the reviews. For example, the Air Force conducts up to five pre-award pre-award peer reviews whereas the Army conducts two (see fig. 3). The military departments plan to refine their policies as they gain experience with the peer review process.

According to officials, both the Air Force and Army modified existing pre- award reviews to incorporate the peer review requirements. The existing reviews were mandatory steps in each department’s contract award process and, as such, focused on the proposed acquisition’s contracting approach, source selection process, and readiness to issue a contract solicitation. Air Force officials stated that the department previously had a post-award review process that focused on cost, schedule, and performance metrics, which was revised to incorporate peer review requirements. Army officials noted that the Army has focused its attention on implementing pre-award peer reviews, but has not yet established a post-award peer review process. These officials noted that the Army plans to issue guidance on conducting post-award reviews in 2010.

In contrast, the Navy developed a new process, modeled on DPAP’s process, to review proposed services acquisitions. Navy officials are considering making some refinements to this process. For example, at the time of our review the Navy had not yet determined the optimal timing of its post-award peer reviews. The department was trying to determine a point at which there had been enough contract performance to evaluate the contractor while still allowing the contracting officers sufficient time to implement any peer review team recommendations prior to exercising an option year.

While DPAP was not required to approve the military departments’ guidance, DPAP officials reported that the guidance issued by the military departments was consistent with the intent of the September 2008 guidance. There are differences, however, in how the military departments addressed certain issues. For example, each of the military departments delegated responsibility for conducting peer reviews to commands and organizational units within their departments based on expected acquisition value. In that regard: The Air Force delegated responsibility for conducting peer reviews to its major commands for proposed services acquisitions valued from $50 million to $1 billion.

The Army delegated responsibility to the head of the contracting activity within each of its commands for conducting peer reviews for services acquisitions valued from $250 million to $1 billion. Similarly, it identified the principal assistant responsible for contracting as being responsible for conducting peer reviews for acquisitions from $50 million to $250 million.

The Navy delegated responsibility to the Deputy Assistant Secretary of the Navy – Acquisition and Logistics Management (DASN-A&LM) for conducting peer reviews for acquisitions valued from $250 million to $1 billion, while individual commands are responsible for conducting reviews of acquisitions valued from $50 million to $250 million.

Further, the Air Force does not require peer reviews on noncompetitive acquisitions—in other words, on contracts awarded using other than full- and-open competition. Air Force officials explained that such contracts are already reviewed under a separate process and therefore believed that an additional peer review would be unnecessary. Similarly, both the Air Force and Army allow the offices responsible for conducting reviews to waive peer reviews under certain circumstances, whereas the Navy does not provide for a waiver process. Air Force guidance allows peer reviews to be waived based on acquisition/source selection history, such as for recurring acquisitions and where there is no history of bid protests. The Army also allows peer reviews to be waived but did not specify in its guidance which acquisitions could be waived.

As of September 2009, the military departments reported conducting hundreds of peer reviews for services acquisitions, but the departments do not have comprehensive processes for determining the exact number of reviews conducted. Specifically: The Navy reported that it had conducted 257 peer reviews for services acquisitions, including 5 post-award reviews. The Navy could not identify how many of the reviews conducted by the commands occurred by September 30, 2009. DASN-A&LM conducted its first 4 peer reviews on September 22, 2009.

Though the Air Force did not know the specific number of peer reviews conducted, officials noted that it had conducted up to five pre- award reviews on approximately 85 services acquisitions as of September 30, 2009.

Army officials stated that though commands had conducted pre-award peer reviews, an exact number of reviews could not be identified because the Army does not have a reporting process. The Army also acknowledged that it did not conduct any post-award reviews because it has not yet established a post-award peer review process.

As peer review processes evolve, the military departments are considering ways to disseminate lessons learned and track recommendations. For example, Navy officials said the department is waiting to see the results of initial reviews and will then develop additional guidance to address lessons learned made during peer reviews. Army officials stated that the department plans to address recommendations and lessons learned in 2010 when it issues guidance on post-award reviews. Finally, Air Force policy requires commands to submit annual reports to the Secretary of the Air Force – Acquisition and Contracting Policy that are to include major issues identified during pre-award peer reviews and the resolutions taken.

Concluding Observations

DOD’s guidance implementing a peer review process for major services acquisitions at the departmental level generally addresses the requirements prescribed by the Act. While DOD has derived benefits from these initial reviews, it has also recognized that there are issues that still need to be addressed, such as how to track recommendations and disseminate lessons learned. Further, at this stage, DOD’s focus has been on evaluating acquisition strategies and proposed contracts at the pre- award stage. DOD has conducted relatively few post-award reviews, in which DOD assesses how well it is managing the contractor’s actual performance. A key issue is whether and how to apply the peer review process to task orders through which DOD obtains much of its contractor- provided services. Few of these are large enough to reach the $1 billion DOD review threshold, but below the threshold they could be so numerous as to overtax the departments’ peer review processes. Addressing these issues, as well as those at the military department level, is important if DOD is to achieve its stated objectives for peer reviews— ensuring consistent and appropriate implementation of policy and regulations, improving the quality of contracting processes, and facilitating sharing best practices and lessons learned—on a more strategic or enterprisewide basis rather than limiting the peer reviews’ benefits to the individual acquisitions being reviewed. Although we are not making any recommendations because DOD plans to address these issues, resolving these concerns in a timely manner is essential if DOD is to maximize the benefits of the peer review process.

Agency Comments

DOD provided written comments on a draft of this report. In its comments, DOD stated that peer reviews had improved the quality of its significant business arrangements. DOD indicated that it will continue to refine its peer review process to better disseminate trends, lessons learned, and best practices that are identified during peer reviews. DOD provided a technical comment, which was incorporated into the report. DOD’s comments are reprinted in appendix II.

We are sending copies of this report to the Secretary of Defense; the Secretaries of the Air Force, Army, and Navy; and interested congressional committees. The report also is available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me at (202) 512-4841. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Scope and Methodology

Section 808 of the National Defense Authorization Act for Fiscal Year 2008 (the Act) directs GAO to report on the Department of Defense’s (DOD) implementation of its guidance and implementing instructions providing for periodic management reviews of contracts for services. In response to this mandate, we (1) assessed the extent to which DOD’s guidance addressed the Act’s requirements at the department level and how the guidance was implemented and (2) determined the status of actions taken by the military departments pursuant to DOD’s guidance.

To do so, we reviewed DOD’s September 2008 and February 2009 guidance issued by the Under Secretary of Defense for Acquisition, Technology and Logistics’ Office of Defense Procurement and Acquisition Policy (DPAP). We compared the guidance and instructions to the requirements stipulated in Section 808 of the Act. The September 2008 guidance indicated that peer reviews were to be conducted for both supplies and services. As the Act’s requirements were specific to services acquisitions, we limited our analysis to services. We also obtained guidance and implementing instructions issued by the Departments of the Air Force, Army, and Navy. We interviewed officials from DPAP and the Departments of the Army, Navy, and Air Force to gain further insight into how each organization developed its guidance and instructions. DOD’s September 2008 memorandum also indicated that defense agencies were required to develop their own guidance. While these were outside the scope of our review, DPAP officials indicated that 13 of 17 defense agencies that DPAP believed would be required to develop guidance had done so at the time of this review.

We obtained information on the number of peer reviews on services acquisitions that DPAP and the military departments reported they had conducted as of September 30, 2009. DPAP was able to identify the number of reviews that it had conducted. We determined this information to be sufficiently reliable for the purposes of our review. The Air Force provided an approximate number of acquisitions that had been reviewed but could not identify the number of individual peer reviews conducted. The Army did not provide any information on the specific number of reviews conducted. The Navy provided information on the number of reviews it had conducted but could not specify how many had been conducted as of September 30, 2009. We could not independently verify the information provided by the military departments because of the lack of available documentation.

To determine the nature of the discussions and the issues addressed during peer reviews, we obtained the summary memoranda from each of the 29 peer reviews conducted by DPAP as of September 30, 2009. These 29 memoranda represented 18 unique acquisitions, as DPAP had reviewed some acquisitions more than once. Twenty-six of the memoranda were for pre-award peer reviews and 3 were for post-award reviews. We analyzed summary memoranda from each of the 29 peer reviews to determine the topics discussed in the memoranda, focusing specifically on the contracting issues identified in the Act. We also interviewed DPAP officials who chaired or participated in these reviews to obtain their views on the peer review process.

We conducted this performance audit from October 2009 through January 2010 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, Timothy DiNapoli, Assistant Director; E. Brandon Booth; Morgan Delaney Ramaker; Christopher Mulkins; Thomas Twambly; and Alyssa Weir made key contributions to this report.