Defense Pilot Programs: DOD Needs to Improve Implementation Process for Pilot Programs
Published: 2003-07-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In fiscal years 1999, 2000, and 2003, the Congress authorized pilot programs to help the Department of Defense (DOD) laboratories and test centers explore innovative business partnerships and human capital strategies. Congressional concerns about DOD's implementation of the pilot programs have been growing. The Congress mandated that GAO review pilot program implementation. GAO (1) identified the pilot initiatives proposed and their current status, (2) examined factors that affected implementation, and (3) assessed implementation challenges the 2003 pilot program faces.
section_title: What GAO Found
paragraphs: The 1999 and 2000 pilot programs have not worked as intended. Since their inception, 178 initiatives have been proposed by the participating laboratories and test centers but only 4--or 2 percent--were implemented under the pilot programs. Participants proposed initiatives covering a variety of areas, including business-like practices, partnerships, and human capital innovations. The pilot programs were not effective because DOD lacked an effective implementation process and proposed human capital initiatives were not consistent with statutory provisions. First, DOD did not provide standardized guidance on proposal requirements, coordinate proposals, or clarify decision-making authority for proposal review and approval. Furthermore, DOD did not designate a strong focal point to provide assistance and advice to participants and advocate process improvements. The lack of a strong focal point exacerbated other process gaps. Second, DOD attorneys advised that the pilot programs did not provide authority to make most of the proposed human capital changes. Implementation of the new 2003 pilot program faces several challenges. First, DOD has not addressed implementation problems. For example, clear guidance is still lacking and decision-making authority is still unclear. Second, the 2003 pilot program provides no change in authority concerning human capital initiatives. Finally, laboratories and test centers may be reluctant to participate. Many participants in the earlier pilots told us they were discouraged by their experience and consequently unwilling to repeat it.

section_title: Letter
paragraphs: Over the last decade a number of studies have raised concerns that  dwindling budgets and an aging workforce have contributed to serious  shortfalls in the infrastructure and capabilities of Department of Defense  (DOD) laboratories and test centers. In fiscal years 1999 and 2000, the  Congress enacted legislation aimed at helping DOD laboratories and  test centers address problems by undertaking pilot programs to explore  innovative partnerships and human capital strategies. In fiscal year 2003,  the Congress extended the 1999 and 2000 pilot programs until 2005 and  enacted a new pilot program that runs until 2006. However, congressional  concerns about pilot program implementation have been growing.  Consequently, the Senate Committee on Armed Services directed us to  review the implementation of the 1999 and 2000 pilot programs. In  response, this report (1) identifies initiatives proposed to date and  determines their current status, (2) examines factors that affected  implementation of proposed initiatives, and (3) assesses implementation  challenges the new 2003 pilot program faces.
subsections: section_title: Background
paragraphs: The United States has a long history of military research and development.  To help conduct and manage this research, DOD has a diverse network  of 80 in-house laboratories and 26 test centers. Their missions range  from basic scientific research to direct technical support to operational  commands. The management, operations, and funding for these disparate  laboratories and test centers also vary among the services.
Over the past decade, several organizations, panels, and commissions  have identified significant personnel and resource problems facing the  laboratories and test centers. For example, several studies found that  the laboratories needed more flexibility in personnel rules governing the  scientific workforce in order to attract and retain staff. Similarly, several  recent studies identified problems with declines in investment and  infrastructure, resulting in outdated facilities and technical equipment.
To help the laboratories and test centers with these problems, the  Congress enacted legislation in fiscal years 1999 and 2000 establishing  pilot programs for laboratories and test centers to propose innovative  partnerships, business-like practices, and human capital initiatives. The  1999 pilot program focused on partnerships and business-like practices,  while the 2000 program focused more on human capital initiatives.  Together, the two pilot programs authorized the Secretary of Defense to  provide one laboratory and one test center in each service the authority to    explore innovative methods for partnering with universities and private  sector entities to conduct defense research and development;    attract a workforce balance between permanent and temporary personnel  and with an appropriate skill and experience level;    develop or expand innovative methods of operation that provide more  defense research for the dollar; and    waive any restrictions on these methods that are not required by law.
A total of 10 laboratories and test centers from all 3 services participated  in the pilot programs. They are listed in appendix I.
Both programs were authorized for 3 years. The 1999 pilot expired in  March 2002; the 2000 pilot, in March 2003. For both programs, DOD  was required to submit preliminary and final reports to the Congress  on program activities. The preliminary report for the 1999 program was  submitted in July 1999. However, as of the date of this report, the three  other reports have not been submitted.
In fiscal year 2003, the Congress authorized another 3-year pilot program  and extended the 1999 and 2000 pilot programs until 2005. Under the new  2003 pilot program, the Secretary of Defense is to provide one laboratory  and one test center in each service the authority to    use innovative personnel management methods to ensure that the  participants can employ and retain an appropriately balanced workforce,  and effectively shape the workforce to fulfill the organization mission;    develop or expand innovative methods of using cooperative agreements  with private sector and educational organizations to promote the  technological industrial base for critical defense technologies and  facilitate the training of a future scientific and technical workforce; and    waive any restrictions not required by law.
As of May 2003, DOD had not identified any participants for the 2003 pilot  program.
The 2003 legislation also requires DOD to issue three reports, including  a January 2003 report on its experience with the 1999 and 2000 pilot  programs, barriers to implementation of these programs, and proposed  solutions to overcome these barriers. According to DOD officials, this  report has been drafted, but as of May 2003, it had not been submitted to  the Congress.
section_title: Many Initiatives Were Proposed but Few Were Implemented under Pilot Programs
paragraphs: Since the inception of the pilot programs in 1999, 178 initiatives have  been proposed, but only 4—or 2 percent—have been implemented  under the pilot programs. Participating laboratories and test centers  proposed initiatives covering a variety of areas, including business-like  practices, partnerships with industry and academia, and human capital  innovations. We found that laboratories focused many of their proposals  on human capital innovations, while test centers tended to concentrate on  business-like practices and partnerships.
subsections: section_title: Range and Volume of Proposed Initiatives
paragraphs: Over the course of the 1999 and 2000 pilot programs, the laboratories  and test centers proposed 178 human capital, business, and partnership  initiatives. As shown in table 1, slightly over half of the initiatives dealt  with human capital and the remainder dealt with business-like practices  and partnerships.
Overall, the laboratories proposed substantially more initiatives than did  the test centers. Furthermore, the laboratories and test centers focused  on different types of initiatives. The laboratories more often proposed  human capital initiatives, while the test centers overwhelmingly focused  on business and partnership initiatives. Laboratory officials told us that  they are especially concerned about attracting top-quality scientists to  replace a retiring workforce. Test center officials told us that they are  focused on modernizing their infrastructure and developing new methods  of sharing the cost of operations.
Proposals for business-like practices included many initiatives to  streamline or improve local operations. Some initiatives focused on  expanding the use of innovative techniques such as other transactions  or cooperative agreements. Several other proposals sought the authority  to reinvest fees or revenues into facilities revitalization. For example, one  Navy laboratory proposed imposing a surcharge for its services and using  that revenue to fund capital investments, and an Air Force laboratory  proposed using facility construction as a valid in-kind contribution under  cooperative agreements.
Partnership proposals included initiatives such as collaborative research  agreements with Arnold Engineering Development Center and the  University of Tennessee Space Institute to create a formal business bond  to pursue research in laser-induced surface improvement technology and  university flight research.
The Army’s Aberdeen Test Center proposed a limited liability company.  Under this concept, industry, academia, and government would form a  profit-making company to conduct research and testing at the installation.  The test center proposed using its share of the profits to reinvest in the  infrastructure at Aberdeen.
Several human capital initiatives focused on recruiting and retention  flexibilities as well as additional voluntary separation incentives.  These proposals included initiatives to streamline hiring of experts  and consultants; accelerate promotions for scientists and engineers;  provide retention bonuses for key scientists; and hire students directly  after graduation. Several participants submitted proposals for direct  hire authority to allow faster hiring of scientists, and several submitted  proposals for voluntary retirement incentives as a mechanism for  reshaping the workforce.
section_title: Few Proposals Were Implemented under Pilot Program Authorities
paragraphs: Almost none of the 178 proposed initiatives were approved and  implemented using the pilot programs’ authorities. As figure 1 shows,  only 2 percent—or 4 proposals—were implemented under the pilot  programs. In contrast, 74 percent were blocked or dropped during the  review process or remain on hold awaiting resolution.
The four implemented initiatives were    donating laboratory equipment directly to local schools,    waiving top-level certification of certain service agreements with    streamlining cooperative agreements to facilitate collaborative work  agreements with outside activities, and    granting temporary relief from some mandatory personnel placement  reviews.
Officials at the laboratories that proposed these initiatives told us that they  were considered minor changes with little impact on the larger problems  facing the laboratories.
Twelve times as many initiatives—24 percent—were implemented  using different authorities than the pilot programs. For example, several  laboratories requested the authority to appoint retired military members  to civilian positions without having to wait the required 180 days. This  requirement was waived using a different authority than the pilot  programs. Another human capital initiative—to appoint senior scientists  from private industry—was authorized by subsequent legislation. In the  business/partnership category, the 46th Test Group at Holloman Air Force  Base used other authorities to negotiate a complex leasing arrangement  with industry to install a radar test facility at White Sands Missile Range.  This effort took several years and overcame many contractual and  regulatory barriers. In addition, a Navy laboratory streamlined foreign  license applications using another authority.
section_title: The Pilot Programs Were Not Effective for Two Primary Reasons
paragraphs: The low level of implementation of the proposed initiatives occurred for  two primary reasons. First, DOD did not develop an effective process for  implementing the pilot programs. Second, DOD determined that proposed  human capital initiatives—for example, requests for the authority to hire  directly or offer voluntary retirement incentives—were in conflict with  statutory provisions.
subsections: section_title: Lack of an Effective Implementation Process
paragraphs: DOD did not provide standardized guidance on proposal requirements  or feedback for improving proposals; coordinate or prioritize proposals;  or clarify decision-making authority for proposal review and approval.  DOD also did not designate a strong focal point to coordinate the pilot  programs, advocate process improvements, and provide assistance and  advice to participants. The lack of a strong focal point exacerbated other  process gaps.
subsections: section_title: Lack of Guidance and Coordination
paragraphs: According to officials at DOD laboratories, test centers, and headquarters,  DOD did not provide standardized guidance on proposal requirements or  feedback for improving proposals (or, in many cases, information on the  status of proposals submitted for approval). Proposals often lacked  specificity and detail. Many were broadly conceptual or generic in nature  and lacked a detailed business case that linked their contribution to  overall objectives for the pilot programs. For example, a proposal to  permit scientists to serve in a leadership role in professional societies  failed to include details of the problems encountered, and the potential to  improve operations. Similarly, several proposals for direct hire authority  failed to include a business case to explain what specific needs this  authority would address or how it would address them. Lack of specificity  and business case detail led to the failure of many initiatives to win  approval. DOD attorneys told us that many proposals were so vague that it  was impossible to determine whether or not the proposed initiatives could  meet legal requirements.
At a department level, DOD also did not coordinate or prioritize proposals,  thereby precluding decisions on how best to pursue common interests  and issues such as direct hiring authority or forming partnerships with  universities. Instead, each participant submitted proposals individually,  and thus multiple independent proposals were often submitted for the  same or similar issues. DOD attorneys pointed out that it would have  been more effective to group proposals by common theme and prioritize  them. They believed a unified approach and prioritized proposals with  clearly written, specific plans for solving well-defined problems would  have enabled them to more effectively assist participants with resolving  legal issues.
section_title: Unclear Decision-Making Authority
paragraphs: DOD did not clarify decision-making authority for proposal review and  approval. Many organizations and individuals were stakeholders in  proposal review and approval, and they often had differing management  structures, concerns, and interests. Stakeholders included military and  civilian leaders, attorneys, and human capital and personnel staff at  several levels: the local installation where participating laboratories and  test centers were housed; the individual service; and OSD. The roles  and decision-making authority of the various stakeholders were  never negotiated and clarified. As a result, many players at multiple  organizational levels had—and took—an opportunity to say “no” to a  particular proposal, but it remained unclear who had the authority to  say “yes.”
For example, some participants believed that the pilot program legislation  gave the director of a participating laboratory or test center the authority  to approve a proposed initiative. OSD officials, however, believed that the  proposed initiatives had to be approved at higher levels. The role of the  services was also unclear. Some laboratory and test center directors  initially sent proposals directly to OSD’s Directorate of Defense Research  and Engineering (DDR&E), bypassing their service headquarters. Others  sent proposals to their service headquarters for approval before  submitting the proposals to DDR&E. Eventually, however, each of the  service headquarters decided to become more heavily involved in the  approval process and provide service-level responses to proposals.  These service-level responses often came into play after proposals had  been sent directly to DDR&E for approval, further complicating the  approval process.
Within OSD, both DDR&E and Personnel and Readiness (P&R) had  substantial stakes in the human capital proposals—DDR&E because it  is charged with oversight and management of defense laboratories and  P&R because it has the authority within DOD for human capital issues.  However, DDR&E and P&R never agreed on a process for approving  proposals. In addition, for the past year P&R’s attention has been focused  primarily on developing DOD’s proposed new civilian human capital  management system, the National Security Personnel System (NSPS),  which the Secretary of Defense recently submitted to the Congress. DOD  officials believe that, if enacted, NSPS will provide flexibility to make  necessary human capital changes. The Undersecretary of Defense P&R  directed that implementation of new personnel initiatives be placed on  hold during the development of NSPS so that the existing system could be  studied to identify needs and best practices. Consequently, P&R officials  believed it would be premature for DOD to implement new personnel  initiatives during this time.
section_title: No Strong Focal Point
paragraphs: DOD did not designate a strong focal point to coordinate the pilot  programs, advocate process improvements, and provide assistance and  advice to participants. This exacerbated the other process gaps. Without  such a focal point, participants found their own individual ways to develop  proposals and get them reviewed. Several officials agreed that a strong  focal point would be helpful. For example, DOD attorneys stated that the  laboratories or someone acting as their focal point needed to define the  issues they wanted to resolve. The attorneys noted that a focal point  could have more successfully drawn upon their expertise and experience  with addressing legal challenges in other innovative programs  (e.g., demonstration projects). Some pilot program participants also  agreed a strong focal point was needed, but they had some concerns  regarding the amount of influence and authority he or she should have.
section_title: Human Capital Proposals Were in Conflict with Existing Statutory Provisions
paragraphs: According to officials at DOD laboratories, test centers, and headquarters,  human capital initiatives were generally in conflict with title 5 of the  United States Code. Title 5 provides the framework for standard and  equitable personnel practices across the federal government and is the  current foundation for management of the DOD civilian workforce. Over  time, the Office of Personnel Management has added implementing rules  and regulations to the framework. Proposed human capital initiatives  often sought relief from these provisions, for example, requests for the  authority to hire directly or offer voluntary retirement incentives.
However, after reviewing the legislation, the DOD Office of General  Counsel advised that the 1999 and 2000 legislation did not provide the  authority to waive personnel rules based on title 5 provisions. Rather, the  office advised that the pilot programs’ authorities allow only for changes  that could already be accomplished under existing DOD regulations. In  other words, the pilot programs did not provide any new or additional  authority to waive existing personnel rules and regulations grounded in  title 5. Consequently, absent statutory authority beyond that provided by  the pilot programs, human capital proposals in conflict with title 5 and its  implementing rules and regulations could not be implemented. Many  initiatives fell into this category.
section_title: The 2003 Pilot Program Faces Implementation Challenges
paragraphs: The 2003 pilot program faces several implementation challenges. First,  as of May 2003, DOD had not addressed implementation problems. Thus,  proposals made via the 2003 pilot program will face the same obstacles as  previous proposals.
Second, human capital initiatives will continue to face title 5 challenges.  Like the earlier legislation, the 2003 legislation does not provide DOD any  new authority. Hence, initiatives proposed under the 2003 pilot program  will encounter the same statutory restrictions as previous initiatives. P&R  officials believe that, if implemented, NSPS will provide the flexibility to  make necessary human capital changes, thereby eliminating the need for  the pilot programs in this area. However, NSPS has not yet been enacted,  and if enacted, it will still require an implementation process.
Finally, laboratories and test centers may be reluctant to participate in the  new pilot program. Many participants in the earlier pilots told us they were  discouraged by their experience and consequently unwilling to repeat it.  Some expressed frustration with the lack of guidance and feedback on  their proposals; others questioned whether management was really  committed to the pilot program. Even those few participants that had  proposals approved were wary of expending additional resources on  another pilot program.
section_title: Conclusions
paragraphs: While DOD appears to recognize a need to address human capital and  business operations issues specific to laboratories and test centers, it has  not effectively managed the pilot programs. If DOD intends to use the  pilot programs to address laboratory and test center issues, it will have to  address the factors—both process and statutory—that blunted previous  proposals made through the pilot programs. The small volume of approved  proposals, coupled with DOD’s not providing status reports required by  the Congress, has left the Congress uninformed about what objectives  DOD would like to achieve with the laboratories and test centers, how it  plans to achieve those objectives, and what vehicles it plans to use. This  information will be important to the success of any future actions.
section_title: Recommendations for Executive Action
paragraphs: We recommend that by March 31, 2004, the Secretary of Defense inform  the Congress of DOD’s objectives regarding human capital and business  operations in the laboratories and test centers, how it plans to meet these  objectives, and what vehicles it will use to meet them.
We also recommend that by March 31, 2004, the Secretary of Defense  develop a process for proposing, evaluating, and implementing human  capital, business, and partnership initiatives for the laboratories and test  centers, regardless whether by the pilot authority or by some other  vehicle. Such a process should include  instructions for proposal requirements such as linking to overall goals and  measurable objectives and the need for a business case, and    specification of procedures for proposal submission and review and  providing feedback on proposal quality and scope.
Finally, we recommend that the Secretary of Defense designate a strong  focal point to  receive, evaluate, and prioritize all proposals and    work with laboratory and test center directors, legal counsel, personnel  and other specialists to develop sound and well-developed business cases  and strategies to obtain needed changes.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, DOD states that it does not  concur with our recommendations because it has already taken actions  that in effect implement them. While the actions DOD cites that it has  taken are important to implementing our recommendations, they are not  sufficiently specific to address the problems identified in our report.  DOD’s written comments are contained in appendix II.
Regarding our first recommendation—that DOD inform the Congress of its  human capital and business objectives for the laboratories and test centers  and the strategies it will employ to meet them—DOD did not concur. DOD  discusses various high-level, agencywide initiatives it has taken to address  human capital and business issues in general and stated that the Congress  has been made aware of these initiatives, obviating the need for additional  reporting. We continue to believe that additional reporting is necessary.  We recognize that the general initiatives DOD discusses may provide ways  of helping the laboratories and test centers; however, to be effective, they  must be made specific, that is, developed into targeted strategies and plans  that address the particular problems the laboratories and test centers face.  DOD has not provided the Congress sufficient details on how the general  initiatives will be used to address laboratories’ and test centers’ objectives  and problems.
Regarding our second recommendation—that DOD develop a process for  proposing, evaluating, and implementing human capital and business-like  practices initiatives for the laboratories and test centers—DOD did not  concur. DOD states that it has already introduced new agencywide  management processes—the Business Initiative Council and the  submission of the NSPS proposal to the Congress—to address human  capital and business issues in general. However, DOD has not detailed  how these general initiatives will apply to the laboratories and test centers  or address our process concerns. For example, while the Business  Initiative Council may have an effective process for proposing, evaluating,  and implementing laboratory and test center business-like practices  initiatives, DOD has not provided sufficient information for us to make  such a determination. We also recognize that NSPS may address some of  the human capital problems faced by the laboratories and test centers, but  this system is still under consideration by the Congress. Until it becomes  law, we believe it is premature to cite it as an effective management tool.
With regard to our third recommendation—that DOD designate a strong  focal point to work with the laboratories and test centers to develop,  evaluate, prioritize, and coordinate proposed initiatives—DOD did not  concur. DOD states that the recently created position of Undersecretary  for Laboratories and Basic Sciences has oversight responsibility for all  laboratory initiatives and that it is establishing a new Defense Test  Resources Management Center that will oversee the test centers. DOD  asserts that these two organizations will perform as focal points. However,  DOD has not detailed how these organizations will fulfill this role and  work with the laboratories and test centers to overcome the many barriers  noted in our report.
section_title: Scope and Methodology
paragraphs: During our review, we met with officials from the following organizations  in the Office of the Secretary of Defense: the Director, Defense Research  and Engineering; the Director, Operational Test and Evaluation; the  General Counsel, and the Deputy Undersecretary of Defense for Personnel  and Readiness. We also met with officials from the Army Research  Laboratory, Aberdeen Test Center, Army Medical Research and Materiel  Command, Naval Research Laboratory, Naval Undersea Warfare Center,  Air Force Research Laboratory, Air Force Research Laboratory’s Space  Vehicles Directorate, and 46th Test Wing. We also discussed pilot program  issues with each participating laboratory or center.
To determine the initiatives proposed to date and their status, we obtained  records from OSD and service officials. From these records and from  discussions with each participant, we compiled a listing of initiatives  proposed by each participating laboratory and test center. We verified  the listing and the current status of each initiative with cognizant  service officials.
To determine what obstacles inhibited DOD’s implementation of the pilot  programs, we obtained documentation and data from pilot program  participants as well as from OSD officials. We also discussed statutory  obstacles with the officials from DOD’s Office of General Counsel and  Undersecretary of Defense for Personnel and Readiness. We discussed  management and procedural obstacles with officials from the Director,  Operational Test and Evaluation and Defense Research and Engineering.  In addition, we discussed all obstacles with the participating laboratories  and test centers.
The problems facing the laboratories and test centers have been  documented by many organizations, panels, and commissions. We did  not independently verify these problems or the findings and conclusions  of these entities. We conducted our review from July 2002 to April 2003  in accordance with generally accepted government auditing standards.
We are sending copies of this report to the Secretary of Defense; the  Secretaries of the Army, Navy, and Air Force; and interested congressional  committees. We will also make copies available to others upon request. In  addition, the report will be available at no charge on the GAO Web site at  http://www.gao.gov.
Major contributors to this report were Catherine Baltzell, Arthur Cobb,  Christopher Durbin, Rae Ann Sapp, Sylvia Schatz, and Katrina Taylor.  If you have any questions regarding this report, please call me at  (202) 512-4841.
section_title: Appendix I: Fiscal Year 1999 and 2000 Pilot Program Participants
section_title: Appendix II: Comments from the Department of Defense