Intelligence Community: Analysis of Alternatives Approach for a New Site Reflects Most Characteristics of a High-Quality Process
Published: 2017-07-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: NGA, a defense agency and element of the Intelligence Community, provides geospatial intelligence to military and intelligence operations to support national security priorities. It currently operates out of two primary facilities—its headquarters in Springfield, Virginia, and its NGA West campus in St. Louis, Missouri. In 2012, NGA determined that a new location for its NGA West facilities was necessary to meet security standards and better support its national security mission. NGA estimates that the construction of the new campus will cost about $945 million.
GAO was asked to evaluate the AOA process that NGA used to select the site for its new campus. This report (1) describes the process NGA used, including the key factors it considered and (2) evaluates the extent to which the AOA process met best practices for such analyses.
GAO visited the existing NGA West campus and the final four alternative sites that were considered, analyzed and assessed reports and information that document NGA's AOA process for selecting the site, and interviewed relevant officials about the process. GAO evaluated NGA's process against best practices identified by GAO as characteristics of a high-quality, reliable AOA process.
section_title: What GAO Found
paragraphs: In 2012, the National Geospatial-Intelligence Agency (NGA) began an analysis of alternatives (AOA) process to evaluate potential sites for its new NGA Campus West (NGA West) using key evaluation factors related to mission, security, development and sustainability, schedule, cost, and environment. NGA's process included levels of analysis and considerations to select the agency-preferred alternative from an original list of 186 potential sites, subsequently narrowed to the final four alternative sites (see figure). The process culminated in the June 2016 selection of the agency-preferred alternative, the St. Louis City site.
NGA's process for selecting a site for the new NGA West campus substantially met three of the four characteristics of a high-quality, reliable AOA process. Specifically, NGA's process substantially met the characteristics that demonstrate a well-documented, comprehensive, and unbiased AOA process. It partially met the credibility characteristic, in part because it did not fully include information on the risks and sensitivities to cost estimates. NGA officials stated that there was no comprehensive DOD guidance to inform its AOA process, and although NGA's AOA process is complete, NGA plans to develop full cost estimates as part of construction, planning, and design. In September 2016, GAO recommended that DOD develop guidance for the use of AOA best practices for certain types of military construction decisions. While DOD did not concur and the recommendation remains open, GAO continues to believe such guidance would help ensure that future AOA processes are reliable and would result in decisions that best meet mission needs.
section_title: What GAO Recommends
paragraphs: GAO is not making recommendations to NGA. In commenting on a draft of this report, NGA expressed concerns about GAO's assessment of NGA's estimates of cost risks and sensitivities. GAO continues to believe its assessment accurately reflects NGA's process.

section_title: Letter
paragraphs: The National Geospatial-Intelligence Agency (NGA), a defense agency  and member of the Intelligence Community, provides geospatial  intelligence to the U.S. military, intelligence operations and analysis, and  homeland defense in support of national security priorities, such as  counterterrorism and counter-proliferation. NGA conducts the majority of  its work in two locations—its Springfield, Virginia, headquarters, known as  NGA Campus East (NGA East), and NGA Campus West (NGA West)  facilities in St. Louis and Arnold, Missouri. In 2012, NGA determined that  it needed a new NGA West facility in order to better support its national  security mission and address physical facility and security challenges of  the current St. Louis campus, among other reasons. Preliminary  estimates put the project’s construction cost at roughly $945 million, and it  is designated a mega-project by the U.S. Army Corps of Engineers  (Corps of Engineers). According to NGA officials as of May 2017,  construction of NGA West is expected to begin in summer 2019.
As part of the 2005 Base Realignment and Closure process, NGA closed  some of its facilities in the Washington, D.C., metropolitan area and  consolidated to a new headquarters facility on Fort Belvoir in Springfield,  Virginia. NGA East was originally planned to cost $1.1 billion but  ultimately cost over $2.5 billion after new requirements were added that  were identified as essential to mission operability.
You asked us to evaluate the analysis of alternatives (AOA) process that  NGA used to select the site for its new NGA West. Our objectives were to  (1) describe the overall site selection and AOA process NGA used,  including the key factors it considered and (2) evaluate the extent to  which NGA’s AOA process met best practices for such analyses.
To describe the process and considerations that informed NGA’s site  selection for its new NGA West, we assessed documents and other  information and interviewed officials from NGA and other DOD  organizations, including the Corps of Engineers and the Office of the  Undersecretary of Defense for Acquisition, Technology and Logistics. In  addition, we visited the existing NGA West in St. Louis, Missouri in order  to observe the condition of the current facilities, and we visited the four  final alternative sites in Missouri and Illinois. We reviewed documentation  that described the condition of the current facilities, and we examined  economic and qualitative analysis conducted in 2010–2012 that informed  the decision to build a new NGA West on a new site.
We further reviewed NGA plans, processes, and internal and public  reports outlining the site selection and evaluation process to understand  the requirements, criteria, and other factors NGA considered in selecting  the site for the new NGA West, from 2012 through its final public decision  in June 2016. These documents include two site location studies  conducted during 2012 through 2014 that identified available sites and  minimum requirements; internal briefing and process documents  regarding each stage of the agency’s process to select its preferred  alternative, which outlined the evaluation criteria as well as how each  alternative was scored and evaluated against those criteria; and  preliminary master plans for each site. We reviewed NGA and Corps of  Engineers cost and schedule assessments; environmental analyses for  the project, including the publicly released draft and final environmental  impact statements and NGA’s record of decision; documents and  memoranda outlining inputs and decisions regarding the agency’s  preferred alternative; and relevant policy and guidance documents  including the 2015 NGA Strategy, and DOD instructions for economic  analysis and acquisition of real property.
To determine the extent to which NGA’s AOA process for the selection of  the site for its new NGA West aligns with best practices for such  analyses, we reviewed all data and documentation developed by or for  DOD, including NGA and the Corps of Engineers, as part of NGA’s site  selection and evaluation process (which we refer to as its “AOA  process”). We discussed these data and this documentation with officials  from NGA and the Corps of Engineers, and with other DOD officials. We  refer to this collection of data and documentation to support NGA’s AOA  process as NGA’s “AOA body of work.” After collecting available data,  documentation, and other information, we evaluated NGA’s AOA body of  work against our 22 AOA best practices.
To evaluate NGA’s AOA process against these best practices, we took  the following steps: (1) two GAO analysts independently examined the  AOA information received from NGA, providing a score for each of the 22  best practices; (2) a third GAO analyst adjudicated any differences  between the two analysts’ initial scoring; (3) a GAO specialist on AOA  best practices, independent of the audit team, reviewed the team’s  adjudicated AOA documentation and scores, cross-checking the scores  and all of the analyses for consistency. The resulting scores for  individual best practices were used to identify the averaged, overall score  for the four summary characteristics of a high-quality, reliable AOA  process—that it should be well-documented, comprehensive, unbiased,  and credible. Next, we shared our draft analysis with NGA, asking the  agency to provide technical comments and any additional documentation  or other information that might affect our assessment. We then  incorporated these additional comments to ensure that our analysis  included all available information. Finally, we applied the same  methodology and scoring process explained above to revise our initial  analysis based on NGA’s technical comments and any other additional  evidence we received, as appropriate.
While applying our AOA process best practices to NGA’s AOA body of  work, we assessed the reasonableness of the information we collected.  Examining NGA’s process for site selection against our AOA best  practices allowed us to assess the strengths and limitations of the  agency’s process. Our best practices were not used to determine whether  NGA had made the correct decision on the location for its new campus or  whether the department would have arrived at a different conclusion had  it more fully conformed to the best practices. Rather, we used our best  practices to assess the degree to which NGA can provide reasonable  assurance that its process met each of the four characteristics of a high- quality, reliable AOA process. Based on discussions with NGA and the  Corps of Engineers, we also determined that the NGA data we used to  understand NGA’s site selection process were sufficiently reliable for our  purposes of describing and understanding NGA’s process and rationale  for choosing the St. Louis City site as the new location.
We conducted this performance audit from September 2016 to July 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Decision to Build a New NGA West
paragraphs: The existing NGA West campus consists of 15 facilities on 27 acres.  Some of the buildings’ original construction dates back to the early 1800s,  and 22 acres of the site are on the National Register of Historic Places,  according to NGA documents. In 2009 through 2010, NGA contracted  with an independent firm to assess the condition of the existing NGA  West. This assessment gave the facilities an overall condition rating of  “poor,” generally because of the insufficiency of the anti-terrorism and  force protection measures, the average age of the structures, numerous  code and accessibility shortfalls, and lack of seismic protection.
Near the end of the completion of the NGA East headquarters  consolidation in 2011, NGA focused its attention on the need to improve  the operational capacity, security requirements, and modernization of its  NGA West facilities. From approximately 2009 through 2012, NGA  conducted a series of evaluations to inform its efforts to modernize NGA  West. These analyses included a condition assessment of the existing  facilities; an economic analysis of alternatives to evaluate the options of  building a new facility (“build new”), fully renovating the existing facilities  (“modernize”), or remaining in the current facilities with minimum essential  repairs (“status quo”); and a qualitative analysis of non-cost  considerations for the build new, modernize, and status quo options  identified in the economic analysis.
In 2012 NGA determined that a new NGA West would best meet the  agency’s mission and resource needs. After examining the options of  renovating its current facility, leasing, or building a new government- owned facility, NGA determined that building a new, government-owned  facility was the preferred option. NGA officials stated that they are in the  process of soliciting design-build proposals and that the final design-build  contract is planned for award near the end of fiscal year 2018.  Construction is expected to begin approximately in the summer of 2019.
section_title: Best Practices for an Analysis of Alternatives Process
paragraphs: We identified 22 best practices for an AOA process in October 2015,  based on government and private-sector guidance and input from  subject-matter experts. Many federal and industry guides have  described approaches to analyses of alternatives; however, there was no  single set of practices for the AOA process that was broadly recognized  by both government and private-sector entities. We developed these best  practices by (1) compiling and reviewing commonly mentioned AOA  policies and guidance used by different government and private-sector  entities and (2) incorporating experts’ comments on a draft set of  practices to develop a final set of practices. The 22 best practices are  grouped into four characteristics that describe a high-quality, reliable AOA  process and can be used to evaluate whether an AOA process meets the  characteristics of well-documented, comprehensive, unbiased, and  credible. These practices can be applied to AOA processes for a broad  range of capability areas, projects, and programs, including military  construction projects and decision-making processes, in which an  alternative must be selected from a set of possible options.
In September 2016, we recommended that DOD develop guidance that  requires the use of AOA best practices, including those practices we  identified, when conducting AOA processes for certain types of military  construction decisions. DOD did not concur with this recommendation  and disagreed that these best practices apply to military construction  decision-making processes. We continue to believe that this  recommendation is valid and that the principles demonstrated by the best  practices—and the practices themselves—draw from related DOD and  other practices.
Our best practices also parallel those found in DOD and Air Force  guidance on military construction and analysis for decision making. For  example, according to an Air Force instruction governing the planning and  programming of military construction projects, one of the required  planning actions is to evaluate alternative solutions. According to a DOD  directive pertaining to military construction, DOD must monitor the  execution of its military construction program to ensure—among other  things—that the program is accomplished in the most cost-effective  way. This guidance for cost effectiveness aligns with our AOA best  practice Develop Life-cycle Cost Estimates, which focuses on providing  decision makers with the information they need to assess the cost- effectiveness of alternatives. Further, DOD Instruction 7041.03, on  economic analysis for decision making, contains numerous cost  estimating principles and procedures that align with those called for in our  AOA best practices. As we reported in 2016, these policy documents  and instructions align with the general intent of our best practices, and  there are many similarities between our best practices and the  department’s guidance.
Additionally, in our previous work reviewing AOA process for other  national security facilities, agencies generally concurred with our  recommendations to consider including our best practices in future  guidance. For example, in 2014 we assessed three National Nuclear  Security Administration construction projects and found each project’s  AOA partially met our best practices for conducting an AOA process. The  Department of Energy agreed with our recommendation and has begun  implementation.
section_title: NGA’s Site Selection Process Began in 2012, and Potential Sites Were Evaluated Based on Mission, Security, Environment, Cost, and Schedule Considerations
subsections: section_title: Site Selection Process Began in 2012 and Concluded with the June 2016 Record of Decision
paragraphs: NGA launched its search for a new NGA West site in 2012 with a site  location study conducted by an outside real estate firm, and it concluded  the search with the issuance of a record of decision in June 2016. The  site location study included a check for existing federal sites that could  accommodate NGA West’s workforce and mission. This search resulted  in a total of 186 sites being identified initially as possible options; the list  was narrowed to 6 sites in the St. Louis metropolitan area for further  study. During preliminary master planning, 4 of the 6 sites identified by  the site location studies were determined to be suitable for further  analysis to select the agency’s preferred alternative. Three of these  sites are in the Missouri cities of Fenton, Mehlville, and St. Louis, and one  is in St. Clair County, Illinois, near Scott Air Force Base. See figure 1 for  the geographic distribution of the 4 sites.
The subsequent site selection process included an environmental impact  statement as required by the National Environmental Policy Act of 1969,  analysis of NGA and the Corps of Engineers’ compliance with related  DOD policies and other federal laws and requirements, preliminary  master planning conducted by the Corps of Engineers, and a site  evaluation process initiated by the NGA West Program Management  Office (PMO), which is responsible for managing the NGA West project.  To select the final site from the four alternatives, NGA initiated a site  evaluation process in August 2015 that was led by the NGA West PMO.  This process involved various teams of experts analyzing the sites and  evaluating them against defined criteria to identify the advantages and  disadvantages of each site.
Figure 2 provides an overview of the key elements and milestones of  NGA’s site selection process, beginning with its earlier decision to build  and concluding with its 2016 selection of the new site and issuance of its  record of decision.
According to NGA officials, there was no NGA or DOD policy or set of  practices to comprehensively guide NGA’s site selection and AOA  process. As a result, NGA relied on various DOD policies and  instructions, other federal guidance, and industry standards. It  incorporated these practices into the site selection process to ensure that  it complied with federal requirements and industry practice to develop its  AOA process, according to NGA and Corps of Engineers officials.  Additionally, NGA officials stated that our AOA best practices would have  been helpful in planning the site selection process for NGA West, but the  process began in 2012, and our 22 best practices were not published until  October 2015.
section_title: Potential Sites Were Assessed Based on Key Factors, and NGA and the Corps of Engineers Conducted Additional Analysis in Order to Reach a Final Determination
subsections: section_title: Potential Sites Were Assessed Based on Mission, Security, Environment, Cost, and Schedule Factors
paragraphs: At the outset of the site evaluation process in August 2015, the PMO set  forth broad sets of criteria to use in analyzing the four alternatives. These  broad sets of criteria, referred to as “evaluation factors,” were mission,  security, development and sustainability, schedule, cost, and  environment. In addition, each site was assessed to ensure that it  complied with key laws, regulations, and directives. The PMO divided the  analysis of the evaluation factors among NGA and Corps of Engineers  teams.
The mission, security, and development and sustainability factors were  assigned to two NGA evaluation teams of subject-matter experts—the  “mission evaluation team” and “security, infrastructure and schedule  evaluation team” (referred to here as security evaluation team). Each of  these teams used its expertise to develop “sub-factors” to assess the  advantages and disadvantages of each site. For example:
The mission evaluation team developed 10 mission-related sub- factors based on the PMO guidance, NGA’s mission, and the strategic  goals outlined in the 2015 NGA Strategy. The mission-related sub- factors focused largely on elements pertaining to NGA’s workforce  and partnerships, such as the sites’ proximity to the existing  workforce, their distance from NGA’s Arnold facility, and the likelihood  that the sites would attract mission partners to create a “GEOINT  Valley.”
The security evaluation team developed 13 sub-factors related to  security and infrastructure based on PMO guidance, DOD and other  federal security and energy requirements, threat analysis, and other  subject-matter expertise. Examples of the sub-factors include a 500- foot setback, perimeter security elements, sustainable characteristics,  and infrastructure resilience.
Separate evaluations of cost, schedule, and environmental considerations  were conducted by the Corps of Engineers in its role as construction  agent as part of the environmental impact analysis. In addition, NGA and  the Corps of Engineers conducted an assessment of relevant laws and  regulations.
The PMO integrated these analyses and provided an additional layer of  review to each of the evaluation factors, in some cases adjusting them.
For instance, the PMO reorganized the 10 mission-related sub-factors for  its review. Specifically, while the mission evaluation team focused the  sub-factors largely on NGA’s strategic goals related to workforce and  partnerships, the PMO’s analysis reorganized those same mission-related  sub-factors by how they supported all four of the 2015 strategic goals.  The PMO listed under three “strategic effects”—“Create GEOINT Valley,”  “Enhance Operations,” and “Attract and Sustain the Workforce”—all of the  sub-factors related to that strategic effect. The PMO re-analyzed the sites  by weighting those strategic effects and sub-factors that were linked to  multiple strategic goals higher than those that were linked to fewer such  goals. The PMO also adjusted some of the sub-factors used in the  evaluation for security and for development and sustainability.
The PMO’s additional analysis did not change the overall outcome of the  evaluation of the sites; rather, it validated the mission evaluation team’s  conclusion and generally supported all but one of the overall findings of  the other analyses. At the conclusion of the PMO’s analysis in  December 2015, the PMO’s conclusion was that no one site had emerged  as a clear preferred alternative.
section_title: NGA Director Requested Additional Analysis in Order to Reach a Decision
paragraphs: Because the master planning and site evaluation process concluded that  all four sites—Fenton, Mehlville, St. Louis City, and St. Clair—could meet  the overall requirements and that no single site held substantial  advantage over another, the NGA Director requested additional analysis  with refined criteria to more clearly differentiate among the final four sites.  Consequently, in January 2016 NGA initiated a new site selection team— consisting of NGA and Corps of Engineers personnel who had previously  been involved in various stages of the process—to reassess the sites  against refined criteria and perspectives in order to determine the agency- preferred alternative.
The site selection team carried forward five of the six original evaluation  criteria from the start of the site evaluation process, as well as compliance  with federal law, policy, and other regulations, to develop its six “refined  criteria.” In reviewing these refined criteria, the site selection team  determined that cost and schedule accounted for the greatest differences  among the sites. The team therefore used the cost and schedule  assessments completed as part of the PMO process to narrow the sites,  concluding that because the Mehlville and Fenton sites were the most  expensive and posed the greatest schedule risk they should be  eliminated from final consideration.
The site selection team then focused its analysis on the final two sites— St. Clair and St. Louis City—to inform the Director’s selection. The team  used the following six refined criteria to evaluate the sites: (1) cost, (2)  schedule, (3) security, (4) mission efficiency and expansion, (5)  applicability of and compliance with federal policies, executive orders, and  federal initiatives; and (6) environmental considerations. The team  proposed narrowing the relevant sub-criteria to those that provided the  greatest differentiation among the sites, according to officials on the team.  For example, the security criterion was narrowed to include 3 of the  original 13 security and infrastructure evaluation sub-factors, and the  adjusted “mission efficiency and expansion” criterion included one of the  mission evaluation team’s 10 original mission sub-factors.
Subsequently, the NGA Director provided additional direction, including  adding a review of potential support from Scott Air Force Base, based on  the support NGA East receives from being located at Ft. Belvoir, as well  as ensuring that the security-related sub-factors carried over from prior  analyses were consistently defined. Additionally, the director added 2  sub-criteria to the mission-related criterion to ensure that the site  evaluation continued in terms of NGA’s strategic goals of partnership and  people:  1.  “Team GEOINT,” which refers to NGA’s current and future  partnerships with academic, public, and private sector partners, and  which parallels the “GEOINT Valley” element evaluated by the  mission evaluation team and PMO.  2.  “Team NGA,” which refers to the potential effects of workforce  recruitment and retention that were also analyzed in the mission  evaluation team and PMO analyses.
According to NGA officials, while certain sub-factors or criteria were  adjusted to provide further layers of analysis, the most important factors  were always seen as mission and security. Additionally, NGA and Corps  of Engineers officials said that adding these two sub-criteria expanded the  analysis of the mission-related criteria to resemble the scope of the  PMO’s analysis and incorporated the NGA Director’s mission and vision  perspective.
Finally, the NGA Director determined the weighting of the final criteria to  evaluate the last two sites, the site selection team provided input on  which of the sites was more advantageous with respect to each criterion,  and in March 2016 this information was used to inform the NGA Director’s  selection of the agency-preferred alternative. The weighting and final  decisions are shown in table 1.
The NGA Director selected the St. Louis City site as the agency-preferred  alternative. It was identified in the publication of the final environmental  impact statement and finalized with the issuance of the record of decision  in June 2016.
section_title: NGA’s AOA Process for Selecting the New NGA West Site Substantially Met Three and Partially Met One Characteristic of a High-Quality, Reliable AOA Process but Lacked Important Cost Information
subsections: section_title: Site Selection Process Substantially Met Three of the Four Characteristics for a High-Quality, Reliable AOA Process but Could Have Been Strengthened
paragraphs: We compared NGA’s AOA process for selecting a site for the new NGA  West campus to our AOA best practices and determined that NGA’s  process substantially met three and partially met one characteristic of a  high-quality, reliable AOA process. Although NGA’s AOA process  substantially met most of the characteristics, we did find areas where the  process could have been strengthened if NGA had more fully  incorporated the AOA best practices. See table 2 for a summary of our  assessment and appendix I for additional details on our scoring of NGA’s  alignment with each of the 22 best practices.
NGA’s AOA process for selecting a site for the new NGA West  substantially met the well-documented characteristic of a high-quality,  reliable AOA process, although we did find areas for improvement.
For example, NGA’s AOA body of work demonstrated that the  assumptions and constraints for each alternative for the site selection  process were documented. NGA West’s Prospective Sites Master Plan  included a set of overall assumptions that guided the preliminary planning  process and provided specific assumptions and constraints for each  alternative. Specifically, the plan identified various assumptions and  constraints for the four final sites, such as calculations of the site  boundaries, the estimated number of parking spaces, the square footage  of the buildings and estimates of the building’s height, site utilities, and  environmental constraints, among other things. In one instance, the plan  documented the assumption that if the Mehlville site were to be used, all  utilities would need to be removed from within the property line and  existing buildings, parking lots, and roads would have to be demolished.
In another example, the Corps of Engineers conducted a schedule and  negotiation risk assessment and recorded scores for each site and some  mitigation strategies for specific issues. The assessment documented  risks to meeting the site acquisition schedule with the St. Louis site  because, among other reasons, the site needed environmental cleanup  that was expected to take six months. The Fenton site had high  negotiation risks, in part because the asking price of the site was  significantly higher than the appraised value. However, NGA did not  provide information on other risks, such as technical feasibility and  resource risks, and did not rank the risks or provide over-arching  mitigation strategies for each alternative. According to the best practice,  not documenting the risks and related mitigation strategies for each  alternative prevents decision makers from performing a meaningful trade- off analysis, which is necessary to select an alternative to be  recommended.
subsections: section_title: Comprehensive Characteristic: Substantially Met
paragraphs: NGA’s AOA process for selecting a site for the new NGA West  substantially met the comprehensiveness characteristic of a high-quality  AOA process, but although it had strengths, we identified some  limitations.
NGA’s AOA process considered a diverse range of alternatives to meet  the mission need and conducted market surveillance and market  research to develop as many alternative solutions as possible. According  to our best practices, an AOA process that encompasses numerous and  diverse alternatives ensures that the study provides a broad view of the  issue and guards against biases to the AOA process. Specifically, NGA’s  AOA process included a site location study that provided a summary of  the thorough analysis that NGA conducted to identify potential site  locations for the new NGA West campus. The study relied on information  from local real estate market databases and input from the local real  estate community, multiple municipal officials and organizations, and the  public to identify an original set of 186 possible sites and narrow that list  to a final 6 for further analysis.
However, although the NGA body of work provides evidence that the  Corps of Engineers developed initial cost estimates that compared each  alternative using different cost categories, NGA’s AOA process did not  include life-cycle cost estimates for the final 4 sites. NGA officials chose  not to analyze total construction and other facility sustainment costs,  because they assumed that since the sites were in the same geographic  area, construction and operating costs would be similar. However, the  estimates did not include sufficient details regarding all of the costs  examined—specifically, how the cost estimates were developed for  information technology trunk line costs. NGA stated that this best practice  had limited application to its AOA process because it had determined that  variation in the life-cycle cost estimates based on the location of the four  sites—all in the St. Louis metropolitan area—was negligible. NGA officials  also stated that the lack of final project design details constrained their  ability to develop full life-cycle cost estimates. However, without estimates  for full life-cycle costs, decision makers may not have a complete picture  of the costs for each alternative and may have difficulty comparing the  alternatives, because comparisons may not be based on accurate and  complete information. NGA and Corps of Engineers officials said that they  are in the process of developing full life-cycle cost estimates for the  construction and design of the new NGA West campus, for the agency- preferred alternative.
section_title: Unbiased Characteristic: Substantially Met
paragraphs: NGA’s AOA process for selecting a site for the new NGA West  substantially met the characteristic of an unbiased AOA process, although  we did identify some limitations.
NGA’s AOA body of work demonstrated that NGA had developed  functional requirements based on the mission need without a  predetermined solution and that the requirements were realistic,  organized, and clear. For example, NGA’s AOA body of work provided  facilities requirements and specifically listed 11 site location and campus  requirements that were tied to mission needs, including requiring a facility  that will support future changes to mission requirements and allow for  continuity of NGA operations. NGA’s AOA body of work also identified  physical requirements for the new NGA West campus, for example, that  the new facility must have at least 800,000 gross square feet and a 500- foot security buffer, and it must allow for a possible expansion in the  future.
However, although the NGA AOA body of work demonstrated a thorough  comparison of the alternatives throughout the site evaluation process, it  did not provide evidence that net present value was used to compare or  differentiate among the alternatives, nor did it provide a rationale for why  net present value could not be used. NGA officials acknowledged that  they did not compare the alternatives using net present value. They  stated that they had normalized some of the costs but that it was not  necessary to normalize all costs, because the estimates were all done  during the same time period. According to our best practice, if net present  value is not used to compare the alternatives, then the AOA team should  document the reason why and explain and describe the other methods  applied. Additionally, comparing items that have been discounted or  normalized with net present value allows for time series comparisons,  since alternatives may have different life cycles or different costs and  benefits.
section_title: Site Selection Process Only Partially Met the Characteristic for a Credible AOA Because It Lacked Important Information on Cost Risks and Sensitivities
paragraphs: NGA’s AOA process for selecting the site for the new NGA West campus  partially met the credible characteristic for an agency’s AOA process.  Although NGA’s AOA process had strengths, it also had limitations, such  as lacking important information related to cost risks and sensitivity  analyses for both cost and benefits identified.
NGA’s AOA body of work described the alternatives in sufficient detail to  allow for robust analysis. Specifically, it provided descriptions of each of  the alternatives at varying levels of detail. For example, the first site  location study provided descriptions of the top 6 potential sites, including  information on size, the sites’ strengths and weaknesses, and any  acquisition or development issues. The NGA AOA body of work also  provided evidence that site master planning was conducted to provide  additional details on the physical and environmental attributes of each  site, as well as constraints and benefits. For example, the NGA West  Prospective Sites Master Plan described the Mehlville site as having  landscape features such as mature trees, waterways, areas of steep  topography, options for public transportation, bike-friendly streets, and  existing utility infrastructure.
However, NGA did not fully include key information on either the risk or  the uncertainty related to cost estimates or the sensitivity to the costs and  benefits identified as part of its AOA process. For example, the NGA body  of work did not include a confidence interval or range for the cost  estimates for each viable alternative in order to document the level of risk  associated with the estimate. NGA’s AOA body of work documented the  estimated alternatives’ initial costs and included contingency costs across  all four alternatives. Corps of Engineers officials told us that they had  developed a 30 percent design and 5 percent construction contingency  cost factor across the four alternatives to account for cost risks in those  areas. However, the NGA AOA body of work did not provide evidence of  a confidence interval or range for the costs provided. NGA acknowledged  that while its AOA body of work did not identify the risk associated with  specific cost elements for each alternative, it did provide a “level of  confidence,” because the methodology behind the cost components in the  estimate implied a high level of confidence. Although we agree that NGA  did provide a contingency factor for the site development costs and  provided cost estimates for all four viable alternatives, NGA did not  develop a confidence interval or risk range for those estimates. NGA’s  cost estimates were used as a determining factor in the final decision  among the four alternatives. However, without understanding the cost risk  and the uncertainty of those costs as outlined in the best practice, a  decision maker might be unable to make an informed decision.
Additionally, the NGA AOA body of work did not demonstrate that NGA  had conducted a sensitivity analysis for the cost and benefit and  effectiveness estimates for each alternative in order to examine how  changes in key assumptions would affect the cost and benefit estimates.  The NGA AOA body of work documented that some sensitivity analysis or  level of risk was analyzed as part of the schedule analysis, and NGA  officials stated that the project considered how different values and  variables affect each other during the criteria and evaluation analysis.  However, the NGA AOA body of work did not document the sensitivity of  cost and benefit estimates to changes in key assumptions for each  alternative, and a sensitivity analysis was not applied to the initial cost  estimates or benefit assumptions that were used to make the final site  selection.
NGA officials stated that this best practice has limited application to its  AOA process, because the lack of variables between sites constrained  their ability to develop full life-cycle cost estimates and complete a  sensitivity analysis. NGA officials stated that their sensitivity analysis was  limited to those considerations that were measurable and sensitive to  change—predominantly schedule risk associated with land acquisition  activities. Further, NGA officials explained that because all the site  alternatives were located within the St. Louis metropolitan area, any  variations in conditions would have equal effect. Although we agree that  NGA did conduct a sensitivity analysis for schedule risks, NGA neither  documented how the schedule sensitivity affected its cost or benefit  estimates nor performed a sensitivity analysis for the various assumptions  used to develop the cost or benefit for each alternative. According to the  DOD instruction on economic analysis, a sensitivity analysis is a “what-if”  exercise that should be performed to test the conclusions and  assumptions of changes in cost and benefit variables and should always  be performed when the results of the economic analysis do not clearly  favor any one alternative. According to our best practice, not  conducting a sensitivity analysis to identify the uncertainties associated  with different assumptions increases the chances that an AOA team will  recommend an alternative without understanding the full effects of costs,  which could lead to cost and schedule overruns.
Although NGA’s AOA process did not reflect all of the characteristics of a  high-quality process, we are not making recommendations in this report,  in part because NGA plans to conduct additional cost analysis and in part  because we made an applicable recommendation to DOD in 2016.  Specifically, although NGA’s AOA process is complete, NGA and Corps  of Engineers officials said that they are developing full life-cycle cost  estimates for the construction and design of the new NGA West campus  and that these estimates will include many elements from our best  practices. Further, we continue to believe that our September 2016  recommendation that DOD develop guidance requiring the use of AOA  best practices for certain military construction decisions could help ensure  that future AOA processes conducted by DOD agencies like NGA are  reliable and that agencies identify a preferred alternative that best meets  mission needs. While DOD did not concur with our recommendation, as  we reported in 2016, our best practices are based on longstanding,  fundamental tenets of sound decision making and economic analysis.  Additionally, our best practices align with many DOD and military policies,  directives, and other guidance pertaining to military construction. Further,  during this review NGA officials stated that DOD did not have a set of  best practices for conducting an AOA to help NGA make decisions  regarding its military construction project, and that our AOA best practices  would have been helpful had they been published prior to the start of  NGA’s site selection process in 2012. Accordingly, we continue to believe  our prior recommendation is relevant and that unless DOD has guidance  directing that certain military construction AOA processes be conducted in  accordance with identified best practices, it may not be providing  Congress with complete information to inform its oversight of DOD’s  future military construction decisions.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report of this report to NGA for review and  comment. NGA’s comments are reprinted in their entirety in appendix II.
In comments on our report, NGA stated that it valued our assessment of  its AOA process, which we judged to have substantially met the  characteristics of a well-documented, comprehensive, and unbiased  process, and would use our findings to continue to refine and improve its  corporate decision making and processes.
NGA raised a concern about our assessment that its AOA process used  to select the site for its new NGA West project partially met the best  practices that demonstrate a credible process. NGA’s specific concern  was that we concluded that the AOA process did not fully include  information on risks and sensitivities to cost estimates. In its letter, NGA  stated that its analysis demonstrated that cost was a factor but not the  most important factor. Moreover, NGA stated that cost elements and  details ranged from well-defined costs, such as real estate costs, to  estimates based on analogy such as an information technology trunk line.  NGA additionally stated that, due to the conceptual nature of the design  of the facility at that time, more detailed cost analysis was judged to  provide no discrimination among alternatives and were thus purposely  excluded from the initial cost estimates that were used in the AOA  process. While NGA may have concluded that the project’s cost was not  the most important factor, the agency estimates that construction of the  campus will cost about $945 million and NGA used the cost estimate as a  determining factor to select from the four final alternatives.  Moreover, our  assessment of the credibility characteristic is based only in part on NGA’s  initial cost estimates and did not penalize NGA for excluding additional  cost estimates. Rather, we assessed that NGA’s AOA body of work did  not provide evidence of documenting the sensitivity of the cost-benefit or  effectiveness estimates to changes in key assumptions for alternatives,  nor was a sensitivity or risk analysis applied to the initial cost estimates  used to make the final site selection.
NGA also stated in its letter that our AOA best practices are not  applicable in all circumstances, and pointed out that DOD did not concur  with a recommendation in a prior report to develop AOA guidance  requiring departmental components to use AOA practices, including the  best practices we identified, for certain future military construction  projects. Our prior report suggested that such guidance might only apply  to certain military construction projects as determined by DOD. In  addition, while DOD’s existing relevant guidance does not require use of  our AOA best practices, the guidance does not prohibit it either. Further,  as discussed in our report, NGA officials told us the AOA best practices  are helpful to such processes, and lacking such DOD guidance NGA had  to draw on expertise, practices, and procedures from a variety of sources  to conduct its AOA for the new NGA West site.
Finally, in its letter NGA proposed that two documents—the  environmental impact statement and record of decision—fulfill the best  practice to document the AOA process in a single document. Specifically,  NGA stated that within the context of the environmental impact statement  process, the record of decision is the authoritative capstone document of  the process, and that together the two documents include discussions of  the decision-making and factors considered by the director in selecting  the agency-preferred alternative. These two documents were prepared to  fulfill requirements of the National Environmental Policy Act of 1969 in  order to determine the environmental impacts of the project, as discussed  earlier in our report. While we recognize that the record of decision and  the environmental impact statement are significant documents that  include summaries of aspects of NGA’s AOA process, as NGA indicated  these are two documents within an expansive AOA body of work. Further,  many of the elements of NGA’s AOA process are diffused throughout  these and several other reports and documents—that were specifically  identified by NGA as the key documentation of its AOA process—rather  than clearly delineated in a single document as prescribed by the best  practice (see appendix I).
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the  appropriate congressional committees; the Secretary of Defense; the  Secretary of the Air Force; the Secretary of the Army; the Under  Secretary of Defense for Acquisitions, Technology and Logistics; the  Under Secretary of Defense for Intelligence; and the Director, National  Geospatial-Intelligence Agency. In addition, the report is available at no  charge on the GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-4523 or leporeb@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix III.
section_title: Appendix I: Analysis of Alternatives (AOA) 22 Best Practices with GAO’s Evaluation of National Geospatial-Intelligence Agency (NGA) AOA Process and Scores
paragraphs: In our earlier discussion of the extent to which NGA’s AOA process met  best practices for such processes, we presented our analysis for specific  best practices. These 22 best practices and their definitions were  originally published and are listed in GAO-16-22. Table 3 summarizes  our analysis of NGA’s AOA process for selecting the site for the new NGA  West and our ratings of that process against all 22 best practices.
section_title: Appendix II: Comments from the National Geospatial-Intelligence Agency
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Brian Mazanec, Assistant  Director; Jim Ashley; Tracy Barnes; Chris Businsky; George Depaoli;  Richard Johnson; Joanne Landesman; Jennifer Leotta; Jamilah Moon;  Joseph Thompson; and Sally Williamson made key contributions to this  report.
section_title: Related GAO Products
paragraphs: Amphibious Combat Vehicle Acquisition: Cost Estimate Meets Best  Practices, but Concurrency between Testing and Production Increases  Risk. GAO-17-402. Washington, D.C.: Apr. 18, 2017.
Joint Intelligence Analysis Complex: DOD Needs to Fully Incorporate  Best Practices into Future Cost Estimates. GAO-17-29. Washington,  D.C.: Nov. 3, 2016.
Joint Intelligence Analysis Complex: DOD Partially Used Best Practices  for Analyzing Alternatives and Should Do So Fully for Future Military  Construction Decisions. GAO-16-853. Washington, D.C.: Sept. 30, 2016.
Patriot Modernization: Oversight Mechanism Needed to Track Progress  and Provide Accountability. GAO-16-488. Washington, D.C.: Aug. 25,  2016.
Amphibious Combat Vehicle: Some Acquisition Activities Demonstrate  Best Practices; Attainment of Amphibious Capability to Be Determined.  GAO-16-22. Washington, D.C.: Oct. 28, 2015.
DOE and NNSA Project Management: Analysis of Alternatives Could Be  Improved by Incorporating Best Practices. GAO-15-37. Washington, D.C.:  Dec. 11, 2014.
Military Bases: DOD Has Processes to Comply with Statutory  Requirements for Closing or Realigning Installations. GAO-13-645. June  27, 2013.
Military Bases: Opportunities Exist to Improve Future Base Realignment  and Closure Rounds. GAO-13-149. Washington, D.C.: Mar. 7, 2013.
Military Base Realignments and Closures: The National Geospatial- Intelligence Agency’s Technology Center Construction Project.  GAO-12-770R. Washington, D.C.: June 29, 2012.
Military Base Realignments and Closures: DOD Is Taking Steps to  Mitigate Challenges but It Is Not Fully Reporting Some Additional Costs.  GAO-10-725R. Washington, D.C.: June 24, 2010.
GAO Cost Estimating and Assessment Guide: Best Practices for  Developing and Managing Capital Program Costs. GAO-09-3SP.  Washington, D.C., Mar. 2, 2009.