Embassy Construction: State Needs to Better Measure Performance of Its New Approach
Published: 2017-03-16T00:00:00

Fast Facts:
Will State's new approach produce better embassies, and how will we know?
In 2011, the State Department began to shift from using a standard design for the construction of its embassies to customizing each embassy's design. This new approach (called Excellence) was meant to improve the appearance and functionality, as well as reduce the operating costs of these embassies.
However, we found that State does not have performance measures or sufficient data to evaluate these and other goals of Excellence. We recommended that State strengthen its performance measures, as well as its monitoring and reporting tools, for the Excellence approach.

Highlights:
section_title: Why GAO Did This Study
paragraphs: In 1998, terrorists bombed the U.S. embassies in Nairobi, Kenya, and Dar es Salaam, Tanzania, killing over 220 people and injuring 4,000. In 1999, State began a new embassy construction program, administered by OBO, which to date has received $21 billion, according to State. OBO's primary goal was to provide secure, safe, and functional workplaces, and it adopted SED with a streamlined, standard design for all embassies. In 2011, OBO replaced the SED with the Excellence approach, which makes use of customized designs for each embassy.
GAO was asked to review the implementation of Excellence. This report examines (1) the reasons for State's shift to the Excellence approach, (2) key elements and tradeoffs of the new approach, and (3) the extent to which State has established guidance and tools to implement and evaluate its Excellence approach. GAO analyzed information from State policy, planning, funding, and reporting documents and interviewed State and industry officials. GAO also surveyed OBO staff about, among other things, the sufficiency of OBO's policies, procedures, and technical guidance for the Excellence approach. GAO will examine project cost and schedule issues in a subsequent report.
section_title: What GAO Found
paragraphs: In 2011, the U.S. Department of State's (State) Bureau of Overseas Buildings Operations (OBO) established the Excellence approach in response to concerns regarding the aesthetics, quality, location, and functionality of embassies built using its Standard Embassy Design (SED). The SED utilized a standard prototypical design for new embassies and consulates along with a streamlined delivery method combining responsibility for design and construction under a single contract. Under the Excellence approach, OBO now directly contracts with design firms to develop customized embassy designs before contracting for construction. OBO officials believe that greater design control under Excellence will improve embassies' appearance in representing the United States, functionality, quality, and operating costs.
Excellence consists of several key elements and involves trade-offs. For example, OBO now allots time and funding to develop customized designs and hires leading design firms to produce them. These design firms have faced initial adjustment challenges designing U.S. embassies, and OBO only recently began evaluating their performance as required by federal regulation. OBO's new approach poses cost and schedule trade-offs since, for example, OBO now has greater design control but may also be responsible if design problems are identified during construction. GAO's survey found that OBO staff who responded held split or conflicting opinions on Excellence compared with SED.
While OBO has established guidance to implement Excellence, it lacks tools to fully evaluate the performance of this new approach. Performance measures are essential tools for managers to evaluate progress toward a program's goals, as noted in Standards for Internal Control in the Federal Government . However, OBO has not established performance measures to specifically evaluate and communicate the effectiveness of Excellence in delivering embassies. Moreover, OBO's bureau-wide strategic measures do not address Excellence priorities, such as greater adaptability to individual locations, functionality, or sustainability. OBO also lacks a reliable system to monitor operating performance, such as building energy usage, and a centralized database to broadly manage the Excellence program, to include effectively reporting on projects' design and construction costs and schedules. Without performance measures and reliable systems to collect and analyze relevant data, OBO cannot fully assess the value of shifting to the Excellence approach and away from the SED.
section_title: What GAO Recommends
paragraphs: GAO is making four recommendations that State take several steps to strengthen performance measures and reporting, monitoring mechanisms, and data systems for the Excellence approach. State concurred with all four recommendations.

section_title: Letter
paragraphs: The U.S. Department of State (State) maintains facilities at more than 285  diplomatic missions (posts) located around the world, including  embassies, consulates, and other diplomatic offices. More than 86,000  U.S. government employees are housed in these facilities. On August 7,  1998, terrorists bombed the U.S. embassies in Nairobi, Kenya, and Dar  es Salaam, Tanzania, killing over 220 people and injuring 4,000 others.  The ensuing Accountability Review Board found that unless State  addressed security vulnerabilities at U.S. embassies, U.S. government  employees would remain at risk from terrorist activity. In 1999, State  began its multiple-year Capital Security Construction Program,  administered by the Bureau of Overseas Buildings Operations (OBO),  which to date has received $21 billion, according to State. In  administering the program, OBO’s primary goal was providing overseas  U.S. diplomatic personnel with secure, safe, and functional workplaces.  OBO sought to achieve that goal by using a Standard Embassy Design  (SED) approach that relied heavily on the use and site-adaptation of a  standard design for new embassies and consulates (both referred to  hereafter as embassies) and a streamlined delivery method combining  responsibility for final design and construction under a single contract.  OBO also intended that new embassies would be efficient, state-of-the-art  facilities. In 2006, we reported that by using the SED and a streamlined  delivery method, OBO had made significant progress in completing new  embassies and, on average, reduced the time required to complete  projects.
In 2011, OBO replaced the SED approach with Design Excellence, now  referred to simply as Excellence, which makes use of customized,  individual designs for each embassy and emphasizes innovation. In April  2011, OBO announced the newly established Excellence approach to its  Industry Advisory Panel and, with it, a revised OBO mission. That new  mission is “to provide safe, secure, and functional facilities that represent  the U.S. government to the host nation and support our staff in the  achievement of U.S. foreign policy objectives. These facilities should  represent American values and the best in American architecture, design,  engineering, technology, sustainability, art, culture, and construction  execution.” Some Members of Congress and others have raised concerns  that this new approach may result in embassies that take longer and cost  more to build. From fiscal year 2011 through fiscal year 2015, OBO  allotted nearly $300 million for the project development and design of 16  Excellence projects and awarded over $900 million in construction  contracts for 6 of those Excellence projects.
You asked us to review the Excellence approach to the design and  construction of new embassies and consulates. This report examines (1)  the reasons for State’s shift to the Excellence approach, (2) key elements  and trade-offs of the new approach, and (3) the extent to which State has  established guidance and tools to implement and evaluate this approach.
To conduct this review, we obtained and analyzed information from  agency policy, planning, funding, and reporting documents, administrative  memos, and select project documentation. We will examine project cost  and schedule issues in a subsequent report. We also interviewed State  officials from OBO; the Bureau of Diplomatic Security (DS); the Office of  Management, Rightsizing, Policy, and Innovation; and the Office of  Acquisitions Management. Within OBO, we spoke with officials in offices  responsible for site acquisition, planning, project development, design  and engineering, cost management, construction management, facility  management, policy and program analysis, and financial management.
We also interviewed officials from a variety of architecture and  engineering design firms and construction contractors that have worked  for State. Additionally,  we spoke with industry groups such as the  American Institute of Architects (AIA),  the Associated General  Contractors of America (AGC), the Bridging Institute of America (BIA),  and the Design-Build Institute of America (DBIA). Also, because the  decision to adopt Excellence was made in 2011—and the work leading up  to the decision was undertaken in 2010—we interviewed key former OBO  officials with direct experience with OBO’s efforts to improve the Capital  Security Construction Program at that time, including some who served  on OBO’s management steering committee for Excellence.
We also conducted a web-based survey of OBO staff from July 15  through August 12, 2016, soliciting their views on the sufficiency of OBO’s  strategic vision, policies, procedures, and technical guidance for the  Excellence approach as well as any particular efficiencies or challenges  brought about by the approach. We sent the survey to 1,511 OBO staff,  705 (47 percent) of whom responded. We do not make any attempt to  extrapolate the findings to the remaining 53 percent of eligible employees  who chose not to complete our survey. The results of our survey provide  measures of employees’ views at the time they completed the survey in  July and August 2016. Over 550 respondents provided responses to at  least one open-ended question in our survey. We analyzed and  categorized these comments for specific questions and have included  selected quotes to characterize the results of that analysis. Respondents  generally provided more negative comments than positive ones; however,  where possible, we have tried to present a balanced selection of positive  and negative comments. In some cases we edited responses for clarity or  grammar. Views expressed in the survey may not be representative of all  OBO staff views on given topics. See appendix I for more detailed  information on our scope and methodology, and see appendix II for  survey results. In addition, see appendix III  for selected responses to  open-ended survey questions that touched upon issues we reviewed  through our separate audit work and that we compiled for illustrative  purposes.
We conducted this performance audit from August 2015 to March 2017 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Events Leading to Development of the SED
paragraphs: Following terrorist attacks against the U.S. embassy in Beirut, Lebanon,  in 1983, State began an embassy construction program—known as the  Inman program—to protect U.S. personnel. However, State completed  only 24 of the 57 planned construction projects, in part due to poor  planning, systemic weaknesses in program management, difficulties  acquiring sites, schedule delays, cost increases, and subsequent funding  limitations. Following the demise of the Inman program in the early 1990s,  State initiated very few new construction projects until after the two 1998  embassy bombings in Africa. Following those attacks, the Secure  Embassy Construction and Counterterrorism Act of 1999 required State  to maintain a list of diplomatic facilities to be scheduled for replacement  based on their vulnerability  to attack. In response, State initiated the  Capital Security Construction Program to construct new, secure facilities  overseas. At that time, State determined that diplomatic facilities at more  than 180 posts—more than half of U.S. overseas missions—needed to be  replaced to meet security standards. In 2016, State reported that from  2000 through 2014, it moved over 30,000 staff into more secure facilities.
The Secure Embassy Construction and Counterterrorism Act of 1999  calls for new diplomatic facilities to be sufficiently sized to ensure that all  U.S. government personnel at a post are located on a single secure site  and that those facilities are set back not less than 100 feet from the site’s  perimeter boundary. Before constructing a new embassy, State must  certify to Congress that, among other things, the facility incorporates  adequate measures for protecting classified information and activities as  well as personnel working in the facilities. OBO contracts with  architectural and engineering firms (design firms) to develop designs  meeting security and other project requirements. These design firms  submit their designs for reviews by OBO and DS to ensure conformance  with building code and security standards, respectively. DS, in  consultation with the Office of the Director of National Intelligence, must  certify that the design meets security standards prior to the start of  construction. While this certification occurs in the design phase of a  project, DS also has other roles in the process, such as participating in  site selection, ensuring OBO contractors have necessary security  clearances, and ensuring facilities are securely constructed.
section_title: The SED Approach
paragraphs: To address some of the performance problems experienced during the  Inman program, OBO implemented reforms to its business processes in  structuring the new Capital Security Construction Program. Among the  most prominent reforms were the development of the SED to expedite the  planning, contract award, design, and construction of new diplomatic  compounds and use of the design-build (DB) project delivery method,  which combines responsibility for design and construction under a single  contract and allows contractors to begin basic construction before the  design is fully completed.
Initially  there were three common SED classes—small, medium, and  large—based on the size of a post. For planning purposes, each size had  predefined schedules and costs associated with them. The SED itself  was a set of documents providing prototypical plans (for a medium SED),  specifications, and design criteria, and explaining how to adapt those to a  particular site and project. The SED was not a complete design but  rather a standardized template for the structural, spatial, and security  requirements of a new embassy compound to guide a contractor’s final  design. Compound elements described by the SED generally included the  main office building; U.S. Marine Security Guards’ living quarters; a  warehouse; a utility building; compound access control buildings and  perimeter walls; and parking facilities. The SED also allowed for the  standardization of building components such as security windows and  doors. Figure 1 shows the prototypical facilities defined by the SED.
The main office building within the SED was organized around two  parallel wings connected by a central lobby. Occasionally site conditions  such as size, shape, or topography required deviating from that typical  configuration; OBO refers to such projects as SED “derivatives.” From  2001 through 2015, OBO constructed more than 50 embassies using the  SED approach. Figure 2 shows examples of both the typical SED and  derivative SED office building configurations.
In 2006, we reported that OBO had made significant progress  constructing new diplomatic compounds using the SED approach. We  found that the average time to design and construct the 18 new embassies completed from 1999 to 2005 was about 3 years (36.7  months). This was nearly 3 years faster than embassies built during the  Inman era, even though the newer facilities were significantly larger and  more complex. We also found that reforms implemented by OBO,  including the switch to the SED and the DB contract delivery method, had  reduced project completion times, although it was difficult to quantify the  effects of any single reform. In 2007, OBO reported that the SED,  combined with DB project delivery, was expected to reduce overall  delivery time—from site selection to occupancy—by 34 percent. In  2008, State’s Inspector General found that OBO’s continued use of the  SED, in conjunction with the DB delivery method, was generally  effective. Additionally,  State’s Inspector General found that the SED  permitted faster certification of project designs and accreditation by DS  because the standardized design specifications were fully vetted for  conformity to security standards.
OBO took some actions to incorporate sustainability principles in the SED  to meet federal energy mandates to reduce energy and water  consumption. In 2006 OBO committed, in concert with 20 other federal  agencies, to seek common strategies for planning, acquiring, siting,  designing, building, operating, and maintaining federal facilities in an  energy efficient and environmentally sustainable manner. In 2008, OBO  established Leadership in Energy and Environmental Design (LEED)  certification as a design standard for its SED projects. In 2009, OBO  documentation indicates OBO elevated its sustainability requirement for  SED projects from LEED Certified to the higher certification level of LEED  Silver.
section_title: The Excellence Approach
paragraphs: In 2011, OBO announced a new project approach it termed Design  Excellence, intended to deliver embassies that (1) best represent the U.S.  government overseas, (2) are functional and secure, (3) incorporate  sustainable design and energy efficiency, (4) are cost-effective to operate  and maintain, (5) have greater proximity to host-government counterparts  and users via more centrally-located urban sites, and (6) better respond  to the unique needs and context of specific posts. OBO subsequently  phased out the SED as the basis for embassy designs, and according to  OBO officials, SED specifications, standards and guidance were  incorporated into OBO’s Design Standards and Design Guide. In 2013,  OBO renamed its approach “Excellence in Diplomatic Facilities”  (Excellence) to convey what OBO officials have said is a more holistic  effort to improve every aspect of OBO’s operations, including real estate  acquisition, security methods and technologies, cost management,  construction management, and facilities management.
Table 1 shows 23 new construction contracts awarded since Excellence  was approved and through fiscal year 2015 at a total value of $3.67  billion, according to State data. Of these, OBO reports 6 as being  Excellence projects; the other projects include projects with certain  Excellence features in terms of site, permit, or other requirements; SEDs;  and derivative SEDs. OBO officials assert that although Excellence was approved in 2011, OBO never planned to award any Excellence  construction contracts until fiscal year 2014. No Excellence projects had  been completed as of the end of fiscal year 2016 (September 30, 2016).
See appendix IV for a timeline illustrating the history of OBO’s building  program from 1998 through 2016.
section_title: OBO Established Excellence in an Effort to Improve Embassy Delivery Approach
paragraphs: Although combining the SED with a DB project delivery method enabled  OBO to accelerate the construction of new embassies, concerns raised  by various stakeholders about the aesthetics, quality, location, and  functionality of SED facilities prompted OBO to take some steps to  improve the SED concept and eventually transition to Excellence. These  steps included the introduction in 2008 of the design-build with bridging  method (bridging), whereby OBO first contracts with a design firm to  develop a project-specific, partial design that a construction contractor  and its design firm then completes. After a nearly yearlong review, in April  2011 OBO approved a series of recommendations and planned actions to  implement Excellence. A significant change announced at this time was  OBO’s increased use of design-bid-build (DBB) as another delivery  method alongside bridging. Generally under DBB, OBO first solicits and  contracts with a design firm to develop a 100-percent design, which is  then used to solicit bids from prospective construction contractors.
subsections: section_title: Concerns with SED and Desire to Improve Embassies Motivated OBO’s Shift to Other Approaches
subsections: section_title: OBO First Shifted to Design- Build with Bridging
paragraphs: During the SED era, OBO predominately used a DB project delivery  method. DB integrates design and construction responsibilities into a  single contract. Under this model, the DB contractor is responsible for  design and construction and thus bears the risks, such as added cost, for  any design problems because it (not OBO) hires the design firm to bring  the design to completion. According to industry experts, DB is generally  recognized as the best project delivery method for supporting accelerated  delivery, in part because the DB contractor may undertake some  construction while design is still in progress. Under OBO policy, in the  SED approach OBO provided the DB contractor with the SED prototypical  design—to include standard site and building plans, technical  specifications, design criteria, and instructions for its adaptation for a  particular project and contract requirements. The contractor’s design firm  would then use the SED documentation to develop a 100-percent design  adapted for a site at a particular post, becoming the architect-of-record.  According to the AIA,  in general, the architect-of-record for a project  prepares the bulk of the design and construction drawings and assumes  professional responsibility for the design. Although the DB contractor’s  design firm completed the project design, OBO’s policy was to hire its  own design firm beforehand to conduct project development activities  such as due-diligence planning surveys, site studies, and other analyses  needed to inform the project’s design. Figure 3 provides an overview of  the embassy construction process under OBO’s implementation of DB.
According to former senior OBO officials, the OBO Director who had  implemented the SED viewed OBO’s mission as needing to build secure  embassies as fast as possible and within a fixed budget, given the large  number of facilities that State needed to replace. They stated that the  then Director’s commitment was that OBO would combine a standardized  design with the DB delivery method to speed design and construction and  limit the costs to build each embassy to no more than $100 million (based  on a large SED). OBO also maintained at that time that the SED generally  would take no more than 36 months to build—inclusive of the time for  contract acquisition, design, and construction—depending on the post  size. According to these former senior OBO officials, those estimates  did not always reflect the budget and time needed to build some SED  embassies. They also stated that adapting the SED to the unique  requirements of some posts—such as a very large consular services  operation—was challenging and that the SED did not always account for  quality and long-term maintenance and operations cost considerations. In  addition, one former OBO official stated that although the emphasis in the  SED approach on speed and cost control enabled OBO to promote that it  had moved a certain number of people into secure facilities each year,  this was an indicator of performance related to a single goal: project  delivery. He noted that OBO did not use any performance indicators  related to design and construction quality to evaluate the new SED  facilities being built.
Although the SED approach enabled OBO to accelerate the construction  of new embassies intended to meet rigorous new security requirements,  some stakeholders raised concerns about the aesthetics, quality, location,  and functionality of those facilities.
Aesthetics. One design firm we spoke with said there were criticisms that  SED embassies were “cookie cutter” facilities that looked like fortresses.  In 2010, then U.S. Senator John Kerry and former Secretary of Defense  William Cohen—key advocates of Excellence—reported newly  constructed embassies were not sending the right message. They  described new embassies as cold concrete facilities at a forbidding  distance hidden away from city life, with little regard for the local  surroundings, undermining U.S. diplomats’ message and mission. They  asserted that State was constructing a standardized “embassy in a box,”  uniform in appearance, quickly assembled, and fortress-like.
Quality. According to some former senior OBO officials, OBO’s emphasis  on speed and cost under the SED approach resulted in some poor-quality  buildings. According to these officials and one design firm we interviewed,  the time and budget pressures sometimes resulted in OBO and its  contractors making trade-off decisions such as using less costly and  lower-quality building systems or materials. For example, one former  official reported that the SED approach resulted in some projects where  contractors used lesser-quality exterior stone or metal cladding on  building exteriors. In some projects, he indicated that contractors installed  heating, ventilation,  and air-conditioning systems that were minimally  acceptable under the SED but not the best solution for the post’s  geographic climate.  Location. OBO officials commented that in some cases the 10-acre lot  specified by the SED required siting the embassy too far from urban  centers where foreign government offices and other embassies are  located. This issue also arose in a 2007 report to State entitled The  Embassy of the Future. Guided by a commission composed of former  U.S. ambassadors, among others, the report recommended that State  avoid constructing embassies in locations remote from urban centers. It  also noted that although the appearance of embassies as influenced by  security requirements deserves careful consideration, their location is of  higher importance.  Functionality. One former OBO official stated that because the SED was  “very complete” as a standardized design concept and was based on a  completed embassy in Africa, its design was not always conducive to  being site adapted and applied to other regions in the world. For example,  the design criteria for the heating and cooling systems generally specified  by the SED may not always have been the best for climates that are very  hot, cold, or humid. Some design firms we spoke with echoed that  assessment, saying that project size, site shape or topography, regional  climate, or special post needs could render the SED difficult to apply.  OBO’s most recent former Director has also stated that the SED did not  always permit OBO to meet posts’ varied needs. Former OBO officials  told us that, in some cases, functional elements such as warehouses were eliminated from SED projects to deliver them on time and within  budget. Additionally,  current OBO officials emphasized a need to  acknowledge the frequency of which the scope of SED projects were  reduced to keep projects on time and within cost, without a corresponding  reduction to schedule and budget.
Issues with the SED approach have been documented in past OBO and  GAO studies. For example, in 2008, OBO initiated a “look back” study to  examine shortcomings with the early SED projects (2001-2007). The  study identified deficiencies in newly completed SEDs stemming from  building functionality issues, construction flaws, maintenance issues, and  de-scoped facilities. Our 2010 review also examined functionality at 22  new embassy compounds where construction began in or after fiscal year  1999 and was completed by September 30, 2009. Officials at 21 of the  22 posts reported that the design of some spaces within their facility did  not fully meet their functional needs, with an average of five functionality- related issues per post. We reported that in some cases, functionality  challenges resulted in the need to conduct costly follow-on projects after  posts occupied the embassy. OBO officials assert that by using the SED  approach, a deficiency in one project was effectively built into each active  project as it was a standardized approach to project delivery. By  comparison, OBO officials assert that by designing each project  individually,  the Excellence approach provides OBO with the ability to  more quickly identify and make changes or improvements from project to  project.
OBO took some steps to improve the exterior look of embassies prior to  adopting the Excellence approach. According to former senior OBO  officials we spoke with, OBO recognized there were some legitimate  criticisms about the aesthetics and architecture of the embassies built  under SED. Those officials indicated that one of OBO’s interim Directors  initiated a study to improve future embassy projects such that they better  fit in with the streets and spaces around an embassy. These officials  cited OBO’s 2011 Embassy Perimeter Improvement Concepts & Design  Guidelines as a direct effort to improve the exterior appearance of  embassies by using various design techniques and landscaping so that  they would look less “fortress-like.”
OBO had previously prepared a report, in 2008, that reviewed its  embassy construction process and the SED. To address some of the  problems found in this report—such as the need to balance SED  standardization with unique post conditions—OBO’s then-Director  approved the use of DB with bridging in 2008. Generally under this  delivery method, OBO first contracts with a design firm (the bridging  architect) to develop a project-specific, partial design package (bridging  design) that conveys State’s design vision and a higher level of detail for  key design requirements. Upon completing a project’s bridging design,  OBO’s procedure is to separately contract with a DB contractor to  complete the design and build the project. Therefore, unlike the SED,  each bridging design is project-specific, customized, and separately  contracted to a design firm. According to senior OBO officials, as well as  construction contractor and design firm officials, the current extent of  design represented by Excellence bridging documents varies by project  but generally approximates an overall 35- to 50-percent design. Those  officials indicated that bridging designs include multiple design disciplines  whereby elements such as architectural design may be developed to a far  greater extent than others, such as electrical design. Under this method,  the DB contractor and its design firm are responsible for completing the  design. Figure 4 provides an overview of the embassy construction  process under bridging.
In 2009, we reported that by providing more design detail up front, OBO  believed bridging would more effectively translate project requirements to  contractors, speed the design security certification process, and enable  construction to begin sooner. OBO documentation also indicates that  OBO believed bridging would better define the desired look and quality for  projects than the SED alone could achieve and provide less room for the  contractor to make interpretations and change OBO’s vision of the  project. However, according to OBO documentation, the effort, cost, and  time to produce a contract solicitation with bridging documents can be  significantly more than that required for a typical DB contract using the  SED.
section_title: OBO Later Shifted to Excellence
paragraphs: In 2009, the American Institute of Architects (AIA) submitted a report to  OBO entitled Design for Diplomacy, New Embassies for the 21st  Century. Informed by a task force composed of architects, engineers,  former ambassadors, staff from the U.S. General Services Administration  (GSA), OBO design professionals, and others, the AIA recommended that  OBO “adopt Design Excellence as a mandate to advance a new  generation of secure, high performance embassies and diplomatic  facilities that support the conduct of American diplomacy.” It outlined  several actions it viewed as necessary to realize the benefits of design  excellence. AIA  officials we spoke with said that AIA never expressly  advocated that the SED be completely abandoned, because that  approach might remain appropriate for some projects. However, these  officials noted that as more SED projects were built, AIA’s members (i.e.,  architects) believed that SEDs were not the optimal choice for most  projects, as the standardized design was not always conducive to  adapting to different climates, countries, or unique post functions.
In April 2010—a year before formally instituting Excellence—OBO  released “Guiding Principles of Design Excellence in Diplomatic Facilities”  (Guiding Principles) to its Industry Advisory Panel. See appendix V for a  summary of these principles. At that time, State also announced OBO’s  intent to create the Design Excellence approach, with the goal to produce  diplomatic facilities outstanding in all respects, including security,  architecture, construction, sustainability, operations and maintenance.  OBO’s “Guide to Excellence in Diplomatic Facilities” (Guide to  Excellence) released in July 2016, states that the new approach will result  in innovative,  new American landmarks around the globe.
OBO’s establishment of Excellence was informed by a nearly yearlong  review—begun in June 2010—by seven internal OBO working groups   overseen by a steering committee composed of OBO’s senior managers  and chaired by OBO’s then Deputy Director (who later served as OBO’s  Director from June 2012 through January 2017). OBO also sought  assistance from GSA, which assigned GSA’s Director of Design  Excellence to subsequently participate as an external advisor to the  Steering Committee. OBO’s working groups were tasked with examining  OBO policies and procedures and providing the steering committee with  recommendations as to how best to integrate design excellence into all of  OBO’s activities. This review resulted in over 60 recommendations and a  series of planned actions that were approved in an April 2011 decision  memo (Excellence decision memo) as a means to implement  Excellence. The review also identified some specific changes to OBO’s  processes in the areas of (1) site selection; (2) project delivery method;  (3) design standards and guidelines; (4) hiring of outside architectural and  engineering design firms; (5) design reviews; and (6) life-cycle cost  analysis, among other areas. See appendix VI for a table describing the  approaches OBO identified to achieve its goals under the Excellence  approach.
A significant change under Excellence since 2011 has been OBO’s shift  to an increased use of the DBB delivery method alongside bridging.  Generally under DBB, OBO first solicits and contracts with a design firm  to develop a 100-percent design. Under this method, OBO then uses  the completed design to solicit bids from prospective construction  contractors. According to OBO documentation, OBO selects a project’s  delivery method, either bridging or DBB, based on an evaluation of local  context, project complexity, construction factors, and urgency. According  to OBO officials, the timing of a construction award (i.e. the planned fiscal  year when OBO expects to receive funding to make an award) is also a  key determining factor regarding delivery method. Figure 5 provides an  overview of the embassy construction process under DBB.
section_title: OBO’s Greater Design Control Requires Greater Up- Front Resources and Has Cost and Schedule Trade-Offs
paragraphs: Changes made under Excellence provide OBO with greater design  control, but carry trade-offs. Key elements under the Excellence approach  include (1) allotting funding and time for developing custom designs; (2)  hiring leading design firms for projects and promoting innovation in  design; (3) conducting peer reviews of designs; and (4) using bridging  and DBB project delivery (rather than DB). We found that OBO now funds  the development of customized designs and provides up to 24 months for  front-end design work. OBO also seeks to hire leading U.S. design firms  to develop those designs for each project. New design firms OBO has  hired for Excellence projects have faced some challenges, and OBO only  recently began assessing their performance. OBO also requires design  reviews by industry advisors. This shift to more design-focused delivery  methods—from DB to bridging and DBB—has design, schedule, and cost  trade-offs. OBO’s staff had split opinions regarding the Excellence  approach compared to the SED approach.
subsections: section_title: OBO Now Allots Project Funding and up to 24 Additional Months to Develop Excellence Designs
paragraphs: OBO’s Excellence approach—using bridging and DBB—represents a new  investment to develop innovative, project-specific designs. Previously, the  SED approach combined with DB delivery made use of the same  standard design, which DB contractors’ design firms would adapt to a  specific site. Thus OBO did not contract with design firms to develop  customized designs. One senior OBO official said that OBO’s intent under  Excellence is to “own the quality of each project” and that contracting for  project-specific designs provides control over the design process to avoid  what OBO reports were quality issues with some SED projects. OBO  reports it has awarded 24 new embassy or consulate design contracts— for either 100-percent designs or partial bridging designs—during fiscal  years 2011 through 2015. The first design contract solicited as an  Excellence project, according to OBO, was awarded in January 2013 for  the new U.S. embassy to be built in Mexico City.
Design-related activities include both actual project design, which entails  the preparation of plans, drawings, and specifications; and project  development, which included due diligence efforts such as boundary, utility, and soil surveys. OBO officials stated that they could not distinctly  segregate the cost for project designs from project development costs  needed to complete a project’s design under any project delivery method,  including DB using the SED. Furthermore, according to OBO officials,  because such costs were funded out of a central pot of money during the  SED era, they cannot be broken out of those earlier contracts. Table 2  shows the amount of funding OBO has authorized for both project design  and development activities from fiscal year 2011 through 2015 at a total  value of over $400 million. OBO identified 16 of these 24 project designs  as being Excellence projects.
OBO’s Excellence approach—using bridging and DBB—also represents a  new investment in up-front design time, potentially  up to 2 additional  years (compared to the SED approach) to develop custom, innovative  designs before OBO contracts for construction. OBO maintains that this  additional design time is integrated into its projects such that planned  construction contract award dates are not affected and remain consistent  with OBO’s overall Capital Security Construction Program schedule for  constructing new secure facilities. In other words, by starting projects  earlier, OBO asserts that it can still meet the Capital Security  Construction Program schedule for delivering new embassies. However,  because the first set of Excellence construction projects were awarded in  fiscal year 2014 and are still in progress, it is currently unknown whether  OBO will deliver new facilities consistent with the overall program  schedule. OBO’s process under both bridging and DBB is now to award  contracts to design firms to produce custom project designs, and OBO  planning documentation generally estimates up to 2 years (24 months) for  the design process of a generic new embassy. OBO documentation  generally indicates that when there is sufficient time available for  conducting planning and design activities before awarding a construction  contract, OBO is inclined to utilize DBB, which OBO asserts often results  in a superior end product. OBO officials cautioned that its generic  timeframes are just a starting point, and that every project can encounter  unique challenges. Figure 6 shows OBO-generic timelines under the prior  SED approach (using DB)—which were established based on the size of  a post—in comparison with OBO’s generic schedules for Excellence  bridging and DBB projects, the latter two providing up to 24 months of  additional design work before construction begins.
section_title: Design Firms New to OBO Have Faced Adjustment Challenges; OBO Recently Began Assessing Their Performance
paragraphs: One of OBO’s Guiding Principles for Excellence is that OBO will hire  leading U.S. design firms based on their design achievements and  portfolio of work. OBO’s intent in hiring leading design firms is, in part, to  promote the innovation of American architecture, engineering, and design  disciplines as well as U.S. technology, manufacturing, and product  design. According to OBO, selection of these firms is based, in part, on  their achievements in the design field and work on projects similar in  scale and complexity to an embassy project. OBO’s guidance indicates  that material advances and new technologies can result in the delivery of  better diplomatic facilities and that OBO must invest in innovation. For  most of its design projects, OBO utilizes an Indefinite Delivery/Indefinite  Quantity (ID/IQ) contract mechanism—under which five design firms have  been hired—to task design firms to develop project designs, when  needed. In some instances, OBO prefers to issue project-specific  solicitations and contracts for more unique and challenging projects, such  as the London and Mexico City embassy designs.
Under the SED approach, design firms typically conducted due diligence  and project development (i.e., planning activities) for OBO to ensure  projects were ready for design and construction by the DB contractors.  Under Excellence, OBO contracts with design firms to develop Excellence  designs before awarding a construction contract, according to OBO. Four  of the five firms hired by OBO under its current ID/IQ are new to embassy  construction work. Officials we spoke with—including DS officials, design  firms, construction contractors, and former OBO officials—identified some  adjustment challenges facing design firms new to OBO that have never  designed an embassy before. For example, having to become familiar  with State’s unique security requirements for diplomatic facilities is a  challenge for those design firms. DS officials reported the firms require a great deal of work to bring them up to speed on State’s security  requirements. To mitigate this situation, DS has been conducting “101  Certification Workshops” for new design firms so that they understand  DS’s approach to the security certification of new embassy designs.  Those officials stated that conducting those workshops and reviewing  customized Excellence designs has increased DS’s workload.
Newer design firms have also been challenged by such issues as lack of  sufficient staff with required security clearances, information systems, or  office space to independently and securely perform the contracts,  according to DS and design firm officials. Some of the new-to-OBO  design firms further indicated that they have contracted—as partners or  subcontractors—design firms that have worked on past OBO projects to  assist the new-to-OBO firms in navigating State’s standards and process.
OBO did not begin conducting performance evaluations of its design firms  until recently. Recommendations from the 2011 Excellence decision  memo indicated that OBO would (1) measure performance of its  designers for Design Excellence and project performance, and (2) use the  federal contractor performance reporting system to promote consistency,  increase data integrity, and motivate contractor performance. The Federal  Acquisition Regulation requires agencies to conduct contractor  performance evaluations. Such evaluations are intended to provide  essential information regarding whether to award future contracts to these  design firms. We found that OBO had not been conducting contract  performance evaluations of the design firms contracted to deliver  Excellence designs. OBO officials acknowledged they had not been  recording performance evaluations. As a result of our inquiry and a  subsequent request from State’s contracting office, OBO has trained staff  and as of August 2016 had initiated design firm evaluations for six of its  Excellence projects.
section_title: Excellence Approach Includes Design Reviews by Industry Advisors
paragraphs: The Excellence approach entails greater involvement by OBO’s industry  advisors. In 2012, OBO made changes to its industry advisory body,  previously called the Industry Advisory Panel. It is now called the  Industry Advisory Group (IAG)  and OBO increased the number of  members from up to 9 advisors representing industry organizations— such as AIA, Associated General Contractors of America (AGC), and  Design-Build Institute of America (DBIA)—to up to 35 members, who  must, among other criteria, be members of professional organizations and  trade groups involved in property management issues, but who represent  the companies employing them (not those organizations or industry  groups). OBO officials indicate the change was made to allow OBO to  have broader industry representation.
In September 2014, OBO established a new policy mandating OBO  senior management and industry peer design reviews, called OBO senior  management and IAG  design reviews. The policy requires that two such  design reviews be conducted for each new project and that OBO’s  Director will designate three members from its IAG or other adjunct  professionals to serve as reviewers. These reviews are intended to assist  OBO in making certain that projects are well-conceived and can be  realized in an efficient and cost-effective manner. The first design  review occurs during the Concept Design Phase. The design firm  awarded the design contract must submit three viable concepts that it  assesses as achievable within the project budget, according to OBO  policy. The design firm must explain the factors that influenced each of  the three proposed designs, including any opportunities and constraints.  During the concept design review, OBO senior management and IAG  panel members may raise and discuss any concerns about the proposed  concepts or issues affecting scope, schedule, or cost. After considering  the IAG  panel’s recommendations, OBO selects one concept to be  designed to a greater level of detail.
The second set of design reviews, outlined in OBO policy, occurs during  the Schematic Design Phase, when the selected concept has been more  fully designed. The schematic review examines aspects of the proposed  design—though not a final design—against the project’s requirements,  approved schedule, and estimated construction contract price. The review  examines site context and surroundings; the proposed building systems,  including security systems and sustainability features; the exterior and  interior design elements and materials; and how the local environment  and construction labor may impact the design. The general purpose is to  highlight opportunities to strengthen the project before the design  progresses further and is completed.
Following both OBO senior management and industry concept and  schematic design reviews, the contracted design firm makes a  presentation to OBO’s Director. OBO’s Director may either approve the  proposed schematic design to be used for a project or indicate necessary  changes to the design for its subsequent approval. After OBO’s Director  has approved the schematic design for a project, the final design must be  in keeping with the approved design. Senior OBO officials indicated  industry advisory design reviews do not add additional time in OBO’s  process, as they occur within the overall time allotted for design. See  appendix VII  for a figure depicting where IAG design reviews occur within  OBO’s overall design process.
The first Excellence project to go through the industry design review  process was the new U.S. embassy planned for Mexico City. Through  April 2016, OBO had conducted a total of 27 industry design reviews on  14 Excellence projects. Figure 7 shows the three design concepts that  underwent an IAG design review and the schematic design of the  selected concept for the new consulate in Hyderabad, India.
Design firms we spoke with had varying views on the utility of OBO’s  industry advisory reviews. For example, two design firms reported that  although the contractual requirement to develop three design concepts for  review and consideration adds some value, making it a formal  requirement and holding a structured, peer-reviewed process adds  additional time, cost, and work. Another design firm new to OBO found  the process valuable, particularly when peer reviewers (members of the  IAG design review panel) already have OBO experience and can provide  advice on potential embassy design pitfalls.
section_title: Different Project Delivery Approaches Offer Distinct Design, Schedule, and Cost Trade-Offs
paragraphs: OBO’s Guide to Excellence indicates that different delivery methods have  design, schedule, and cost implications that must be evaluated relative to  the characteristics of each project and that OBO’s Director must approve  the delivery method for each project. Since 2011, OBO has generally  been using both bridging and DBB as delivery methods to have more  control over project designs, according to OBO officials. OBO officials  believe that greater design control under Excellence will improve  embassies’ appearance in representing the United States, functionality,  quality, and operating costs. Table 3 lists various design, schedule, and  cost trade-offs inherent in the DB, bridging, and DBB project delivery  methods identified by industry studies and experts we interviewed.
Construction contractor representatives we spoke with reported seeing  these issues play out in the execution of OBO’s Excellence approach.  AGC’s representative told us that the more customized designs under  Excellence create increased risks for design problems or errors that could  result in cost and schedule increases. Two OBO construction contractors  we spoke with reported that OBO’s bridging and DBB projects cost more  and take longer from start of design to completion when compared with a  SED DB project. In part, these contractors said they had found some  problems with some aspects of the designs, which took time to resolve  with OBO and OBO’s contracted design firms. Those contractors also  said OBO’s Excellence projects tend to specify more unique materials or  custom-made products, which also adds to construction costs. These  contractors also stated that they are tracking more change orders and  redesign work on current OBO projects, which further indicates the  potential for cost and schedule growth under Excellence. Finally,  contractors we spoke with said that while OBO now uses bridging to  develop a partial customized design for some OBO projects, their own  firms’ design costs for a bridging project will not be lower than a similar  SED DB project. They said this is because their own design firms are  still responsible for the design and must validate any design information  that OBO’s bridging architects develop.
section_title: OBO Staff Held Split Opinions on the Excellence and SED Approaches
subsections: section_title: Benefits and Challenges of Excellence
paragraphs: OBO staff expressed a wide range of opinions in response to our survey  request for comments regarding any specific benefits or challenges  brought about by Excellence in their specific area of expertise. Some  421 respondents provided comments covering diverse topics that we  evaluated and grouped into 20 categories. Staff often held opposing  views regarding a wide range of Excellence issues such as developing  Excellence standards or procedures, facilitating stakeholder input, and  focusing on maintenance and sustainability. Staff providing comments  generally provided more negative narratives than positive ones. Table 4  summarizes the results of our analysis.
For examples of specific benefits and challenges cited, see appendix III.
section_title: Comparison of Excellence and SED Approaches
paragraphs: We asked OBO staff to characterize Excellence compared with the SED  approach in terms of producing diplomatic facilities that are outstanding in  all respects, including security, architecture, construction, sustainability,  operations and maintenance. Of the 339 staff expressing an opinion, 157  (46 percent) identified the SED as generally more effective, 109 (32  percent) identified Excellence as generally more effective, and 73 (22  percent) believed they were equally effective.
We analyzed responses by length of service in OBO. Some 288 staff with  5 years or less experience responded to the question. These respondents  were hired around or after the introduction of Excellence in 2011. Most  respondents with 5 years or less experience, 204, did not provide an  opinion about Excellence or SED. Of the 84 respondents with 5 years or  less who did express an opinion, 37 (44 percent) indicated Excellence  was more effective, 31 (37 percent) reported SED was more effective,  and 16 (19 percent) found both equally effective. On the other hand,  many OBO staff with 6 or more years of experience who responded (255  out of 395) offered an opinion. Of the 255 staff with more than 5 years’  experience who had an opinion, 72 (28 percent) indicated Excellence was  more effective, 126 (49 percent) reported SED as more effective, and 57  (22 percent) found both equally effective.
We also selected offices for further analysis based on size. Of the  respondents expressing an opinion to the question from particular offices  within OBO (specifically not including those with “No opinion/no basis to  judge” or who provided no response), the four largest offices found SED  generally more effective. Staff from Facility Management were most  closely divided. Of the 62 Facility Management staff who had an opinion,  24 (39 percent) said that the SED program is generally more effective  than the Excellence program, while 20 (32 percent) said that Excellence  is generally more effective, with the remaining 18 (29 percent) reporting  one program as effective as the other. A larger percentage of  Construction Management and Design and Engineering staff reported  SED as generally more effective. Of the 79 Construction Management  staff who had an opinion, 44 (56 percent) said that the SED program is  generally more effective, compared to 20 (25 percent) who said  Excellence is generally more effective. Of the 58 Design and Engineering  staff who had an opinion, 28 (48 percent) said that the SED program is  generally more effective, compared to 18 (31 percent) who said  Excellence is generally more effective.
The greatest division among the large offices was in Security  Management, with 21 out of 29 (72 percent) who had an opinion reporting  SED was generally more effective than Excellence, compared with 3 out  of 29 (10 percent) reporting Excellence was generally more effective.  These offices are among those most directly involved in the planning,  design, construction, maintenance, and security at U.S. facilities  worldwide.
Other offices were more supportive of Excellence. Of the 24 respondents  from the Office of Project Development and Coordination who had an  opinion, 13 (54 percent) reported that Excellence is generally more  effective than SED. All remaining offices were more narrowly split, with 35  out of 87 (40 percent) reporting Excellence as more effective than SED  and about 32 out of 87 (37 percent) reporting SED more effective. The  OBO Front Office firmly supported Excellence as more effective than SED  with five of six (83 percent) of respondents who had an opinion saying so.
Some 403 staff provided narrative comments comparing SED with  Excellence. We categorized the comments, some of which fell into more  than one category. The most common positive comment regarding  Excellence cited aesthetic or architectural improvements, while the most  common negative comment noted higher costs under Excellence  compared to SED. The tally for the categories is in table 5 below.
Figure 8 lists some selected comments comparing the SED and  Excellence approaches.
When asked whether Excellence had generally improved the Capital  Security Construction Program (i.e., the embassy construction program),  OBO staff who responded were more evenly divided. Of the 470  respondents expressing an opinion, 174 (37 percent) generally agreed  that Excellence improved the program, while 161 (34 percent)  respondents generally disagreed, and 135 (29 percent) neither agreed  nor disagreed.
section_title: OBO Has Established Some Implementation Guidance but Lacks Tools to Assess Performance under Excellence
paragraphs: While OBO has established some policies and other guidance to  implement Excellence, it lacks tools to fully evaluate the performance of  the new approach. OBO continues to document changes in its policies,  procedures, standards, and other guidance. In our survey, OBO staff  generally were evenly split on the sufficiency of OBO’s efforts in these  areas. However, OBO has not defined performance measures specific to  Excellence goals at either the strategic or project level, such as greater  adaptability to individual locations, functionality, or environmental  sustainability. OBO also lacks a centralized database to broadly manage  Excellence by enabling, for example, effective reporting on projects’  design and construction costs and schedules. Without performance  measures specific to Excellence and sufficient systems to collect and  analyze relevant data, OBO will not be able to demonstrate whether the  performance of Excellence projects over time justifies the increased  emphasis on and investment in their designs.
subsections: section_title: OBO Continues to Document Changes in Its Policies, Procedures, Standards, and Guidance
paragraphs: While OBO has created or updated some policies and other guidance to  implement Excellence, it has taken more time to do this than OBO  estimated in 2011. Key guidance deliverables in OBO’s 2011 Excellence  decision memo—identified as “critical elements” by OBO—were to be  produced within the first year after Excellence was approved. However, it  took more time than OBO estimated to issue some of those key elements.  For example, OBO replaced the SED with the new OBO Design  Standards in 2013 and released its Guide to Excellence in 2016, despite  its initial plan to release these documents within a year of the memo.  New or updated policies issued in support of implementing Excellence  were also not in place until nearly 2 years or more after Excellence was  approved in 2011. For example, in recent years OBO has finalized  several policies, such as the following:
2013 Site Selection: This new policy emphasizes criteria for urban  sites; attributes of the preferred site include (1) considering American  values in promoting a sense of openness, accessibility, and  transparency through location; (2) proximity to key host-government  facilities, embassies of other countries, and businesses and cultural  centers; and (3) an urban setting that provides connectivity to public  transportation and infrastructure, making the mission accessible to  visitors and clients.
2014 OBO Senior Management and Industry Advisory Group  Design Reviews: This new policy requires two reviews by external  industry advisors and approval of Excellence designs by OBO’s  Director.
2015 OBO Core Project Team: This new policy requires OBO’s  Design Manager to be an integrated team member—with OBO’s  Project Manager, Project Director, and Construction Executive—to  ensure that decisions about a project’s design are integrated at  project inception and maintained through project completion.
2016 Architect/Engineer Team Selection: This revision to an  existing policy governs the evaluation and award of design contracts  to design firms. The revision expands the evaluation panel from at  least three to up to seven members, with key changes being the  addition of an OBO Director’s designee; a representative with a  connection to the post or regional bureau; and an external advisor (a  federal employee from another agency).
section_title: OBO Has Introduced Innovation but Faced Challenges in Implementing Excellence
paragraphs: A report commissioned by OBO provided insight into the challenges faced  in implementing Excellence. In June 2014, OBO modified an existing task  order with one of its design firms to require the firm to participate in a  roundtable discussion about Excellence and identify ways Excellence  might be more effectively communicated and managed. The design firm  subsequently delivered a report, based on the roundtable and its own  experiences, with the firm’s findings and recommendations on how to  improve Excellence.
The roundtable included OBO officials, DS, the Office of Logistics  Management, and contractors. The resulting consultant’s report noted  that OBO had transformed its design approach, design guidelines, project  requirements, and preferred project delivery methods. According to the  report, this transformation presented significant opportunities. For  example, the introduction of highly regarded design firms new to working  with OBO offered tremendous potential for innovation and overall quality  of building design and performance. However, the report also identified  challenges with Excellence, including the following:
There did not appear to be an entity responsible for instituting and  managing the significant organizational change that the Excellence  approach imposed.
The Excellence process had altered the internal practices at OBO and  its offices; with every project, its implementation has resulted in  nuances in the definitions of the Design Standards that led to  variations in their implementation, document submittal requirements,  milestone analysis, and security risk assessments.
While the new Design Standards were more comprehensive, they  were very difficult to navigate, particularly as a Portable Document  Format (PDF) file with over 8,000 pages.
Senior management reviews and the IAG peer review process  extended project schedules, and their purpose appeared unclear.
The consultant’s report made numerous recommendations to OBO,  including that OBO (1) seek to manage and clarify change internally; (2)  assign dedicated staff to be responsible for instituting change; (3) utilize  more standardization for project requirements, while acknowledging the  recommendation may seem counterintuitive in light of OBO’s move away  from the SED; (4) further define the Excellence program to capture the  new standards and processes OBO was instituting; and (5) train OBO  personnel and modify internal systems and practices to be compatible  with the new OBO project delivery methods and design standards.
In discussing OBO’s implementation of Excellence with us, senior OBO  officials stated that they continue to work to improve OBO’s processes.  They noted that the development or updating of OBO policies and  procedures takes considerable time because numerous OBO technical  offices must weigh in on any needed or proposed changes and that OBO  management must then review and approve those changes (or send  them back for revisions). Senior OBO officials also maintained that it was  difficult to “describe a program at the same time that you are  implementing it.” One former senior OBO official stated that OBO’s  priority during the transition to Excellence was trying to implement the  new program and that they may have lagged in establishing policies and  procedures to document changes to OBO’s processes.
section_title: Survey Respondents Held Mixed Opinions on OBO’s Provision of Guidance Related to Excellence
paragraphs: In our survey seeking feedback on Excellence, we asked OBO staff  whether they agreed or disagreed with seven statements about OBO’s  establishment and communication of strategic direction, policies, and  guidance for doing their daily jobs. OBO staff who responded agreed  most strongly with statements about the provision of policies and  standards for their daily jobs. They divided more narrowly on statements  about strategic vision and guidance (see table 6). The largest percent of  OBO staff who responded generally agreed with the statement: “Since  2011, OBO has provided clear and comprehensive technical standards  and guidelines related to my job.” The largest percent of OBO staff who  responded generally disagreed with the statement: “Since 2011, OBO has  provided clear and comprehensive strategic or long-term guidance to  implement its planning, design, construction, and maintenance approach.”
section_title: OBO Lacks Performance Measures to Evaluate the Potential Costs and Benefits of Excellence
paragraphs: While OBO has established Excellence and taken some steps to  implement it, OBO has not established strategic or project-level  performance measures to evaluate and communicate the effectiveness of  the Excellence approach in delivering embassies under the Capital  Security Construction Program. Performance measures are essential  tools for managers to evaluate progress toward a program’s objectives.  GAO’s Standards for Internal Control in the Federal Government state  that agencies’ internal controls should include the establishment and  review of performance indicators. Furthermore, State’s Foreign Affairs  Manual indicates that State must maintain effective systems of internal  controls that incorporate GAO’s internal control standards. In addition,  the AIA  2009 report stated that “OBO should be willing to evaluate and  explain the benefits of integrating security with design excellence, and the  potential benefits to life-cycle costs, design, operations, maintenance,  public image, and public diplomacy. OBO’s ability to explain the benefits  will require some empirical evidence of claims made for those tangible  items such as cost benefit and operations.”
Both OBO’s 2011 approval of Excellence and 2016 Guide to Excellence  assert that a design excellence program will provide the best value for the  U.S. taxpayer. According to GAO’s Business Process Reengineering  Assessment Guide, if an agency decides to initiate a reengineering  project, it should develop and communicate a compelling business case  to customers and stakeholders that supports this decision. Such a  business case should contain critical performance measures relating to  the organization’s core business processes, such as cost, quality, service,  and speed. As an agency completes its process redesign work, the  business case should be updated to present a full picture of the benefits,  costs, and risks involved in moving to a new process. Without meaningful  performance indicators, an agency has no way of knowing if the new  process has produced the desired results and whether those results  compare favorably or not to the previous process.
subsections: section_title: OBO Has Not Established Strategic Performance Measures Specific to Excellence
paragraphs: OBO’s strategic plan does not define how OBO intends to evaluate the  performance of the Excellence approach. State’s 2010 press release  announcing Excellence and OBO’s 2011 Excellence decision memo both  noted that a comprehensive strategic plan was to be implemented in 2011  and would act as a roadmap for developing Excellence policies and  procedures. OBO senior officials told us that a 2010 presentation— briefed to the then Secretary—was OBO’s strategic plan for Excellence  implementation. The briefing document does not say how Excellence is  to be evaluated—one of the functions of a strategic plan—nor does it  outline any performance indicators to show how OBO would assess and  report on the extent to which Excellence facilities are any more safe,  secure, functional, sustainable, or more effective in better supporting U.S.  diplomacy than the SED facilities.
State’s department-level fiscal year 2014-2017 strategic plan is largely  silent on Excellence. Its single Capital Security Construction Program- related performance indicator is the relocation of 6,000 U.S. government  employees into more secure and functional facilities by September 30,  2017. OBO used a similar performance indicator under the SED  approach. This indicator provides no performance assessment on the  extent to which Excellence facilities are any more functional, sustainable,  or effective in supporting U.S. diplomacy. Furthermore, the projected  target may be low relative to past performance, since 6,000 employees  moved by September 2017 equates to an average of 1,500 employees  relocated per fiscal year (2014 through 2017). However, State reports that from 2000 through 2014 it moved over 30,000 people into more  secure facilities—which equates to an average of over 2,100 people per  year (based on actual performance). As a result, it is unclear whether  State’s target is an appropriate measure given OBO’s past performance.
Excellence is briefly discussed in OBO’s bureau-level Functional Bureau  Strategy for fiscal years 2015–2017. It states that OBO is implementing  design innovation and that Excellence will introduce improved use of  functionality, sustainability, and security for diplomatic facilities. That  strategy document includes bureau-level design and construction related  performance indicators, among others related to other OBO operations.  Those indicators include the following, among others: average duration  and cost growth for capital construction design standards are met and updated on an annual basis  incorporating lessons learned and other feedback from stakeholders  from prior years; and percent of new embassy and consulate compounds designed to  achieve LEED Silver certification.
While the first set of indicators can quantitatively measure performance  and enable OBO to report on the efficiency of project delivery under  Excellence (as OBO did previously under the Inman program and under  SED), those schedule and cost indicators do not address new aspects of  Excellence, such as lower operating costs or better support for U.S.  conduct of diplomacy. In addition, the latter two indicators in the list above  are, if anything, even less useful for assessing Excellence performance.  First, OBO updates its design standards annually and conducts design  reviews to ensure that projects meet those standards. Thus, it is unlikely  OBO would fail to meet this performance indicator. Second, according to  OBO documentation, LEED Silver certification has been an OBO design  standard since 2009, before Excellence. Thus, to meet design standards,  every Excellence embassy built—with an emphasis on greater sustainability—should be at least LEED Silver, so the indicator should be  100 percent, or very near to it. Furthermore, the LEED indicator assesses  only the performance implied by the design itself, not the actual building  operations and maintenance performance and whether the actual utility  usage and costs are equal to or less than initially  estimated in the  designs.
While no Excellence projects can be evaluated yet, as none have been  completed, without additional performance indicators relevant to the goals  of the Excellence approach, OBO has no way of knowing if its new  process is achieving the desired results. Furthermore, it lacks an  important tool for reporting on the Excellence approach to congressional  overseers, the public, and other State stakeholders such as other U.S.  diplomatic agencies that must help pay some of the costs for constructing  and maintaining new embassies.
section_title: OBO Is Exploring Ways to Better Track and Evaluate Long-Term, Project-Level Performance
paragraphs: OBO also lacks post-specific performance measures to track and  evaluate the long-term performance of its embassies. According to Office  of Management and Budget guidance, more than 80 percent of a  building’s total cost over its lifespan can consist of ownership costs such  as operations, maintenance, and energy usage. When combined with  front-end costs such as design and construction, these costs embody a  project’s “life-cycle costs.” OBO has attempted to address long-term  operations and maintenance costs on the front-end by, for example,  committing to include LEED Silver certification in its design standards  since 2009, according to OBO officials. Other sustainability “stretch”  initiatives OBO considers desirable (though not required) under  Excellence include trying to achieve LEED Platinum certification,  increasing use of renewable energy sources, reducing greenhouse gas  emissions, and achieving net-zero energy and water consumption on its  compounds, whereby enough renewable energy or water is generated to  meet a post’s requirements. OBO design standards also require design  firms to incorporate operations and maintenance cost analysis into  embassy designs through sustainability studies. From 2001 through 2015,  in locations where OBO has constructed a new diplomatic compound,  OBO reported it has constructed 26 new LEED-certified embassy or  consulate office buildings, 19 of which were SEDs.
However, despite the additional emphasis now focused on operations and  maintenance on the front end during design, OBO has no post-specific  performance measures related to operations and maintenance cost  performance after a new embassy is constructed. One reason for this is  the lack of available or reliable data. OBO officials stated that although  some embassies do have utility meters on site, getting data from there  back to Washington, D.C., is challenging. While OBO does have a data  system in place to capture some operations and maintenance  information, such as utility usage, it is dependent upon manual entry of  data at each specific post. According to OBO officials, this lowers data  reliability, and differences in data entry compliance by posts over time  make historical analysis of operations and maintenance costs difficult.  Also, while some posts have building-level  meters for the main office  building, other posts have compound-level utility meters that track data for  multiple buildings, making broader data comparisons difficult across  posts.
OBO is taking some steps to address this situation. According to OBO’s  2016 Guide to Excellence, OBO is in the process of developing  project  and portfolio operations and maintenance cost assessment procedures to  account for these costs over the estimated life of embassies. In 2016  OBO initiated an effort to develop a methodology and process to better  assess the full life-cycle cost of its projects. OBO’s July 2016 statement of  work on its Life-Cycle Cost Assessment effort shows that OBO intends to  develop a methodology and plan to assess the total cost of ownership of  projects and facilities, which takes into account the costs of (1)  acquisition, (2) design, (3) construction, and (4) operations and  maintenance. According to OBO officials, this effort represents a gradual  shift in OBO’s orientation, whereby OBO’s portfolio is expected to reflect  less emphasis on new construction and greater attention to maintenance,  repair, and renovations. Therefore, decisions must be made regarding  what metrics should be tracked.
OBO is also working with State’s Office of Management, Rightsizing,  Policy, and Innovation  on a pilot effort—called “MeterNet”—with the intent  to install more metering systems on embassy compounds and to transmit  performance data back to Washington. Under MeterNet,  State intends to  automate and improve the collection of data on electrical energy usage  (both utility and renewable sources), water usage, and fuel consumption.  According to OBO officials, MeterNet  should relieve posts of manual data  entry and also enable OBO to more accurately monitor, collect, and  analyze more reliable data on sustainable energy and water performance.  OBO anticipates that this in turn will enable facility managers to manage  energy consumption data across State’s facilities, as well as analyze and  track energy usage trends over time, such as energy per square foot or  overall electricity demand. OBO has also been working with the  Department of Energy to address challenges with its existing utility data  system. According to OBO officials, OBO has not yet determined how  MeterNet will interface with OBO’s existing data systems.
The steps OBO has taken to improve monitoring of post-specific  operations and maintenance costs are at a very early stage. Until OBO  clearly defines a process to assess the performance of its projects after  construction and establishes reliable data systems to track and report this  performance, OBO will lack an essential tool for determining whether  completed projects—whether Excellence or SED facilities—are  performing as intended by their designs from either a sustainability (e.g.,  energy and water usage) or an operating and maintenance cost  standpoint.
section_title: OBO Is Exploring a Centralized Data Solution to Better Manage Projects and Track Cost and Schedule Performance
paragraphs: OBO currently lacks easily accessible data to provide overall project  management information. During our review, we requested a variety of  data related to OBO’s embassy construction projects from January 2001  through September 2015, such as contract award amounts, site  acquisition costs, delivery method, completion dates, and other data.  However, OBO was unable to easily provide such information. According  to OBO officials, while these data did exist and could be retrieved, the  data were not available in any centralized data source. Rather, each OBO  office maintained separate data relevant to its own operations, and so  consolidated and current data to provide overall project information were  unavailable.
OBO offices consolidate certain project management information in  periodic project performance reviews, whereby individual  offices and  project teams report on cost, schedule, and scope for specific embassy  projects. However, while this can facilitate some data retrieval from a  specific ongoing project, according to OBO officials it can be difficult and  time consuming to find information on older, completed projects for which  there is no longer an active project team. We reported on similar data  issues in September 2014, when OBO could not provide all of the real  property files we requested, at that time also citing lack of centralized  data, maintenance by different groups within OBO, and difficulty of  retrieval.
According to federal internal control standards, quality information is an  essential tool for agency management to achieve an agency’s  objectives. According to these standards, a process should exist that  uses the agency’s objectives and related risks to identify the information  requirements needed to achieve the objectives and address the risks.  Furthermore, such data should be sufficiently relevant, reliable, and  accessible to agency management. Additionally,  OBO’s 2011 Excellence  decision memo cited the need for a comprehensive information  technology platform that would integrate and make available all OBO  project information; promote effective review, communication, and  decision making; and support the maintenance and operations of  completed facilities.
No such system existed at OBO in October 2015 at the time of our data  request. In response to our request, OBO began assembling a wide  range of project management data to fulfill our request as well as to better  provide information to Congress. We received these data in the form of a  spreadsheet 10 months later in August 2016. OBO officials attributed this  delay to the aforementioned difficulty of retrieving historical project data  as well as having to address concurrent information requests from  Congress and State’s Inspector General. The database we received  covers projects from January 2001 through September 2015, includes  many elements of the information we requested, and also includes some  other information useful to OBO management. According to OBO officials,  these data were compiled by OBO office units and project managers  based on the latest documentation available.
OBO recently established an initiative—termed the Ideal Operational  State—to explore long-term ways to centralize and standardize data  collection across OBO’s operations. According to OBO officials, this  Excellence-related initiative is intended to provide a long-term data  solution that will allow for better program management across OBO’s  business activities as well as better tracking of project metrics such as  cost and schedule performance. The study group tasked with assessing  OBO’s current information technology systems and potential market  alternatives held a kickoff in May 2016 and, after a series of working  sessions and vendor evaluations, recommended a series of actions to  OBO’s senior management, including an upgrade and modification of  existing OBO management software. OBO management approved action  on these recommendations in October 2016.
Until OBO develops an effective, centralized data system capturing  essential and reliable project management data as well as cost and  schedule performance across its project portfolio, not only will OBO  management lack a critical tool for consolidating key project data,  assessing performance, and guiding strategic oversight and internal  control, but it will also be hampered in responding to oversight queries by  Congress, GAO, and State’s Inspector General.
section_title: Conclusions
paragraphs: At the heart of OBO’s changes under the Excellence approach is the  premise that greater design focus and control will produce more  innovative, functional, and sustainable embassies that are just as secure  as those built using the SED but that will be more cost efficient to operate  and maintain. From fiscal year 2011 through fiscal year 2015, OBO has  allotted hundreds of millions of dollars to fund more customized designs  rather than applying a standardized design to build new embassies.  Though a greater upfront investment in design may yield embassy  improvements, it carries with it increased risk to project costs and  schedule. While OBO is attempting to manage this risk, without strategic  or project-level performance measures specific to the goals of Excellence,  OBO cannot fully assess the merits of this new approach. Furthermore,  as projects initiated during Excellence’s implementation come to fruition  and begin operations, such measures will be essential to any long-term  assessment of their performance. Establishing reliable data systems to  measure, record, and report on building performance can help OBO  management evaluate all costs that occur over a building’s lifespan.  Further, centralized project management data are also needed to allow  OBO to quantify and assess design and construction costs under  Excellence for each project. While OBO has begun efforts to establish  such systems, it will take time to complete these initiatives and collect  these crucial data. Nevertheless these steps are essential to creating safe  and lasting buildings that best represent the United States while ensuring  that projects make efficient use of resources and assess the value of  shifting to the Excellence approach rather than continuing to use the  SED.
section_title: Recommendations for Executive Action
paragraphs: To better assess OBO’s performance, we recommend that the Secretary  of State take the following four actions:  1.  Determine whether the existing OBO program performance measure  and annual target of moving 1,500 people into safe, secure, and  functional facilities is still appropriate or needs to be revised.  2.  Establish additional performance measures applicable to the new  goals of the Excellence approach in support of the Capital Security  Construction Program.  3.  Finalize the mechanisms OBO will use to better track and evaluate  the actual operations and maintenance performance of its buildings— whether Excellence or SED—and document through appropriate  policies, procedures, or guidance.  4.  Finalize the mechanisms OBO will use to centrally manage project  management data (to include project cost and schedule information),  currently termed the Ideal Operational State, and document through  appropriate policies, procedures, or guidance.
section_title: Agency Comments
paragraphs: We provided a draft of this report to State for comment. State provided  technical comments on the draft, which we incorporated as appropriate.  State also provided written comments that are reproduced in appendix  VIII.  In its written comments, State concurred with our four  recommendations and described actions planned or under way to  address each of them. State said it will  1.  Perform a comprehensive evaluation of its performance measure and  annual target of moving 1,500 people into safe, secure, and functional  facilities and determine whether that target remains appropriate.  2.  Develop new metrics applicable to the Excellence approach.  3.  Finalize the mechanisms it will use to better track and evaluate the  actual operations and maintenance performance of its buildings,  stating that this will occur after its life cycle cost analysis methodology  project produces its final report.  4.  Finalize the mechanisms OBO will use to centrally manage project  management data, noting that State expects the ultimate product of  this multiyear effort to provide a comprehensive framework for  managing project data.
We are sending copies of this report to the appropriate congressional  committees and the Secretary of State. In addition, the report is available  at no charge on the GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  either Michael J. Courts at (202) 512-8980 or at courtsm@gao.gov or  David J. Wise at (202) 512-5731 or at wised@gao.gov. Contact points for  our Office of Congressional Relations and Office of Public Affairs can be  found on the last page of this report. GAO staff who made key  contributions to this report are listed in appendix IX.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report examines (1) reasons for the Department of State’s (State)  shift to its Excellence approach, (2) key elements and trade-offs of the  new approach, and (3) the extent to which State has established  guidance and tools to implement and evaluate its Excellence approach.
To conduct this review, we obtained and analyzed information from State  policy, planning, funding, and reporting documents, administrative  memos, and select project documentation. We also interviewed officials  from State’s Bureau of Overseas Buildings Operations (OBO); Bureau of  Diplomatic Security (DS); Office of Management, Rightsizing, Policy, and  Innovation;  and Office of Acquisitions Management. Within OBO, we  spoke with officials from offices responsible for site acquisition, planning,  project development, design and engineering, cost management,  construction management, facility management, policy and program  analysis, and financial management. We also interviewed officials from a  variety of architecture and engineering (design) firms and construction  contractors that have worked for State. Additionally,  we met with experts  from several industry groups. In general, we did not review acquisition  plans, the complete contracts for each project, or the terms and  conditions that could have impacted cost, schedule, and performance of  any project.
To identify the reasons for State’s shift to its Excellence approach, we  analyzed relevant industry studies and OBO assessments from before the  introduction of Excellence. We also examined the outputs from OBO’s  2011 Excellence working groups as well as other Excellence  documentation, such as OBO’s “Guiding Principles of Design Excellence  in Diplomatic Facilities” and OBO’s 2011 memo approving the Excellence  approach. Also, because the decision to adopt Excellence was made in  2011—and the work leading up to the decision was undertaken in 2010— we interviewed key former OBO officials with direct experience with  OBO’s efforts to improve the Capital Security Construction Program at  that time, including some who served on OBO’s management steering  committee for Excellence.
To examine the key elements and trade-offs of the new approach, we  collected and analyzed OBO policy and procedures directives,  administrative memos, budget documentation, project authorization  documents, design standards, and design-related documentation. We  discussed changes in OBO’s process with relevant officials from OBO,  DS, and the Office of Acquisitions Management. We also discussed these  changes with officials from design firms and construction contractors that  had previously worked, or are currently working for State. Furthermore,  we consulted industry studies and spoke with experts from industry  groups, including the American Institute of Architects, the Associated  General Contractors of America, the Bridging Institute of America, and the  Design-Build Institute of America to determine the trade-offs inherent to  different delivery approaches.
To determine the extent to which State has established guidance and  tools to implement and evaluate its Excellence approach, we examined  changes to OBO’s policies and procedures directives, design standards,  standard operating procedures, and other guidance since 2011. We  compared these changes to goals and recommendations from OBO’s  approval of Excellence and also reviewed an OBO-sponsored study of its  implementation progress. Additionally,  we reviewed strategic planning  documentation, to include State’s strategic plan, OBO’s Functional  Bureau Strategy, and State’s Annual Performance Report. We also  consulted federal standards for internal control and business process  reengineering guidance. We also met with officials from OBO and the  Office of Management, Rightsizing, Policy, and Innovation  to discuss  efforts to evaluate embassy buildings and to improve OBO’s data  management.
To supplement our findings, we conducted a web-based survey of OBO  staff from July 15 through August 12, 2016, soliciting their views on the  sufficiency of OBO’s strategic vision, policies, procedures, and technical  guidance for the Excellence approach as well as any particular  efficiencies or challenges brought about by the Excellence approach.  This survey was sent to 1,511 OBO staff, 705 (47 percent) of whom  responded. We do not make any attempt to extrapolate our findings to  the remaining 53 percent of eligible employees who chose not to  complete our survey. The results of our survey provide measures of employees’ views at the time they completed the survey in July and  August 2016.
Because we surveyed all OBO staff, the survey did not involve sampling  errors. To minimize nonsampling errors, and to enhance data quality, we  employed recognized survey design practices in the development of the  questionnaire and in the collection, processing, and analysis of the survey  data. To minimize errors arising from differences in how questions might  be interpreted and to reduce variability  in responses that should be  qualitatively  the same, we conducted pretests with six OBO employees.  To ensure that we obtained a variety of perspectives on our survey, we  randomly selected one employee from each of the following offices to  pretest the survey: Area Management; Construction, Facility, and Security  Management; Design and Engineering; Planning and Real Estate;  Program Development, Coordination and Support; and Security  Management. Based on feedback from these pretests, we revised the  survey in order to improve the clarity of the questions. An independent  survey specialist within GAO also reviewed a draft of the questionnaire  prior to its administration. To reduce nonresponse, another source of  nonsampling error, we followed up by e-mail with employees who had not  responded to the survey to encourage them to complete it.
To analyze open-ended comments provided by those responding to the  survey, we conducted a content analysis in two steps. In the first step,  analysts read the comments and jointly developed categories for the  responses. In the second step, each open-ended response was coded by  one analyst, and then those codes were verified by another analyst. Any  coding discrepancies were resolved by the analysts agreeing on what the  codes should be.
Additionally,  many comments touched upon findings we developed  through our separate audit work. We have included some of these  comments for illustrative purposes in appendix III.  Respondents generally  provided more negative comments than positive ones; however, where  possible, we have tried to present a balanced set of positive and negative  comments. In some cases, we edited responses for clarity or grammar.  Views expressed in the survey may not be representative of all OBO staff  views on given topics.
We conducted this performance audit from August 2015 to March 2017 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Results of GAO’s Survey of Staff of the Department of State’s Bureau of Overseas Buildings Operations
paragraphs: We conducted a web-based survey of the Department of State’s (State)  Bureau of Overseas Buildings Operations (OBO) staff from July 15  through August 12, 2016, soliciting their views on the sufficiency of OBO’s  strategic vision, policies, procedures, and technical guidance for the  Excellence approach to the design and construction of U.S. embassies  and consulates, as well as any particular efficiencies or challenges  brought about by the approach. We sent the survey to 1,511 OBO staff,  705 (47 percent) of whom responded. We do not make any attempt to  extrapolate the findings to the remaining 53 percent of eligible employees  who chose not to complete our survey. The results of our survey provide  measures of employees’ views at the time they completed the survey in  July and August 2016. The questions we asked in our survey are shown  below. Our survey comprised both fixed-choice and open-ended  questions. In this appendix, we include all survey questions and  aggregate results of responses to the fixed-choice questions and the  number of responses provided to the open-ended questions. We do not  provide text of the responses to the open-ended questions. For a more  detailed discussion of our survey methodology, see appendix I. For our  summary analysis and selected examples of comments provided in  response to open-ended questions, see appendix III.
Numeric Responses to GAO’s Survey of State’s Embassy  Construction Program  1.  About how long have you worked with OBO?    Less than 1 year     1 to 5 years   6 to 10 years       11 to 20 years     More than 20 years   Provided no answer to this question   76 respondents  218 respondents  150 respondents  170 respondents  89 respondents  2 respondents  2.  Which of the following categories best describes your position?    Foreign Service     PSC     Other: _______________________     Provided no answer to this question   264 respondents  214 respondents  134 respondents  86 respondents  7 respondents  3.  How many new NEC/NCC projects have you worked on or supported  as an OBO employee? Please do not include annexes or where you  were covering for someone else.   None 127 respondents      162 respondents      100 respondents       1 to 3    72 respondents      239 respondents       4 to 6   5 respondents     7 to 12    More than 12    Provided no answer to this question  4.  Where is your current posting?    Headquarters (SA-6)      Overseas   Provided no answer to this question  501 respondents  196 respondents  8 respondents  5.  In which OBO office do you currently work?
Operations (OPS):    Area Management     Fire Protection     Safety Health and Environmental   Resource Management (RM):    Financial Management     Office of the Executive Director     Policy and Program Analysis    8 respondents  25 respondents  5 respondents  Other:    Other: ____________________________    52 respondents   42 respondents  3 respondents I’d rather not identify my office     Provided no answer to this question    6.  Do you agree or disagree with the following statements about the  program direction of OBO’s construction program? What perspectives  or specific examples can you provide to illustrate your answers?
7. Since 2011, what, if any, benefits or efficiencies—related to how OBO  now plans, designs, constructs, and maintains NECs/NCCs—have been  introduced within your specific area of expertise (i.e., site acquisition,  planning, cost estimating, project management, scheduling, design,  construction, security management, and facility management)?
We received 373 written responses to this question.  8. Since 2011, what, if any, challenges or inefficiencies—related to how  OBO now plans, designs, constructs, and maintains NECs/NCCs—have  been introduced within your specific area of expertise (i.e., site  acquisition, planning, cost estimating, project management, scheduling,  design, construction, security management, and facility management)?
We received 387 written responses to this question.  9. To the extent you have knowledge, how would you characterize the  Excellence program (roughly 2011 to present) as compared to the SED  program (roughly 2001 to 2011) in terms of producing diplomatic facilities  that are outstanding in all respects, including security, architecture,  construction, sustainability, operations and maintenance?  1. The Excellence program is generally more        effective than the SED program.     2. The SED program is generally more effective        than the Excellence program.    3. The Excellence program is generally as        effective as the SED program.    4. No opinion/no basis to judge  Provided no responses to this question   9a. What specific examples regarding security, architecture, construction,  sustainability, or operations and maintenance can you provide to illustrate  your answer?
We received 403 written responses to this question.  10. What additional information, if any, would you like to share in order to  further elaborate on any of the responses you provided above?
We received 338 written responses to this question.
section_title: Appendix III: Selected Comments from Our Survey of Bureau of Overseas Buildings Operations Staff, with Summary Tabular Analyses
paragraphs: Of the 705 respondents, 550 provided comments in response to at least  one open-ended question in our survey of Department of State Bureau of  Overseas Buildings Operations (OBO) staff. For specific questions, we  analyzed and categorized respondents’ comments and have reproduced  selected comments below to characterize the results of that analysis. In  addition, since many of the comments touched upon findings we  developed through our separate audit work, we have also included some  of those comments for illustrative purposes. Respondents generally  provided more negative comments than positive ones; however, where  possible, we have tried to present a balanced selection of positive and  negative comments. In some cases, we edited responses for clarity or  grammar. Views expressed in the survey may not be representative of all  OBO staff views on given topics.
Table 9 summarizes the results of our analysis to categorize comments  expressing opinions on the benefits and challenges of OBO’s Excellence  approach to the design and construction of new embassies and  consulates.
The two text boxes that follow contain selected narrative responses on,  respectively, the top four most-cited benefits and the top four most-cited  challenges of the Excellence approach.
Selected Survey Comments on the Top Four Most-Cited Benefits of the Excellence Approach  Development and/or improvement of standards, processes, procedures, templates, documents, etc.:  The 2016 OBO Design Standards have greatly improved the design process.
We’ve started doing constructability reviews during design, which is helpful at making sure the designers are realistic in what they  propose and that projects can be implemented in the specific region.
Improved coordination;  facilitating input from different stakeholders/internal teams; consensus and leadership around  program objectives:  The different offices in OBO seem to work more closely together through the life of the project. There is still some “stove-piping,”  but it generally works better than previous years.  Seems to be better coordination between OBO and the embassies and consulates.
Increased focus on maintenance/sustainability/life cycle analyses:
OBO has recently directed its attention to the large role Facility Management undertakes during and after the construction of [an  embassy]. It takes OBO 3 to 5 years to get a  constructed, but OBO Facility Management has the responsibility for  operations and maintenance for  following 50 years of life. Facility Management now has a role in design and is  working toward obtaining a meaningful role in the construction phase.
Operations and Maintenance (O&M) are finally being brought into the full realm of planning, design, and construction of new  facilities by current management. The greatest expense to the  is not in the planning or the design or the  construction of new facilities, it is in the O&M  of facilities over their 50-year average life cycle. OBO must bring greater focus as  an organization on not just design excellence but on the “Total Ownership Cost” by incorporating the facility management experts  and into every  single facet  in order to minimize the O&M costs to the taxpayer over that  50-year life cycle. This will require resources,  not currently at hand.
Greater control/flexibility  of designs and site selection:
My  understanding is that previous to 2011 most projects were SED design-build. While this is a very  efficient delivery method for  some projects, it is not necessarily the best fit for every  project. I believe that the Excellence Initiative provides the flexibility to  review and select the most appropriate design and delivery method for each project taking into account the unique budget,  schedule, and site-specific parameters each project has.  Since 2011, we have co-led the effort to search for and legitimize smaller sites in downtown or urban locations. Prior to 2011, the  practice was to find overscaled sites outside of the urban core. The Excellence initiative allows for greater flexibility and  customization to a specific site (relative to SED), allowing for greater efficiencies. Those efficiencies can, and often do, lead to  reduced construction and operational costs.
Legend: OBO = Bureau of Overseas Buildings Operations; SED  = Standard Embassy Design.
Selected Survey Comments on the Top Four Most-Cited Challenges of the Excellence Approach  Lack of, inadequate, or inconsistent application of policies/procedures/standards/systems; uncertain impact of new policies,  etc.:  The transition from “design-build” back to “design-bid-build” construction has been poorly implemented. Project design documents  have not fully achieved the transition to the level of quality and detail required for overseas “design-bid-build” construction. The  effort to move from standard design documents has further made achieving the level of quality for construction and security more  difficult.  Clear, applicable, specific, and enforceable standards have been watered down or replaced with less specific and in some cases  tentative suggestions. Senior OBO management has difficulty objectively articulating design excellence goals or even attempting  to measure results. Contract performance in particular is difficult to measure or, in some cases not obtained due to lack of  quantifiable criteria.
New/slow/problematic processes and/or requirements resulting from more complex and varied projects:  Challenges have been introduced in realizing design intent in most of the underdeveloped countries that construction occurs in.  Designs are complex and the materials exotic for the location. Inefficiencies have been introduced in requiring senior  management reviews of  projects without fully defining what the intent of such reviews are. Challenges have been  introduced in the constructability of building features. Challenges introduced with reliance on outside architect firms to develop  plans and drawings and to judge design intent or answer questions.
My  required time spent on projects has at least doubled because of the policies put in place since 2011. In terms of  improvements, more aesthetically pleasing facilities are being produced, but in my opinion the amount of effort required to attain  good design is very disproportionate to the effort required to achieve this goal. This is due to a poorly organized process and a  severe lack of communication throughout the organization.
Schedule challenges; extended timelines:  Schedules are consistently moved forward with longer timelines to accomplish the design portion of the project.  Planning and design take much more time and effort than before. Assurance of meeting security criteria is challenging, since  every  new embassy is a “one of a kind” project. More daunting is the process by which the Front Office approves design  concepts. That alone has added 3 to 6 months to the planning and design schedule.
Budget challenges; high costs:  Since no two posts have the same size, plan, finish materials, exterior components, or operating systems, I see no efficiencies  whatsoever. The push for high initial budgets is to be able to cover the costs associated with the fancier design elements.  Site acquisition is more difficult, and more costly than ever. Only urban locations and not cheaper bigger suburban sites are even  considered. Project design costs are exponentially higher through the Excellence program. Higher costs mean fewer projects  being planned for or constructed in any given year.
Legend: OBO = Bureau of Overseas Buildings Operations; SED  = Standard Embassy Design.
Excellence costs more compared with SED  (“budget challenges”)
Excellence takes more time compared with SED  (“schedule challenges”)
Excellence introduces new/ slow/ problematic  processes and/or requirements resulting from  more complex and varied projects      Under Excellence (compared with SED): Lack of,  inadequate, or inconsistent application of  policies/ procedures/ standards/ systems; poor  communication of policies, etc.
Legend: — = No contrasting comments w ere identified in this category.
Selected Survey Comments Regarding the SED and Excellence Approaches  The SED model streamlined many processes which apparently translated into a more expedient overall delivery of new facilities.  The Excellence program successfully addresses many of the SED program drawbacks (“embassies looking similar and like  fortresses”), however, at a price (arguably longer and more expensive projects). The decision clearly needs to be made whether it  is worth that price.
While the SED program was severely limiting, it is my belief that there is a middle ground between the SED and the Excellence  program—where OBO has a set of standards, specifications, and requirements that are clearly communicated to the design firm  while allowing them to customize the footprint and design features of the building. OBO has buildings in its portfolio that were built  using “SED Criteria” that are not “prison-like” and forbidding. It can be done in a thoughtful and efficient way that would appeal to  people architecturally. In plain words—OBO didn’t have to throw the baby out with the bathwater.
On the positive side, designing unique facilities improves the aesthetics of the U.S. presence abroad and sets an example for  building system efficiency and innovative systems to the world. On the negative side, each unique design requires re-inventing  the wheel and creates additional challenges for the designer to integrate physical and technical security, as well as accounting for  building maintenance and upkeep. These designs are often difficult to implement overseas, and the state-of-the-art systems are  difficult to maintain in some countries with unique equipment and a significant increase in facilities staff. Unique designs are more  difficult and frequently more expensive to implement, creating an impact on time and cost.  The Excellence program designs buildings that are architecturally pleasing, but in order to achieve nice aesthetics and achieve  the security mandates that embassies must adhere to, I believe a cost premium exists in order to meet the aesthetics of the  embassies designed under the Excellence program. I think a compromise can be achieved, and in that compromise some cost  savings could be realized, but in my opinion the pendulum has swung completely opposite that of the SED, and the evidence is in  a cost comparison between the embassies designed under the SED program to those designed under the Excellence program.  The designs of  are more challenging or difficult to introduce, implement, and execute. Every  design is unique and  different, takes longer to evaluate, and requires more coordination; more issues are encountered that need to be resolved. The  construction of  is also more challenging or difficult to execute. The distinct construction materials used are more  expensive, take longer to procure and transport, and are more difficult to install. Therefore, the schedule is longer, especially if  there are more issues encountered during design and construction, and the cost of construction and contingencies are much  higher.  SEDs were all about schedule and budget and when either of these were threatened, things were de-scoped or glossed over.  Many  times those were support annexes (shops, storage facilities) that allowed the facilities to be properly maintained or morale/  welfare/recreation facilities or amenities, meaning that these things had to be added after the fact, putting a great deal of pressure  on posts. Excellence does a better job in terms of incorporating architectural features, energy efficiency and sustainability design,  and delivering a fully realized compound.  I believe that the Excellence program produces a better product and platform for diplomacy; however, most of us also  acknowledge that the SED program gave us the ability to execute faster when needed. They should not be mutually exclusive.  There are principles in the SED that can be applied and probably should not have been just “thrown out.” It should have been a  good solution for an expeditious need, in a place where appropriate. We also know that it did not work well everywhere, and in  places with stringent code, zoning, or other restrictions (including the desire to operate in an urban context), it was definitely not a  good solution. As long as the Front Office doesn’t focus solely on aesthetics, the Excellence program should produce some  outstanding facilities that have far more flexibility than the SED, better functionality, and improved suitability for the country in  which they reside.
Legend: OBO = Bureau of Overseas Buildings Operations; SED  = Standard Embassy Design.
Many responses to open-ended questions in the survey touched upon  issues we reviewed through our separate audit work. We have included  some of these responses for illustrative purposes in the text boxes that  follow. In some cases, we edited comments for clarity or grammar.
Selected Survey Comments on Concerns with SED  The SED program was a failure for site specific implementation. It is generally known around OBO that there was no such thing  as a real SED building. Due to the technical requirements and the site specific requirements, all SED compounds required  adaptation, usually at significant cost and time.  The SED had plenty of problems. Yes, the main issue was the boring and unimaginative concrete box; however, it did produce  embassies. There is a middle road that opens up the options and opportunities for improved design without going overboard.  The SED program was effective in quickly and efficiently providing safe and secure facilities to perform diplomatic activities  overseas using the  delivery method. However, the designs were not very attractive  and didn’t represent the best in  American architecture.
Legend: OBO = Bureau of Overseas Buildings Operations; SED  = Standard Embassy Design.
Selected Survey Comments on the Desire to Improve Embassies under the Excellence Approach  The Excellence initiative is a better way to design the  Department’s projects and buildings. It takes advantage of  innovation in the design generated by architects and engineers from different points of view. The SED program produced more  and more generic buildings that translated into more or less a template mentality.  The architecture is less fortress-like, more approachable to local populations, more culturally sensitive (more diplomatic). It is  more attractive and something that Americans can be prouder of, as well as being greener and increasingly sustainable.  From my limited experience with OBO, it appears as though the Excellence program attracts better design firms to the program.  Better design firms should result in a better end product. The Excellence program also appears to result in more excitement from  the host country and allows for embracing culturally specific designs.
Legend: OBO = Bureau of Overseas Buildings Operations; SED  = Standard Embassy Design.
Selected Survey Comments on Industry and Senior OBO Management Design Reviews  The incorporation of the Industry Advisory  Reviews gives design firms real-time feedback from their peers on not only how to  support the Excellence program, but how to save costs and make buildings more efficient during the design process.  I think that the new processes are moving forward, and we are seeing some positive results with stronger designs and hopefully,  construction. For example, there were positive reactions by both the staff and private sector architects to the Industry Advisory  Reviews that I attended.  Too much time and effort is spent on reviews by OBO Front Office and outsiders. Front Office seems to be inconsistent in what it  wants in a design and will change its mind from one review to the next. We are hiring professional design firms. They should be  able to do their jobs.  The design process has been significantly impacted by the introduction of numerous senior management and Industry Advisory  Group reviews. These reviews are expensive and time consuming. An independent evaluation of their value versus their cost is  warranted.
Legend: OBO = Bureau of Overseas Buildings Operations; SED  = Standard Embassy Design.
Selected Survey Comments on OBO Guidance Related to the Excellence Approach  I do believe OBO as a whole is trying to maintain and keep up with outdated policies. In my observations OBO is very  understaffed or doesn’t have the right mix of professionals necessary to maintain policy and regulations.  There is no mention anywhere of changes to the security program brought on by Design Excellence. In my opinion, no one has  ever thought through what it means to construct anything other than a SED. There is no Design Excellence guidance for security  professionals.  The Guiding Principles are widely available to staff and clearly reflect the vision of the senior leadership.  The Design Excellence guide took 2 years to write; then they wanted the operations and maintenance piece added but gave us  initially only 2 weeks.  Case in point. Just yesterday an OBO notice came out on the Excellence Initiative. No doubt it had something to do with the  timing of this   survey.
OBO has been working to modify policies and procedures directives (PPDs), etc.; however, it takes time, and it is difficult to get  people to take the time away from their regular duties to step back and look at policies.
Legend: OBO = Bureau of Overseas Buildings Operations; SED  = Standard Embassy Design.
Selected Survey Comments on the Need for Excellence-Specific Performance Metrics  I have not seen any performance metrics that compare before/after the Excellence Initiative.
What are the performance metrics used to measure this ?
Senior OBO management has difficulty objectively articulating Design Excellence goals or even attempting to measure results.  I am not aware of any metrics for evaluating the “Excellence” of a project and the many facets involved in assessing a building.
What metrics does OBO employ to measure “effectiveness” or “efficiency,” specifically, strategically? Answer: none. Each project  takes as long as it takes. The project costs what it costs.  Several presentations have been given and a few documents produced to try to articulate Design Excellence; however, much of  the concept is subjective, due to the emphasis on architecture, and cannot be measured or effectively monitored for performance.  There are no real performance measures; total operations and maintenance costs are unknown.
Legend: OBO = Bureau of Overseas Buildings Operations; SED  = Standard Embassy Design.
Selected Survey Comments on the Need for Better Project Data and Systems  Project database does not have a user manual or guide, either. Who does this? Why create a database and never provide a user  guide or how-to manual for the user of the data? We are supposed to have a new and better projects database.
We used the same databases/networks and computers for the last 10+ years. Not enough server space to store the size/volume  of electronic files/docs that we produce. The “Ideal Operational State” research/data collection effort, however, we hope, will  resolve some of these technological issues.  The Ideal Operational State effort is an excellent opportunity to manage the Life Cycle Cost Analysis of the entire business  process life cycle. The existence of the Ideal Operational State, while itself a very  good thing, indicates the need for more  strategic alignment of process with and definition of a vision.
Legend: OBO = Bureau of Overseas Buildings Operations; SED  = Standard Embassy Design.
section_title: Appendix V: Guiding Principles of Excellence in Diplomatic Facilities; U.S. Department of State Bureau of Overseas Buildings Operations
subsections: section_title: Construction and craftsmanship
paragraphs: Intent  Embassies and consulates have two essential purposes: to be safe, secure, functional, and  inspiring places for the conduct of diplomacy, and to physically represent the U.S. government to  the host nation.   The site and location of an embassy have practical as well as symbolic implications. OBO will  develop sites that best represent the U.S. government and its goals, and enhance the conduct of  diplomacy. Whenever possible, sites will be selected in urban areas, allowing U.S. embassies to  contribute to the civic and urban fabric of host cities.   OBO will evaluate designs on the basis of their success in skillfully balancing requirements, and on  how well the design represents the United States to the host nation. Designs are to be functionally  simple and spatially flexible to meet changing needs and be enduring over time. An official embassy  style will be avoided. Designs will be cost-effective. Each design will be responsive to its context,  to  include the site, its surroundings, and the local culture and climate. The designs will make use of  contextually appropriate and durable materials and incorporate the latest in security and safety  features. The grounds should be functional and representational spaces. They will be sustainable,  include indigenous plantings, and incorporate existing site resources, such as mature trees.  The facilities will incorporate advanced methods, systems, technologies, and materials appropriate  to the facility and local conditions, including the site, climate, natural hazards, security, and the  practical reality of construction, maintenance, and operations in the host nation.  The safety and security of staff and visitors is paramount. Designs and construction will meet or  exceed all security and safety standards and specifications.   Buildings and grounds will incorporate sustainable design and energy efficiency. Construction,  maintenance, and operations practices will be sustainable.  OBO will hire leading American architects and engineers. Selection will be based on the quality of  their design achievements and portfolio of work. The selection methodology will be open,  competitive, and transparent.  Construction professionals will be engaged throughout the process to ensure the best possible  design and implementation. OBO is committed to selecting the most qualified building contractors  with a record of delivering high quality projects.
Operations and maintenance  Operations and maintenance professionals will be engaged throughout the design and construction  Historically, architecturally, or  culturally significant  properties and collections  process. Buildings and sites will be economical to operate and maintain and will utilize equipment  and materials that are durable, dependable, and suitable. Designs will be based on life-cycle  analysis of options that take into account longterm operations and maintenance.   Embassy buildings and grounds are an opportunity to showcase the best of American and host  nation art and culture. OBO is committed to integrating art into its facilities such that each property  will be both an individual expression of Excellence and part of a larger body of work representing  the best that America’s designers and artists can leave to later generations.  OBO is committed to preserving the State Department’s historical, cultural, and architectural legacy.  The Secretary of State’s Register of Culturally Significant Property is the official listing of important  diplomatic architecture overseas and properties that figure prominently in our country’s international  heritage.
section_title: Appendix VI: Approaches to Achieve Excellence; U.S. State Department, Bureau of Overseas Buildings Operations
paragraphs: Appendix VI: Approaches to Achieve  Excellence; U.S. State Department, Bureau of  Overseas Buildings Operations Approaches to Achieve Excellence; U.S. State Department, Bureau of Overseas Buildings  Operations (OBO)  Strategy General Intent  Holistic approach to project Project teams include all key stakeholders such as users, tenant agencies, and OBO disciplines, as  delivery well as members of the architectural, engineering, and construction contractor teams.  Information technology  (IT)   OBO’s IT platform integrates and makes available all project information, promoting effective review,  Architect and engineer (A-E)  selection  Design process  Construction contractor  selection  Best value contract award  communication, and decisionmaking during project development, construction, maintenance and  operations.  OBO uses either Design/Build or Design/Bid/Build. Neither delivery method is a default. Context,  complexity, construction environment, and urgency are evaluated when selecting a method for each  project.  OBO recognizes the representational and symbolic importance of site location. OBO has revised  site scoring criteria to acquire sites in urban areas. OBO considers redevelopment of strategically- located U.S. government owned sites.  OBO contracts with the most talented A-E firms, whether long-established or emerging new firms.  The selection process focuses on the portfolio of work, team members, and past performance.  OBO effectuates high-quality design through design processes such as on-site design charrettes,  on-board working sessions, constructability and maintainability reviews, senior management  approvals, and peer reviews.   The Guiding Principles outline the fundamental design goals of all of our projects. These include the  integration of purpose, function, security, safety, flexibility, sustainability, maintainability, and art.  OBO uses sustainability principles and life-cycle cost analysis to ensure that facilities provide the  lowest overall long-term cost of ownership, consistent with quality and function.   OBO is working to expand the pool of contractors and reach out to new emerging firms to promote  competition and ensure the best outcome.  OBO is using the Best Value method, which includes factors such as past performance and team  qualifications, as well as consideration of lifecycle costs in the evaluation process.
Early contractor involvement  OBO involves construction contractors in early stages, particularly on long-term and complex  projects, to ensure the best outcome and reduce risk.
Operations and maintenance  Reference guides and training programs are developed for each major project. The guide includes  Revised architectural and  engineering design  guidelines  Recognizing  Excellence  information such as design intent, systems information, maintenance requirements, and  troubleshooting. This ensures that facilities are operated and maintained properly and that future  modifications to the building are in keeping with the original design intent.  A Guide to Excellence in Diplomatic Facilities has been released on the OBO website  (www.state.gov/obo). The guide is comprehensive and highlights how Excellence goals and  priorities will be achieved in each phase of a project. The Guide is a basic “how to” manual.  Design requirements have been revised to support Excellence. The requirements emphasize high- performance buildings, flexibility and best design practices, while moving away from a fixed solution.
An Excellence awards program is in development. The program will recognize projects that  exemplify Excellence.
section_title: Appendix VII: Timing of Industry Advisory Reviews within the Bureau of Overseas Buildings Operations Design Process
section_title: Appendix VIII: Comments from the U.S. Department of State
section_title: Appendix IX: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: Michael J. Courts, (202) 512-8980 or courtsm@gao.gov  David J. Wise, (202) 512-5731 or wised@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Leslie Holen (Assistant Director,  International  Affairs and Trade), Michael Armes (Assistant Director,  Physical Infrastructure), David Hancock, John Bauckman, Eugene Beye,  Melissa Wohlgemuth, and Elisa Yoshiara made key contributions to this  report. Technical assistance was provided by David Dayton, Jill Lacey,  Alex Welsh, and Neil Doherty.