Federal Buildings: GSA Should Establish Goals and Performance Measures to Manage the Smart Buildings Program
Released: 2018-01-30T00:00:00

Letter

Background

GSA maintains custody and control of real property for many civilian federal agencies and has a large portfolio of federally owned and leased properties that GSA rents to its federal agency customers. It is responsible for approximately 1,600 federally owned buildings, and the agency generally provides operations and maintenance services for building systems—such as heating, cooling, and lighting systems—used in building operations. According to GSA officials, their federally owned smart buildings are managed by a GSA building manager who oversees a private operations and maintenance services contractor.

According to GSA officials, the agency began implementing what would become its smart buildings program around 2005 in response to numerous federal policies aimed at improving federal building energy and environmental management. These officials told us that the smart buildings program includes two key technologies: advanced utility meters and a computer software program known as “GSAlink.” According to GSA officials, outfitting buildings with these technologies allows for more precise monitoring of energy use and equipment operations in these buildings, and was initially based on the use of advanced utility meters to meet federal mandates. Later, this concept was expanded to include use of analytics, through GSAlink, aimed at reducing energy consumption and increasing the efficiency of operations and maintenance activities. According to GSA officials, GSA’s smart buildings use these technologies to connect and monitor multiple pieces of building equipment, such as heating and air conditioning system components. Further, according to these officials, the program is intended to achieve efficiencies in energy use and in operations and maintenance activities while also providing a comfortable workplace potentially conducive to improved tenant productivity. As GSAlink and advanced meters are Internet-connected, GSA officials told us that they implemented protections that are intended to help mitigate potential cyberattacks, including using firewalls.

Advanced Utility Meters: In response to energy reduction and advanced metering requirements established in the Energy Policy Act of 2005—as well as subsequent amendments and an Executive Order—GSA began installing advanced meters in its federally owned buildings starting around 2005. Internet-connected advanced utility meters measure utility use in real-time, which GSA officials told us allows GSA’s building managers to identify opportunities to reduce energy use or anomalies that contribute to energy waste. For example, GSA officials said that advanced utility meters can be used to monitor energy consumption patterns and detect lights or other building systems being used after normal business hours. According to a senior GSA official, GSA currently has 675 advanced meters installed in the agency’s approximately 1,600 federally owned buildings.

GSAlink: GSA officials told us that GSAlink is a computer software program that collects and analyzes data from advanced meters— including gas, electric, and water meters—and from a facility’s “building automation system” and uses this information to alert building staff to potential problems. Further, GSA officials said that GSAlink allows them to identify building problems that occur over time that may not be readily observable through the building automation system, which generally presents information to building personnel on how a building system is operating in real-time, not over a longer time frame. For example, GSA officials told us that GSAlink can collect data on the temperature and pressure of chilled water that is being circulated through a building’s cooling system and identify equipment that is operating outside of normal parameters or normal business hours when a building automation system may not be actively monitored. If GSAlink detects a potential issue, GSA officials told us the software creates a record so that building maintenance staff can investigate and remedy that issue. GSA building managers as well as GSA staff at the regional and national levels told us they can log in to GSAlink to check on the status of building system issues. According to GSA officials, the contract for GSAlink was awarded in 2012 and GSAlink is currently in use in 81 buildings, with at least one GSAlink- equipped building in each of GSA’s 11 regions. A senior GSA official told us that eighty of these buildings are also equipped with advanced meters. Further, in September 2017, this official told us that GSA contracted to equip 4 additional buildings with GSAlink. According to GSA officials, GSA generally plans to limit installation of GSAlink in additional buildings until more is learned about using the technology in the buildings in which it is currently installed.

Figure 1 illustrates an example of a GSA smart building that includes advanced meters, GSAlink, and the building systems monitored by these technologies.

Limited Quantified Information Exists on the Costs and Benefits of Key Smart Buildings Program Technologies

The Smart Buildings Program’s Installation Costs Are Affected by Building Characteristics and Can Be Difficult to Quantify

According to GSA officials, the approximate cost of equipping a building with smart building technologies ranged from between about $48,000 to $155,000. This includes costs for installing: advanced utility meters (approximately $25,000 to $55,000), and

GSAlink (approximately $23,000 to $100,000).

The cost of installing GSAlink depends on the condition of the building automation system to which GSAlink is connected as well as the number of individual building components (e.g., chilled water pumps, cooling tower fans, thermostats) to be monitored by GSAlink. GSA officials anticipate that advances in system architecture and reduced software licensing costs will lower the cost of future installations. For example, a senior GSA official told us in October 2017 that the cost to install GSAlink in four additional buildings—the most recent buildings in which GSAlink was installed—ranged between $23,000 and $25,000.

In addition, GSA is undertaking a broader effort to upgrade building automation systems in its buildings to enable these systems and connected applications, such as GSAlink, to operate on GSA’s protected information technology network. According to GSA officials, GSA can only install GSAlink in buildings whose building automation system operates on GSA’s protected network. To date, GSA has upgraded building automation systems to operate on the agency’s protected network in approximately 400 buildings. GSA officials told us that the cost of these upgrades has varied by building and depends on several factors, including the size of the building, the complexity or condition of its building automation system, and its age. According to GSA officials, upgrading building automation system components to enable them to operate on the protected network has cost approximately $90,000 per building, on average. However, in some cases, these costs can be much higher; integrating older systems in larger buildings has cost up to $3 million, according to GSA officials.

Further, according to GSA officials, accurately calculating smart building implementation costs can be difficult because GSA typically installs key technologies—that is, advanced meters and GSAlink—and makes upgrades necessary to install GSAlink in selected buildings incrementally, sometimes as part of other capital improvement projects. For example, the American Recovery and Reinvestment Act of 2009 and annual appropriations have provided funding to GSA for energy and conservation measures, including the purchase and installation of advanced meters.

GSA Has Taken Steps toward Assessing Benefits of the Smart Buildings Program, but Efforts to Quantify Benefits Have Been Limited

GSA officials we interviewed at the central office, regional, and individual building levels identified perceived operational benefits from implementing the smart buildings program, including that it (1) enables them to identify problems with building equipment or system operations more quickly and more thoroughly and (2) allows for their greater oversight of operations and maintenance services contractors relative to other GSA buildings. For example, according to GSA regional staff we spoke to, both advanced meters and GSAlink could detect if the cooling system was operating when tenants were not occupying the building, thereby allowing the building managers to adjust operations to avoid unneeded energy use and wear on the cooling system equipment. Regarding contractor oversight, GSA building managers stated that GSAlink allows the agency to better monitor operations and maintenance contractors’ performance, potentially yielding a better-run building with lower operations and maintenance costs. For example, GSA officials described how the analytic capability of GSAlink might allow building managers to precisely identify and address a problem with a building before that problem is noticed by tenants. This may result in, for example, a reduction in the number of maintenance service requests from tenants and contribute to lower building operating costs. In addition, GSA officials told us that GSAlink allows GSA building managers to confirm the information operations and maintenance services contractors present to them on the status of issues identified by GSAlink. Further, according to these officials, GSAlink allows building managers to monitor contractor compliance with GSA’s requirement that contractors address building issues identified by GSAlink within 30 days, thereby giving GSA officials closer oversight of contractor performance.

GSA has taken some steps in the past to quantify the benefits associated with the smart buildings program. While those efforts have identified benefits, they have had some limitations. For example, in 2009—after having begun installing advanced meters but before installing GSAlink— GSA attempted to forecast benefits of the smart buildings program by commissioning a business case analysis. The business case concluded that GSA’s energy and operating costs could be reduced by a smart buildings program and that such a program would pay for itself in 1.7 years based on combined energy and operational savings. However, this business case’s estimates of the program’s benefits have limited usefulness for evaluating the current program because this study took place before the program was fully implemented and did not account for constraints affecting building operations. For example, a senior GSA official told us that GSA’s operations and maintenance service contracts are generally for multiple years at a fixed price, calling into question whether operational cost savings can be realized to achieve payback within the time frame estimated by the study.

In addition, GSA’s service contractor developed an application within GSAlink that automatically estimates the costs that would be avoided by addressing each type of fault that GSAlink identifies. According to GSA officials, these estimates are imprecise and do not reflect actual avoided costs, which thereby precludes their use in quantifying program benefits. However, according to these officials, these estimates can be used to compare the relative benefits expected to be achieved by addressing identified faults and to prioritize maintenance and repair actions. GSA officials told us that they took steps in June 2017 to improve the accuracy of avoided cost estimates produced by this application, for example, by enabling adjustments to account for differences in weather conditions and building size, and plan to continue their efforts to adjust and refine this tool.

In a separate study in October 2016, GSA—in collaboration with researchers at Carnegie Mellon University—analyzed the energy use changes associated with both capital upgrades and operational initiatives, including the use of smart building technologies. Capital upgrades include actions such as installing new energy-efficient building systems and equipment, whereas operational initiatives include, among other things, changes to building operations based on the analysis of advanced meter and GSAlink data. While the researchers concluded that the use of advanced meter and GSAlink data led to reductions in energy use, the researchers found that GSA’s utility consumption records were incomplete and that GSA records of capital upgrades often do not include key details, such as project start or completion dates, to indicate when GSA would have received the benefit derived from the capital project. This lack of complete data adds to the difficulty of estimating the reduced energy consumption attributable to specific factors, including use of advanced meters and GSAlink.

GSA Does Not Have Documented, Clearly Defined Performance Goals or Measures to Help It Manage the Smart Buildings Program

We have previously found that results-oriented organizations set performance goals to clearly define desired program outcomes and develop performance measures that are clearly linked to the performance goals. Program goals communicate what results the agency seeks and allow agencies to assess or demonstrate the degree to which those desired results are achieved. Performance measures also show the progress the agency is making toward achieving program goals. We have previously reported that performance measurement gives managers crucial information to identify gaps in program performance and plan any needed improvements.

GSA has not documented the smart buildings program’s goals, contrary to leading practices we identified in our prior work, which call for program goals to clearly define desired program outcomes. GSA officials verbally described to us broad goals for the smart buildings program: (1) reducing energy consumption, (2) generating operations and maintenance cost savings, and (3) creating a comfortable work environment conducive to improved tenant productivity. However, GSA has not documented these goals—for example, in the agency’s performance plan or in other program documents. GSA officials could not provide a reason for why the agency has not documented the smart buildings program’s goals.

Further, because GSA has not clearly defined its verbally expressed goals, it cannot demonstrate progress in achieving them. This lack of clearly defined goals is contrary to federal internal control standards, which state that agency management should define objectives in measurable terms so that performance toward those objectives can be assessed. GSA could potentially measure progress toward its stated smart buildings program goals of reducing energy consumption and generating operations and maintenance cost savings, if data were available to do so, as these goals seek to identify changes in quantifiable outcomes, specifically energy use and cost savings. However, GSA officials said that the agency cannot measure progress toward the stated goal of improving tenant productivity and comfort because of the subjective nature of individual tenant preferences, such as for office temperatures. This subjectivity is consistent with statements from the industry stakeholders we spoke with, who also said that identifying the existence of a causal relationship between a building’s environment and the productivity of its inhabitants is challenging. For example, an industry stakeholder we spoke to told us that different building occupants have different temperature or ventilation preferences and may accordingly be the most productive at different ambient temperatures, making it challenging to determine a building’s optimal temperature. Without documented, clearly defined goals, it will be challenging for GSA to determine what type of evaluative information it will need to monitor the progress of the smart buildings program.

In addition, contrary to the leading practices we have identified in our previous work, GSA has not developed performance measures for the smart buildings program. According to these leading practices, performance measures allow for an assessment of progress toward achieving goals by including concrete, objective, and observable ways to measure the program’s performance and compare this with the program’s expected results. Further, federal internal control standards call for federal program managers to use quality information to achieve that program’s objectives and make informed decisions. However, GSA lacks quality information that can be used to measure program performance. As discussed in the previous section, GSA’s efforts to quantify the smart buildings program’s benefits, including energy reductions and cost savings, have been limited because GSA has had difficulty in compiling data that would allow it to do so. For example, GSAlink’s calculation of avoided costs estimated to be achieved by addressing identified faults is useful for prioritizing maintenance actions but not for measuring program performance because, according to GSA officials, the estimates lack precision and relation to actual costs. In addition, GSA’s October 2016 study on energy use reductions attributable to the program faced problems owing to incomplete records on utility consumption and capital upgrades. While we recognize that determining what data can be collected in a cost-effective manner and can be used to measure the performance of the smart buildings program may be difficult, without such data and measures, GSA lacks the ability to determine the program’s progress and make informed decisions about its current and future operations.

GSA Faces Some Challenges in Implementing Smart Building Technologies and Is Taking Steps to Mitigate Them

GSA Is Taking Actions That May Mitigate Challenges Related to Cybersecurity

GSA faces cybersecurity challenges to its buildings, but is taking steps intended to mitigate these challenges. According to GSA officials, advanced meters and GSAlink operate in conjunction with Internet- connected building automation systems on the protected GSA information technology network. GSA regional staff and industry stakeholders we interviewed stated that cybersecurity presents challenges to those operating smart building technologies, including GSA. Specifically, because these building automation systems are connected to the Internet, they provide a potential pathway for cyberattacks on GSA’s network. According to our prior work, this connectivity could compromise security, hamper GSA’s ability to carry out its mission, or cause physical harm to GSA’s facilities or their occupants.

GSA has taken several actions that are intended to help mitigate cybersecurity challenges to its buildings, including those that affect the smart buildings program:

GSA has instituted policies and procedures addressing cybersecurity threats and known vulnerabilities in its building systems. In December 2015, GSA published an information technology security policy, defining the roles and responsibilities of GSA staff and establishing controls to ensure compliance with federal regulations, laws, and GSA directives. For example, this policy defines the role of the Federal Government Authorizing Official whose responsibilities include ensuring that monthly operating system scans, database scans, and web application scans are performed and that all vulnerabilities identified are resolved.

According to a GSA senior official, under GSA’s Building Monitoring and Controls Program, which provides the infrastructure support needed to connect a building to GSA’s network, GSA is taking steps to mitigate the effects of potential external cyberattacks by moving building automation systems of GSA-controlled buildings away from public networks to GSA’s secured network. GSA officials told us that there are currently approximately 400 federally owned buildings on GSA’s secured network, which includes the 81 buildings equipped with GSAlink. According to GSA officials, a building automation system must be on GSA’s secured network before GSAlink can be installed.

According to GSA officials, GSA also performs regular assessments to validate that GSAlink system controls comply with relevant statutes, such as the Federal Information Security Management Act of 2002, National Institute of Standards and Technology security standards, and GSA policies and procedures. In December 2014, we reported on GSA’s efforts to address cyber risks in federal buildings in compliance with relevant statute and guidance, finding that GSA had not conducted security control assessments for all of its systems in about 1,500 federally owned facilities. We recommended that GSA assess its building control systems in a manner fully consistent with federal law and related implementation guidelines. GSA has since implemented this recommendation.

According to GSA documentation and officials, GSA conducts regular vulnerability scanning of the equipment and systems involved in the smart buildings program. For example, according to GSA regional staff, a recent vulnerability in the GSA system that manages maintenance requests was identified by GSA central office and was remedied through a software upgrade.

GSA Is Taking Actions That May Mitigate Challenges with Stakeholder Support

GSA faces smart building technology implementation challenges related to the limited technological proficiency of or lack of buy-in from some GSA building managers and operations and maintenance services contractors, but the agency is taking steps that are intended to engage these stakeholders and ensure they are learning to use the smart buildings program’s technologies. GSA regional staff acknowledge that there can be inconsistencies among building managers and operations and maintenance services contractors in terms of their familiarity and comfort with using computers and computer-based analytical tools. According to GSA officials, GSAlink proficiency and adoption varies by building and as such, some buildings may obtain greater benefits from the system than others. A lack of proficiency among building managers in smart building technologies not only affects GSA, but is also an industry-wide concern, according to industry stakeholders we interviewed. Industry stakeholders we interviewed stated that operations and maintenance services contractors are generally not well trained on smart building operations or the differences between managing a smart building and managing a traditional building.

GSA regional staff and GSAlink’s support contractor we interviewed also identified operations and maintenance services contractors’ limited buy-in to the smart buildings technologies as a challenge affecting implementation of the program. According to GSA officials, this limited buy-in to the smart buildings technologies could potentially lead to loss of support for the program among operations and maintenance services contractors, posing a risk to the program’s successful implementation. GSA officials, regional staff, and GSAlink’s support contractor acknowledge it is important to demonstrate how GSAlink, for example, can make the operations and maintenance services contractors’ jobs easier. According to GSA officials, if GSAlink can help a building’s systems operate more efficiently, that improvement should result in less unscheduled maintenance and fewer work orders for the contractor. Additionally, industry stakeholders we interviewed suggested that operations and maintenance services contractors do not currently have a stake in whether a smart buildings program is successful.

According to those we interviewed, GSA has taken several actions that are intended to help address these challenges:

GSA officials and regional staff told us that GSA provided initial training to building managers and operations and maintenance services contractors when GSAlink was first installed. According to GSA officials, refresher training is available online through recorded training sessions. Additionally, GSA regional staff told us that knowledgeable GSA staff provide training to newly hired staff as needed.

GSAlink’s support contractor staff told us that they lead regularly scheduled teleconferences with each smart building’s staff either monthly or quarterly depending on each building’s needs. At these meetings, the support contractor remotely accesses GSAlink data for a particular building to discuss the status of GSAlink notifications of building system issues and recommend adjustments to building equipment or systems to ensure optimal operations. GSA regional staff we spoke with stated that this meeting serves as a form of training and helps educate participants on how to use GSAlink.

To ensure that building personnel are using smart buildings technologies, GSA officials told us that GSA’s central office monitors a key performance indicator requiring GSA building managers and operations and maintenance services contractors to address all GSAlink notifications of building system issues within 30 days. According to GSA officials, GSA central office and regional staff also have the ability to remotely monitor advanced meter and GSAlink data for individual buildings.

According to a senior GSA official, new operations and maintenance services contracts will expressly require contractors to use smart building technologies as part of their efforts to optimally operate GSA buildings.

Conclusions

According to GSA officials, the agency’s smart buildings program is intended to allow its staff and contractors to more efficiently manage energy consumption and operations and maintenance actions aimed at promoting cost-efficient operation of building systems and creating a comfortable work environment for tenants in GSA’s buildings. Given GSA’s recent decision to expand the use of GSAlink technology, it is important that the agency be able to determine whether use of the technology achieves these intended results. However, without documented, clearly defined goals, performance measures linked to those goals, and quality information to measure progress, GSA is limited in its ability to make informed decisions about the smart buildings program’s current or future operations as it develops plans to enlarge the program to serve a greater proportion of its buildings portfolio. As a result, GSA risks continuing to expend resources on a program that the agency cannot demonstrate is meeting its intended objectives.

Recommendations for Executive Action

We are making the following two recommendations to GSA: The Administrator of the General Services Administration should establish clearly defined goals and related performance measures for the smart buildings program. (Recommendation 1)

The Administrator of the General Services Administration should identify and develop data that can be used to measure progress in achieving the smart buildings program’s goals. (Recommendation 2)

Agency Comments

We provided a draft of this report to GSA for comment. In its written comments, reproduced in appendix II, GSA stated that it concurred with our recommendations and is developing a plan to address them. In addition, GSA clarified that the agency has been upgrading building automation systems across its buildings inventory for a variety of reasons, to include providing needed safeguards to comply with GSA’s information technology security protocols. GSA also provided information on the methodology used and results reported in its October 2016 study on energy savings realized from combined investments in advanced metering and GSAlink.

We are sending copies of this report to the appropriate congressional committees and the Administrator of the General Services Administration. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-2834 or rectanusl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Organizations Contacted

Federal Government

GSA Smart Buildings – Washington, DC

GSA Smart Buildings – San Francisco, California

Industry Stakeholders

Appendix II: Comments from the General Services Administration

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Lori Rectanus, (202) 512-2834 or rectanusl@gao.gov.

Staff Acknowledgments

In addition to the contact named above, Michael Armes (Assistant Director); Daniel Paepke (Analyst in Charge); Edward Alexander, Jr.; Jenny Chanley; John de Ferrari; Peter Haderlein; Geoffrey Hamilton; Thomas Johnson; Nick Marinos; Malika Rice; Stephen Schluth; Elaine Vaurio; Jack Wang; Michelle Weathers; and Dave Wise made key contributions to this report.