Public-Safety Broadband Network: Network Deployment Is Progressing, but FirstNet Could Strengthen Its Oversight
Released: 2020-01-27T00:00:00

Letter

Background

The FirstNet-AT&T network contract and its associated task orders define the requirements AT&T must meet. The contract currently involves five task orders, four of which relate directly to the network’s deployment.

Task orders 1 and 2 (actions complete). Required AT&T to develop and deliver individual network deployment plans for each of the 56 states, territories, and the District of Columbia (hereafter, states). The governor of each state had the opportunity to review the plan and opt in to allow FirstNet and AT&T to build the network in their state. All governors opted in by the applicable deadline. The result of this process was a state deployment plan that included state-specific commitments made by AT&T.

Task order 3 (actions ongoing). Requires AT&T to deploy, operate, and maintain the network’s “core” and all of its functions, and provide for the development of device and application ecosystems for the network. A network core consists of national and regional data centers and other elements that store, process, and secure network users’ traffic (activity), and interface with federal, state, and local networks. AT&T deployed the core in March 2018. The network uses the spectrum reserved for public-safety use (“Band 14”), as well as the spectrum that AT&T’s existing, commercial network operates on. When Band 14 spectrum capacity is not being used by public-safety users, AT&T can use the excess capacity for its non-public-safety, commercial-network users. As such, among the functions that task order 3 provides for are capabilities that allow prioritizing a public- safety user’s network access and traffic over other users and, when necessary, preempting other users altogether. These functions are commonly referred to as “priority and preemption.”

Task order 4 (actions ongoing). Requires AT&T to deploy the network’s Band 14 coverage in the states, including building the “radio-access network” in each state that connects to the network’s core and backhaul (which carries network users’ traffic) and fulfilling the state-specific commitments. Radio-access networks consist of cell towers, sites, and other elements that connect network-users’ devices to the network core. This task order also requires AT&T to provide 72 “deployable” cellular assets—meaning, transportable equipment (typically in a vehicle) that can provide additional network coverage when needed—dedicated solely for FirstNet network users. The task order also provides for access to at least 300 additional deployables in AT&T’s fleet.

The contract and task orders 3 and 4 outline a phased approach for deploying the network’s capabilities and coverage (in both non-rural and rural areas), with five “initial” operating-capability phases that build to a “final” operating capability expected in 2023, as well as ongoing performance, maintenance, and continuous improvement through 2042. As described further below, each phase provides for increased capabilities and coverage—and some outline goals for network user adoption—and AT&T must meet certain required milestones in each phase to receive payment for that phase from FirstNet. Figure 1 depicts the phased timeline for task orders 3 and 4.

The Band 14 spectrum on which AT&T is building the network is a key component that differentiates it from other commercial networks, as the network’s full capabilities and functionality are only available via Band 14. For example, certain high-power user equipment can transmit at stronger signals; this signal increase can only be done using the Band 14 spectrum. However, at its expected final operating capability, the network using Band 14 spectrum will not cover the entire country. Public-safety network users will also have access to the non-Band 14 LTE spectrum that AT&T uses for its existing, commercial network (with priority and preemption), though this spectrum does not have all the full capabilities of Band 14, as in the high-power user equipment example above. According to AT&T, when including this non-Band 14 spectrum, the network will cover 76.2 percent of the U.S. geographically and around 99 percent of the population. Network users are to also have access, by request, to deployable assets that can provide temporary coverage when needed, such as in remote and wilderness areas that will not have permanent coverage.

FirstNet’s Contractor Is Meeting, or Is on Track to Meet, All Nationwide Contractual Coverage and Adoption Milestones, but State- Level Progress Varies

AT&T Has Met the First Nationwide Coverage Milestone but Coverage Is Not Uniform across States

AT&T has met, and exceeded, the first required nationwide network- coverage milestone. According to FirstNet documentation, AT&T is required to meet certain coverage milestones in both non-rural and rural areas and by the end of March 2019, AT&T had met the requirement to provide at least 20 percent of the total expected Band 14 coverage in both non-rural and rural areas. The Band 14 coverage milestones that AT&T is contractually required to meet to receive payment increase each year through March 2023, when AT&T is to have completed 100 percent of the total expected Band 14 coverage. For example, by March 2021, the coverage milestones are 80 percent of the total expected Band 14 coverage in both non-rural and rural areas and by March 2022, 95 percent. Per the terms of the contract, prior to meeting the first milestone, AT&T provided initial coverage via its existing, commercial wireless network and made 72 deployables (such as mobile cell sites on trucks) available for network users. AT&T fulfilled the deployables requirement through a combination of deployables built specifically for network users and others allocated from AT&T’s existing fleet of deployables used for disaster relief.

Specifically, to complete the first coverage milestone, AT&T delivered Band 14 coverage in about 63 percent of the total square miles required by 2023 in non-rural areas, and about 21 percent of the total square miles required by 2023 in rural areas, according to FirstNet documentation. For meeting this milestone, FirstNet paid AT&T approximately $1.2 of the $6.5 billion. Since completing this milestone, AT&T has continued to expand coverage and, according to FirstNet officials, is also on track to meet the next coverage milestone (due March 2020) early, although FirstNet was in the process of completing final verification and validation activities as of September 2019.

AT&T constructed or “delivered” (i.e., these sites are all on-air) thousands of Band 14 cell sites to produce the level of coverage needed to meet the March 2019 milestone. Since then, according to FirstNet documentation, AT&T has continued adding Band 14 sites, delivering—as of July 2019— more than one-third of the total Band 14 cell sites planned for the entire network. AT&T may deliver these cell sites through a combination of constructing new sites, retrofitting existing AT&T sites, or acquiring or contracting with local providers, such as rural telecommunications carriers. Although FirstNet tracks the status of planned cell sites (such as which sites are undergoing environmental policy review or are currently operational, or on-air), cell sites are not an explicit part of the contractual coverage milestones required for AT&T to receive payment. That is, AT&T’s payment is not contingent upon getting a certain number, type, or location of cell sites on-air, but rather the amount of coverage (in square miles) provided on a nationwide level by these sites.

While AT&T met the first coverage milestone and has delivered more than a third of the planned cell sites nationwide, AT&T also has state- specific commitments. These commitments or targets, like the delivery of sites, are not explicit contractual payment milestones. AT&T and the states negotiated the commitments during the state opt-in process, and AT&T delineated them in the state plans. For example, among states in our review, AT&T made commitments regarding the number of Band 14 cell sites, including new cell sites, and future coordination with state, local, or tribal authorities to discuss governance or priority coverage areas, among other things.

According to our analysis of FirstNet documentation, progress toward meeting state-specific coverage commitments has varied. For example, among our case-study states as of July 2019, AT&T’s progress meeting the total coverage commitment in non-rural areas ranged from approximately 20 percent complete in one state to nearly 100 percent in others. In comparison, AT&T’s coverage progress in rural areas ranged from about 14 percent complete in one state to about 91 percent in another. Likewise, AT&T’s progress meeting state-specific commitments for delivery of Band 14 cell sites has varied across states. For example, in our case-study states, AT&T delivered between 9 and 71 percent of the total committed Band 14 cell sites as of July 2019.

According to FirstNet documentation and officials, variances in state progress are allowable, as the contractual payment requirements focus on outcomes related to nationwide milestones. FirstNet documentation specifies that if the nationwide payment milestone was met, regardless of the amount of coverage that was deployed in a specific state, FirstNet deemed AT&T to have fulfilled that phase for all states. Moreover, FirstNet officials explained that multiple factors can contribute to delays or variance in progress across states, including natural and man-made disasters, subcontractor issues that AT&T must work through with local partners, and technical challenges common to cellular networks, such as degraded performance due to mixing of radio-frequency signals. Furthermore, FirstNet officials explained that AT&T has the first 5 years of the contract to meet all commitments made to the states.

AT&T Is on Track to Meet the First Nationwide Adoption Milestone, with Adoption among Some Users and States Outpacing Others

AT&T is on track to meet the first adoption milestone, which is to have a certain number of devices connected or subscribed onto the network (“device connections”) by the end of March 2020. FirstNet uses device connections as a proxy for adoption and has set or “forecasted” monthly targets that build up to the nationwide connections expected by March 2020. Our analysis of FirstNet documentation indicates that AT&T is making progress in meeting the monthly nationwide targets leading up to March 2020. Specifically, we found that AT&T was at approximately 165 percent of the July 2019 target. See figure 2 for a comparison of actual nationwide device connections versus the forecasted targets by month through July 2019.

Furthermore, while AT&T must meet the nationwide device-connection milestone to receive payment for the phase ending March 2020, the targets are to be prorated depending on the month that AT&T meets the corresponding nationwide coverage requirement. Thus, if AT&T meets this requirement early (i.e., before March 2020), then the required adoption milestone is to be reduced accordingly. For example, if AT&T completes the coverage milestone in September 2019, then it would be required to meet a corresponding adoption target for that timeframe.

While AT&T is on track to meet the nationwide, forecasted device- connection targets that serve as the payment milestone, our analysis found that there is variation in who is adopting the network. The targets are broken out by device connections associated with “primary” versus “extended-primary” users in different states. FirstNet defines primary users as those in the law-enforcement, fire, and emergency medical- services disciplines, whereas extended primary encompasses a myriad of other types of public-safety entities. For example, according to our analysis of FirstNet documentation, there are extended-primary users from transit agencies; public-utility and tow-truck companies; school districts; a state child-protective-services agency; airports; and television- media news outlets. Nationwide, with regard to primary users, AT&T was at 196 percent of the July 2019 target. For extended-primary users, AT&T was at approximately 106 percent of the nationwide target. These device connections are also distributed amongst the different types of public- safety entities. For example, for primary device connections, AT&T was at more than twice the forecasted nationwide target for law enforcement, as of July 2019.

Our analysis also shows that there is wide variance in where adoption is occurring. Specifically, we found that AT&T is exceeding the device connection targets forecasted in certain states but lagging in others. Among our case-study states as of July 2019, for example, device connections for primary users in one state were more than 5 times the target, whereas in another state, AT&T had met only 33 percent of the target by July 2019. Adoption by extended-primary users among our case-study states also varied, with one state at 3 times the target compared to only 7 percent of the target met in another.

Many types of devices are connected to the network and users’ experiences with network performance can vary based on the specific device they use. According to FirstNet documentation as of April 2019, 93 device types, 47 of which are Band-14 capable, were vetted and published on the list of devices certified for use on the network maintained by Commerce’s National Institute of Standards and Technology. Our analysis found that a variety of devices and device models are being used on the network, including smartphones, mobile hotspots, trunk modems, laptops, and tablets. As of July 2019, the most prevalent type of device was smartphones. FirstNet has acknowledged that user experiences on the network may vary depending on the type and model of device. Some public-safety officials we interviewed described inferior experiences on certain types or models of devices. In at least one case, AT&T worked with the public-safety entity to address identified device performance issues.

Aside from device connections, FirstNet also tracks and has reported— via press releases, board presentations, and its most recent annual report to Congress—on the number of public-safety entities that have started using the network. For example, in April 2019, FirstNet reported to Congress that more than 7,000 public-safety agencies were using the network. This number represents agencies with at least one device connection, which may indicate piloting of the network. For example, one agency we interviewed had only about 2 dozen of its approximately 1,300 total devices on the network. Similarly, officials from multiple other public- safety agencies explained they were in the piloting phase (i.e., testing a small number or types of devices to gauge network performance) and that they were using or would continue to use another carrier for broadband services to ensure effective redundancy and emergency planning. According to FirstNet officials, AT&T provides the count of public-safety agencies at periodic program-review meetings and documents it in a required contract deliverable. We analyzed this deliverable and were able to approximate FirstNet’s reported numbers.

Many FirstNet Oversight Mechanisms Align with Key Practices, but Weaknesses in Some Mechanisms Limit Their Effectiveness

FirstNet’s Approaches to Contract Oversight Generally Align with Key Practices

FirstNet employs a variety of mechanisms to manage and oversee AT&T’s deployment of the network and monitor contract performance. We found that many of FirstNet’s approaches to managing and overseeing AT&T’s network deployment and contract performance generally align with the key contract-oversight practices identified in federal acquisition regulations and other government, academic, and industry guidance on contract oversight that we reviewed, as shown in table 1.

We analyzed the key performance indicators and other documentation related to all 46 quality assurance elements that FirstNet monitors as of April 2019 and found that AT&T’s performance was rated as “excellent” in over half of these elements but “unsatisfactory” in almost a quarter. Regarding the number of unsatisfactory ratings, FirstNet officials stated that these ratings did not raise concerns given where AT&T was in the deployment lifecycle at the time of our review. That is, the rating may measure performance on an item that was not yet contractually due. For example, AT&T cannot achieve an excellent rating for certain elements that relate to coverage deployment until it is closer to the network’s final operating capability, expected in March 2023. Relatedly, according to FirstNet documentation as of April 2019, FirstNet had issued only one corrective action report since awarding the contract. According to FirstNet officials at the time of our review, although FirstNet has rejected or requested corrections to some items submitted by AT&T, no other concerns have risen to this level because they have been successful in resolving issues at lower levels first.

FirstNet’s oversight activities leading up to the March 2019 coverage milestone were the first wherein it had to validate AT&T’s delivery of Band 14 coverage. FirstNet’s methodology for doing so included verifying AT&T’s prediction of the signal strength at which the necessary throughput—or, capacity, the amount of data transported successfully in a given time period—would be achieved, and reviewing AT&T’s lab and field tests. FirstNet then engaged in a process to verify the validity of AT&T’s coverage-prediction maps to ensure they were an acceptable representation of coverage in the field. Finally, FirstNet confirmed that the on-air coverage as compared to the expected total coverage at the network’s final operating capability met the contractual requirement. FirstNet’s methodology did not include conducting its own coverage tests in the field. According to FirstNet officials, FirstNet does not perform independent verification of network coverage in the field because FirstNet officials believe the contract provides an appropriate level of detail within the contractual deliverables and supporting information that is used to validate and verify the coverage milestones.

Some FirstNet Oversight Mechanisms Have Weaknesses That Limit Their Effectiveness

While many of FirstNet’s contract-oversight mechanisms generally align with key practices, we found that some have weaknesses that limit their effectiveness. Specifically, FirstNet lacks: (1) a reliable master schedule to review, (2) communication with relevant stakeholders regarding contract oversight, and (3) meaningful information on end-users’ satisfaction to gauge performance quality.

Reliable Master Schedule

Key practices for contract oversight call for tracking the contractor’s performance and progress toward the expected schedule. Furthermore, GAO’s Schedule Assessment Guide identifies 10 best practices associated with effective scheduling, and they are grouped into 4 characteristics of a reliable schedule—comprehensive, well-constructed, credible, and controlled. The contract cites this guide when detailing the schedule’s requirements.

As described above, AT&T must provide a current master schedule to FirstNet monthly. However, we found that FirstNet’s use of the schedule AT&T provides is limited because, based on our assessment, it only partially or minimally meets the characteristics of a reliable schedule, as shown in table 2 and described further below.

Comprehensive. We found that the schedule did not reflect all of the work to be performed, precluding a comprehensive view of the entire program. For example, although a master schedule should be a comprehensive plan of all government, contractor, and subcontractor work that must be performed to complete the project, the schedule did not capture all government (e.g., FirstNet) activities or cover the entire contract period. Our schedule guide notes that management should be aware of how long government activities take because they often have a clear effect on schedules. An integrated master schedule should reflect all efforts necessary to successfully complete the program. Failing to include all work for all deliverables, regardless of whether they are the government’s responsibility or the contractor’s, can hamper program members’ understanding of the complete plan. Further, our analysis showed that there was a 1:1 detail-to-milestone ratio, meaning there was 1 detail activity for every milestone in the schedule, which is a low level of planning detail. Activities contained in the schedule did not always have manageable or reasonable durations; for example, over 50 percent of remaining activities had durations greater than 2 standard working months, with 25 percent of those having durations greater than 1 year. Our schedule guide notes that, for a schedule to provide a more accurate view of progress, longer activities should be broken down into smaller efforts where possible. While some of these activities had long durations because FirstNet expects AT&T to plan them in the future, some were not designated as such and had no other noted justification. Moreover, the schedule did not show any resources (i.e., labor, materials, travel, facilities, equipment, etc.). Our schedule guide also notes that resources must be considered in the creation of a schedule because their availability directly affects an activity’s duration, and a schedule without resources implies their unlimited supply and availability.

Well-constructed. We found that the schedule had a high number of date constraints and an unreasonable amount of total float (or slack). For example, 60 percent of remaining activities and milestones in the schedule had “start-no-earlier-than” constraints. These date constraints confine the schedule by preventing tasks from starting earlier even if predecessor activities are completed ahead of schedule, which prevent the constrained activities from taking advantage of possible savings being introduced by predecessor activities. Our schedule guide recommends minimizing and justifying (in documentation) date constraints because they override the schedule’s logic and restrict how planned dates respond to accomplished effort. Schedules with constrained dates can portray an artificial view of the program and begin to look more like calendars than schedules. Moreover, over 50 percent of remaining activities had total float greater than 2 standard working months, with the average being over 200 days. In other words, activities in the schedule can slip an average of 200 working days before delaying the project’s finish date. Our schedule guide notes that without accurate values of total float, the schedule cannot be used to identify activities that could be permitted to slip and thus release and reallocate resources to activities that require more resources to be completed on time. Finally, while we found that the schedule had continuous critical paths, there was not enough detail activities to track the work necessary to achieve project milestones.

Credible. We found that there was no risk analysis performed for the schedule. Our schedule guide notes that data about program risks should be incorporated into a statistical simulation to predict the level of confidence in meeting a program’s completion date; to determine the contingency, or reserve of time, needed for a level of confidence; and to identify high-priority risks. Additionally, our schedule guide notes that a schedule should be (1) “horizontally traceable,” meaning that it should link products and outcomes associated with other sequenced activities; such links are commonly referred to as “hand- offs” and serve to depict the relationships between different program elements and verify that activities are arranged in the right order, and (2) “vertically traceable,” meaning data are consistent between different levels of the schedule. Our analysis found that the schedule responded when significant delays were introduced into the planned activities; that is, when we tested the robustness of the schedule by extending activities’ durations, forecasted dates recalculated appropriately. However, as described above, we found that the schedule did not capture all activities or provide sufficient detail, meaning it cannot be fully traceable horizontally. We also found that, in general, the schedule provided good vertical traceability—that is, dates were traceable between status reports and the schedule. However, when we compared other reported information to the schedule, there were instances where this traceability was not the case. For example, one monthly report stated that baseline information was included for all tasks and milestones of a particular task order, but we found that the schedule did not in fact include this information. Vertical traceability provides assurance that the representation of the schedule to different audiences is consistent and accurate.

Controlled. We found that the schedule was updated regularly using actual progress and logic by trained AT&T personnel, with supporting documentation and review procedures. We also found that not all activities in the schedule had baseline dates. According to FirstNet officials, portions of the schedule are baselined on a rolling basis once the next requirements traceability matrixes are created. However, some activities with no baseline dates had already begun or been completed. Further, FirstNet officials stated that no “basis document” exists for the baselined schedule. Our schedule guide notes that a corresponding basis document is important because it explains the overall approach to the program, defines custom fields in the schedule file, details assumptions used in developing the schedule, and justifies constraints, lags, long activity durations, and any other unique features of the schedule. Furthermore, while AT&T was submitting schedule variance information, it covered only tasks that had been baselined, when the majority of activities in the schedule were missing baseline dates. Without formally established baseline-schedule start and finish dates to measure performance against, FirstNet is limited in how it can use the schedule to identify or mitigate the effect of unfavorable performance.

Overall, FirstNet officials said they are not concerned about the gaps in the AT&T master schedule for a variety of reasons. Namely, officials stated that FirstNet entered into a contract with AT&T that lays out specific milestones that AT&T must meet or it does not receive payment. Accordingly, they said that the summary level of detail is sufficient for FirstNet’s purposes, as AT&T’s program management office determines what activities are appropriate to track to meet those milestones and AT&T maintains its own, more detailed schedule. They further added that given the firm-fixed price nature of the contract, it is not practical or helpful for FirstNet to collect information on the resources for AT&T’s deliverables; if it takes AT&T 50 or 50,000 individuals to complete the requirement that decision is for AT&T to determine. As such, although the contract cites GAO’s schedule guide when detailing the schedule’s requirements, FirstNet excluded requirements related to resources. Similarly, FirstNet excluded requirements related to schedule risk analysis primarily, according to FirstNet officials, because risks to the established schedule milestones were largely considered when evaluating AT&T’s proposal prior to contract award. Finally, FirstNet officials highlighted that the schedule is not the only measure for progress and reporting, noting that it employs many other mechanisms to monitor and oversee AT&T’s progress and performance, and discusses the schedule during program management review and other meetings with AT&T.

However, the contract itself states that FirstNet is responsible for ensuring the overall success of the network and that, to do so, its responsibilities after contract award include overseeing the program schedule. Regarding resources in particular, the contract also states that these responsibilities include managing schedule resources. Thus, while it may not be necessary for FirstNet to collect information from AT&T on every resource detail, as FirstNet has stated, it is nevertheless important for FirstNet to gain an understanding of the overall resources needed to complete the work. This understanding could include, for example, evidence that sufficient resources were assigned to activities in the more detailed schedule that AT&T maintains. Our schedule guide notes that resources must be considered in the creation of a schedule—and it is important that FirstNet have sufficient insight into those resources—because their availability directly affects an activity’s duration. Regarding schedule risk analyses, consideration of risks to the milestones prior to contract award may not serve as a substitute for a risk analysis of the current schedule, which would include detail on activities and risks that could not have been known or fully understood prior to the award. Finally, while FirstNet utilizes a variety of other mechanisms to oversee AT&T’s performance, having a more detailed master schedule from AT&T would strengthen FirstNet’s use of the schedule as a management and oversight tool. For example, such a schedule could improve FirstNet’s insight into the activities driving AT&T’s deployment of the network and completion of requirements, how each activity relates to others, and any potential risks. It could also provide FirstNet with additional information that could help it and AT&T manage tradeoffs and make decisions to maximize the program’s success across the entire country.

Contract Oversight Communication with Stakeholders

Key practices for contract oversight call for communicating appropriate information to relevant stakeholders and reporting on monitoring results. Additionally, the 2012 Act requires FirstNet to consult—via a designated single point of contact (SPOC) in each state—with regional, state, local, and tribal jurisdictions regarding a host of activities, such as: ongoing compliance review and monitoring of the management and operation of the network; practices, procedures, and standards for the management and operation of the network; terms of service for use of the network; radio-access network build out, placement of cell towers, and coverage areas; and assignment of priority and selection of entities seeking use of the network.

Furthermore, the contract requires AT&T to report, by state, on the state- specific commitments made as a result of the state opt-in process. Portions of this report are to be shareable with states, and it is to detail the deadline by which the commitments will be fulfilled, the status of fulfilling them, and include evidence of the state’s satisfaction with progress. Beginning April 2018, AT&T is required to deliver this report semi-annually. Although two such state-specific commitment reports were due as of July 2019, only one has been completed by AT&T and accepted by FirstNet. Additionally, according to FirstNet officials as of October 2019, the report was not shared with the states.

Numerous state, local, and tribal stakeholders we interviewed described having had very little contact with FirstNet or being generally dissatisfied with the level or quality of information they had received from FirstNet and AT&T. These officials said that FirstNet had communicated little to no information on AT&T’s progress deploying the network in their area, or if and how FirstNet was monitoring performance. For example, many officials said that they had limited interaction with FirstNet beyond public relations emails or events promoting the network, or noted that their interactions lacked substantive information and details that would be of more value. The SPOCs were particularly dissatisfied with the lack of transparency surrounding the contractual requirements or FirstNet’s oversight of progress to date. Many of these state officials noted that the level of communication and information shared by FirstNet post contract award stood in stark contrast to the level of engagement prior to the state’s opt-in decision.

Numerous state, local, and tribal stakeholders we interviewed said that additional information on AT&T’s deployment and FirstNet’s oversight would be helpful or that greater transparency was needed. Officials wanted additional information on, among other things: contract requirements, milestones, and progress; technical details on the network including operational status and location of cell sites; subscribers within the official’s agency or agencies across the state that had adopted the network; and FirstNet’s oversight activities and results, including assurance from FirstNet that network coverage and performance had been verified. Even public-safety officials who were pleased with their experiences on the network to date or their relationship with FirstNet representatives reported that having more information was important. In the absence of this type of information, many public-safety entities we contacted expressed concern that they did not know whether FirstNet was holding AT&T accountable. For example, several officials indicated they did not know whether FirstNet or AT&T was “running the show.”

State, local, and tribal stakeholders we interviewed gave a variety of reasons for wanting greater transparency on contractual requirements and oversight. Numerous public-safety officials said that they needed to know this information for tactical response and planning, or state and local contracting purposes. For example, some local public-safety officials described wanting to have basic information on the contract coverage phases in their states so that they could confidently plan out equipment lifecycles. Additionally, many SPOCs said that there was a duty for FirstNet as the contracting agency to oversee that state-specific commitments were met. Many SPOCs also stated that their attempts to obtain more information from FirstNet or AT&T per the agreed-upon commitments had been delayed. At times, when they reached out to FirstNet, they were directed back to AT&T, or vice versa. Numerous stakeholders agreed that given the nature of the network as a public resource—involving public investment and funds, with the expressed purpose of serving public safety—they expected greater transparency from both FirstNet and AT&T.

FirstNet officials provided several reasons for not communicating the additional information cited by the stakeholders we spoke to and for not reporting on monitoring results. In particular, FirstNet officials told us there is no contractual requirement to communicate or share information collected, including any performance information or monitoring results, with any stakeholders or network users. However, its Public Safety Advocacy team serves as the primary interface to the public-safety community and conducts considerable outreach to stakeholders, as described above. Regarding the SPOCs, the officials further said that they believe the 2012 Act’s consultation requirement applied only to the initial planning stages (namely, the development of the request for proposal prior to contract award). As such, they do not believe they are legally obligated to continue to communicate specifically as identified in the 2012 Act. Additionally, FirstNet has stated that much of the information AT&T provides is proprietary and, therefore, cannot be disclosed to stakeholders. Finally, regarding the state-commitments report, FirstNet officials have said that FirstNet shares subsets of this information with states that request it during consultative interactions with FirstNet and in coordination with AT&T, but does not routinely share the full report to protect confidential commercial or trade-secret information.

While the 2012 Act does require consultation to occur “in developing requests for proposals,” it also states “and otherwise carrying out its responsibilities,” suggesting a broader application than just the initial planning stages, which is FirstNet’s interpretation. Moreover, while there are valid concerns about disclosing proprietary information and statutory prohibitions on doing so, there are opportunities for FirstNet to communicate additional information in ways it deems appropriate. For example, communicating how it oversees AT&T, the mechanisms it employs, and the performance areas it monitors could be done in a manner that does not disclose proprietary AT&T information, as these are government activities. Additionally, a state official and some local government officials we spoke to said that certain AT&T commercial information (e.g., the location of cell towers) could already be publicly available through local permitting offices. Further, federal internal-control standards note that management may select appropriate methods for external reporting, meaning management can consider what methods are appropriate for different audiences when communicating and reporting information. Finally, the contract states that except as specifically indicated or with explicit written permission from FirstNet, AT&T’s deliverables documentation shall not contain proprietary information or have any restriction on reproduction and/or distribution, suggesting that upon awarding the contract, FirstNet recognized the value of limiting these instances.

Industry guidance on project management that we reviewed—and which is cited in the contract—notes that analyses of high-profile project failures highlight the importance of stakeholder engagement. It also notes that communicating with stakeholders in an appropriate way can mean the difference between a project’s success and failure. Stakeholders’ lack of information on the program and FirstNet’s oversight of AT&T can make it difficult for stakeholders to assess what benefits have, or have not, been realized, which may affect their enthusiasm and continued support of the program. This scarcity of information has also left them speculating about other matters such as what, if any, oversight FirstNet conducts of AT&T. By not communicating additional information and reporting on monitoring results, FirstNet could be unknowingly reinforcing nascent skepticism of the program overall and of itself as the entity charged with holding AT&T accountable.

Information on End-Users’ Satisfaction

Key practices for contract oversight call for obtaining information on end- users’ satisfaction that can be used as a metric to gauge performance quality. For example, industry guidance on program management emphasizes that end-users’ satisfaction is a powerful metric that should be obtained to gauge program quality, noting that the benefits, product, or service delivered is best evaluated by those who receive it.

While FirstNet collects some information—via its QASP monitoring, as described above—that could relate to end-users’ satisfaction, these metrics provide limited insight into users’ experiences. For example, although AT&T surveys some customers to ask them whether they would recommend FirstNet services to a colleague to satisfy a QASP requirement, a user could recommend the service not because they are satisfied but because they have limited alternatives.

Indeed, while many state and local public-safety officials we spoke to were pleased with their experience migrating to or piloting the network, numerous officials told us about experiences that fell short of their expectations for a public-safety broadband network backed by the government. Numerous officials told us that they had concerns about misleading or disorganized sales tactics from AT&T representatives. For example, while some officials said that their AT&T representative had been candid in explaining the limited available coverage in their area, many officials told us about instances when AT&T representatives had shown them maps depicting more coverage than actually existed or that were insufficiently granular for their mission work. Similarly, while many officials recounted positive experiences with network coverage or performance or AT&T representatives, many also described instances when equipment failed to work or perform as expected during piloting phases or exercises. In some instances, these officials stated that FirstNet or AT&T representatives explained, after the fact, that differences in user experience were to be expected depending on the device model or subscriber identity module (SIM) card being employed. Specifically, FirstNet or AT&T officials explained that the optimal performance could only be achieved when Band 14 devices connected to a Band 14 cell site.

According to FirstNet officials, the best experience will be when subscribers use a Band 14-capable FirstNet-ready device with a FirstNet SIM card while in a Band 14 coverage area. The officials said any other combination could result in slightly degraded performance or features being unavailable. This is notable given that Band 14 coverage is still limited and generally state and local public-safety officials do not have insight as to where these sites were located or when, if ever, coverage will be expanding, as previously discussed. As stated above, at its final operating capability, the network utilizing Band 14 spectrum will not cover the entire country.

Many officials also expressed concerns about the network’s quality of service, priority, and preemption capabilities over the long run or during a catastrophic event. They speculated about the type or expanding number of subscribers allowed on the network or whether at some point in the future, the network would become saturated because non-public safety organizations or individuals (either extended-primary users or non-verified public-safety subscribers) were being granted priority and preemption capabilities. Exacerbating these concerns, many officials noted that they did not have insight into who had subscribed even within their own agency or state, or lacked confidence in how FirstNet or AT&T verifies individuals’ public-safety status, based on anecdotal experiences. Further, some officials also raised concerns about their inability to test the network during congested periods or simulate catastrophic power failures and lack of insight into if or how AT&T had hardened the network. Many officials discussed or shared after-action reports or their testing results with us, and several communicated that they had shared or would be willing to share such information with FirstNet as well to support validation of the network’s actual performance.

According to FirstNet officials, the key performance indicators identified via the QASP are the performance quality measures, not end-users’ satisfaction. They also stated that “disincentive” payments embedded in the contract serve as an incentive for AT&T to ensure end-users’ satisfaction. Specifically, if AT&T does not meet user adoption (i.e., device connection) goals specified in the contract, it has to make payments to FirstNet on a timetable identified in the contract. Additionally, according to FirstNet officials, they informally hear information on end- users’ satisfaction and the network’s performance through many of the engagements its Public Safety Advocacy team conducts, which they can informally share with AT&T.

However, disincentive payments (and the user-adoption goals tied to them) may be a limited reflection of end-users’ satisfaction for various reasons. For example, users may continue to subscribe to the service not because they are satisfied with it but because agency procurement lifecycles and budgets prevent them from changing providers, or because they find it difficult to break a sales contract, have already sunk costs into the transition, or lack alternatives in the market. Additionally, if AT&T perceives that the value derived from its commercial customers’ use of the excess Band 14 spectrum capacity is greater than the disincentive payment it must make to FirstNet, it may view making the payment as an acceptable tradeoff. Alternatively, aggressively pursuing sales contracts with potential public-safety users to avoid the payments may not be welcomed by the public-safety community, which could result in negatively, not positively, affecting end-users’ satisfaction, as some public-safety network users we spoke to said it had. Finally, while the informal collection and sharing of information on satisfaction can be valuable, it does not serve as a formal performance-quality measure, which could provide FirstNet with additional recourse should issues arise.

End-user adoption is both a goal of the program and how AT&T plans to fund the $40 billion of investment in the network. Adoption may be driven by satisfaction in addition to need. Ultimately, end-users’ dissatisfaction could affect the success of the program. Thus, FirstNet’s lack of formal insight into end-users’ satisfaction hampers its ability to take actions that could increase the program’s chance of succeeding. By not obtaining and using this information to inform its oversight or related activities, FirstNet could be missing an opportunity to increase assurance of the program’s long-term success.

Conclusions

The FirstNet public-safety broadband network has the potential to save lives every day. Since beginning their 25-year partnership, AT&T has made progress deploying the network and meeting contractual milestones and goals, and FirstNet has employed a variety of mechanisms—many of which align with key practices—to oversee AT&T’s performance. However, the success of the network depends not only on AT&T’s contract execution and FirstNet’s oversight but also on the confidence of the end users, the nation’s first responders. As FirstNet enters the next phases of its partnership with AT&T, it could reduce the risks to the network’s long-term success by strengthening its schedule oversight; increasing transparency, communication, and reporting of additional information to states and other public-safety stakeholders; and obtaining and using meaningful information on the satisfaction of the first responders for whom the network is intended.

Recommendations for Executive Action

We are making the following four recommendations to FirstNet:

FirstNet’s Chief Executive Officer should take steps to ensure that the integrated master schedule for the program is developed and maintained in accordance with the best practices provided in GAO’s Schedule Assessment Guide. (Recommendation 1)

FirstNet’s Chief Executive Officer should identify additional information about the program, including FirstNet’s oversight and monitoring activities, that can be shared with public-safety stakeholders and periodically communicate and report this information to them. (Recommendation 2)

FirstNet’s Chief Executive Officer should share relevant portions of the accepted state-specific commitment reports with the states, as specified in the contract. (Recommendation 3)

FirstNet’s Chief Executive Officer should, in consultation with public- safety stakeholders and its contractor, as appropriate, identify and obtain periodic information or meaningful indicators on end-users’ satisfaction that would serve as a metric to gauge performance quality, including the effect of the FirstNet network and products on public-safety operations. (Recommendation 4)

Agency Comments

We provided a draft of the sensitive report to FirstNet for review and comment. FirstNet’s comments on the sensitive report are reprinted in appendix II. In these comments, FirstNet stated that it agreed with all of our recommendations; will take appropriate additional steps to apply lessons learned and address our concerns; and will continue to find ways to improve transparency with and feedback from its stakeholders, in addition to refining the integrated master schedule. Separately, FirstNet also provided technical comments, which we incorporated as appropriate.

We are sending copies of this report to the appropriate congressional committees, the Chief Executive Officer of FirstNet, the Secretary of Commerce, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-2834 or vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

This report examines the extent to which (1) AT&T is meeting the established milestones for deploying the nationwide public-safety broadband network, including coverage and adoption goals, via its contract with the First Responder Network Authority (FirstNet), and (2) FirstNet is overseeing AT&T’s deployment of the network in accordance with key practices.

To assess progress toward the coverage and adoption milestones, we reviewed the FirstNet-AT&T network contract, corresponding task orders, and relevant documentation contained in FirstNet’s contract files, including information or “deliverables” submitted by AT&T that had been reviewed by FirstNet for contract compliance. We also reviewed additional FirstNet documentation, such as board-meeting materials, annual reports to Congress, press releases, fact sheets, and official blog postings. We reviewed the Middle Class Tax Relief and Job Creation Act of 2012 (the 2012 Act), which created FirstNet as an independent authority charged with establishing a nationwide public-safety broadband network that would, among other things, be deployed in phases that included substantial coverage milestones in rural areas. Within the contract, we identified the various coverage and adoption milestones and focused our analysis primarily on task order 3, phase 3 (which spanned March 31, 2018, to March 30, 2019) and task order 4, phase 2 (which spanned October 1, 2018, to March 30, 2019) milestones. We focused on these task orders because they are most relevant to the network’s coverage deployment and adoption, and on these phases because they were the phases under way at the time we began our review. We did not review activities or progress as described in AT&T deliverables dated beyond September 2019 given the timing of our review. We also did not make any conclusions about progress toward the final phases of these task orders. However, we did assess the master schedule to determine its reliability and validity for planning and tracking progress toward the final phases as described further below and in our report.

The contractual deliverables that we reviewed in some cases included detailed data broken out by state and public-safety discipline. In particular, we analyzed data that indicated progress toward nationwide and state Band 14 network coverage (in square miles); cell site delivery; monthly adoption targets (i.e., device connections) by discipline; and types of devices connected. When analyzing these data, in all cases, we used the most currently available data at the time of our request for the information, and we report data as of September 2019. Although all data were the most currently available as of September 2019, because the deliverables have varying cycles for when AT&T is contractually required to report the information, we specify throughout the report the “as of” period these data represent. We assessed the reliability of these data by asking FirstNet officials questions about how they review the deliverables and about data sources, quality, and timeliness, as well as by electronically testing the dataset for missing or invalid entries. We removed a small number of missing or invalid entries from our analysis of device types and models and count of public-safety agencies. We did not assess AT&T’s underlying systems or databases, nor did we interview AT&T officials about their protocols for producing this data. We found these data reliable for the purpose of describing FirstNet’s current and projected progress toward coverage and adoption milestones for the related task orders and phases.

To further assess deployment progress, we conducted case studies of seven states to illustrate and obtain greater context on variations in state- level coverage and adoption. We selected our case-study sample to include states that had very high-density counties; relatively large numbers of low-population density counties; high poverty rates (due to budgetary challenges public-safety entities may face); varying levels of progress in cell site delivery as of January 2019 (the most currently available data at the time of our selection); and geographic diversity and tribal lands. In total, the selected states represent almost a third of the contract dollars allocated for network coverage deployment. Our case- study analyses included reviewing and comparing the deployment plans and commitment letters for these seven states (detailing the agreed-upon, state-specific commitments AT&T made to these states) against the deliverables describing the progress AT&T made on some of these commitments, as of July 2019. It also included interviewing state, local, and tribal officials and first responders from these states, as described further below. The case studies and stakeholders’ views illustrate experiences with FirstNet’s deployment of the network across a wide cross section of geographies and network users to date but are not generalizable to those of all FirstNet stakeholders or the network as a whole. We also interviewed FirstNet officials to obtain their perspectives on AT&T’s progress and factors that may explain the variance across states.

To examine FirstNet’s oversight efforts, we reviewed the FirstNet-AT&T network contract and documentation contained in FirstNet’s contract files, as well as additional FirstNet documentation. In addition to the material described above, this documentation included, for example, the Quality Assurance Surveillance Plan, requirements traceability matrixes, verification reports, memos, Contract Administration Plan, FirstNet Acquisition Manual, guidance documents on contract management and procedures, and FirstNet officials’ written responses to questions we posed. For the same reasons described above, we focused primarily on material related to task order 3, phase 3 and task order 4, phase 2. We interviewed FirstNet officials to obtain greater context on FirstNet’s oversight mechanisms and their use, and to observe FirstNet’s verification activities and the platform it uses to manage its contract files.

Further, we reviewed key acquisition and contract-oversight practices established in the Federal Acquisition Regulation and the Commerce Acquisition Regulation, as well as the Commerce Acquisition Manual and other academic and industry guidance. We also reviewed the 2012 Act and federal standards for internal control. We selected those practices that were most appropriate given FirstNet’s contract approach (i.e., Indefinite-Delivery/Indefinite-Quantity, Firm-Fixed-Price contract vehicle) and the stage of the acquisition process FirstNet was in during the course of our review. We assessed FirstNet’s oversight efforts against these practices. We also compared the network’s integrated master schedule, which AT&T provides to FirstNet, to scheduling best practices in GAO’s schedule guide. Collectively, these best practices are organized into four characteristics of a reliable schedule. A schedule is considered reliable if each of the four characteristics is substantially or fully met; if any of the characteristics are not met, or minimally or partially met, the schedule cannot be considered reliable. We reviewed the schedule as of its status date January 31, 2019, which represented the latest status update to the schedule at the time we began our schedule analysis. In reviewing the schedule, we also reviewed the schedule dictionary, work breakdown structure, and program management review or monthly progress reports dated October 2018 to January 2019, among other documents. We provided our criteria and draft schedule analyses to FirstNet for review.

To inform both of our objectives, we conducted about 40 interviews with state, local, and tribal officials and first responders. These interviews represented almost 30 different states’ single point of contact (SPOC) to FirstNet or their designees, and over 30 different state, local, or tribal public-safety entities. The public-safety entities we interviewed included police and fire departments, sheriffs’ offices, emergency medical-services providers, and emergency-management agencies, among others.

We interviewed the SPOC from each of our case-study states and received information from other SPOCs (or a designee) via a multi-state focus-group discussion and written responses to the semi-structured discussion questions and prompts we posed. A GAO moderator led the discussion to establish ground rules and keep participants focused on the specified issues within the discussion time frame. We selected state, local, and tribal public-safety entities within our case-study states to interview. To select the state and local public-safety entities to interview, we reviewed the AT&T subscription management report provided to FirstNet as of February 2019 (the most current available at the time of our selection) and asked the SPOCs for recommendations within their state. Generally, we selected among the largest subscribers (meaning, the most number of devices on the network) in each of the primary public-safety disciplines (law enforcement, fire, emergency-medical services) in each state, and selected others to ensure representation among urban, suburban, and rural areas. To select the tribal entities to interview we asked the National Tribal Emergency Management Council for a recommendation in each state. Not all public-safety entities accepted our interview requests. Among our case-study states, we conducted a site visit in one state region. We selected this region for our visit because of the concentration of subscribers within reasonable geographic proximity to each other. For additional context, during this visit we also met with the FirstNet Public Safety Advisors that serve the state and attended a FirstNet presentation and town hall meeting hosted by the local chapter of the Association of Public Safety Communications Officials. Because stakeholders varied in their expertise with various topics, not every stakeholder provided an opinion on every topic. Throughout this report we refer to “some” stakeholders if officials from 3–5 entities, “several” if 6–9, “many” if 10–19, and “numerous” if 20 or more expressed the view. Finally, for additional perspective we also interviewed the National Public Safety Telecommunications Council because of its role as a federation of organizations whose mission is to improve public-safety communications and interoperability. As noted above, stakeholders’ views are not generalizable to those of all FirstNet stakeholders.

Appendix II: Comments from FirstNet

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Sally Moino (Assistant Director); Nalylee Padilla (Analyst in Charge); David Aja; Melissa Bodeau; Andrew Burton; Mark Goldstein; Yvette Gutierrez; David Hooper; Jason Lee; Andrew Stavisky; Hai Tran; William Woods; and Friendly Vang-Johnson made key contributions to this report.