Emergency Alerting: Agencies Need to Address Pending Applications and Monitor Industry Progress on System Improvements
Released: 2020-02-06T00:00:00

Letter

Background

In 2004, FEMA initiated the IPAWS program to integrate EAS and other public-alerting systems into a larger, more comprehensive public-alerting system. As shown in figure 1, IPAWS serves as a centralized gateway to deliver alerts to the public. After an alerting authority creates and sends an alert to IPAWS, the system then routes the alert to the public using one or more of the following pathways:

Emergency Alert System. Allows authorized federal, state, territorial, tribal, and local government agencies to use EAS media platforms— including radio and television—to send alerts. IPAWS also allows the U.S. President to activate EAS to communicate to the public through all EAS media platforms during a national emergency.

Wireless Emergency Alerts. Allows authorized federal, state, territorial, tribal, and local government agencies to send text-like messages to mobile devices in geographically selected areas as one-way cellular broadcasts. Various factors affect whether a WEA message will be received on a mobile device, such as whether the device is WEA-capable and within range of a cell tower where a participating wireless carrier provides WEA services to its customers. According to CTIA, a wireless industry association, more than 100 nationwide and regional wireless carriers participate and have the capability to provide WEA messages to 99 percent of American wireless subscribers.

IPAWS alert feed for internet services. Allows internet companies authorized by FEMA—such as Google, Facebook, and The Weather Channel—to retrieve IPAWS alerts and distribute them to social media, websites, applications, and subscription services.

Government agencies and industry organizations play different roles in providing, protecting, and leveraging the nation’s emergency alerting capability.

FEMA. FEMA is responsible for operating, maintaining, and administering access to IPAWS, including managing the application process. As discussed earlier, public safety agencies that wish to use IPAWS must apply to FEMA to become approved alerting authorities. FEMA, in consultation and coordination with FCC, must carry out various actions to modernize and implement IPAWS. For example, FEMA must ensure IPAWS can send alerts to a specific geographic location and to multiple communications systems and technologies, educate government users of IPAWS and provide training opportunities to them, and conduct nationwide tests of IPAWS, among other things. Legislation was enacted that expands FEMA’s responsibilities for IPAWS.

FCC. FCC creates the rules for EAS and WEA, the two primary alerting pathways authorities use to send public alerts through IPAWS. FCC establishes the technical standards, procedures, and testing protocols for EAS participants. FCC also manages an online system used to collect and analyze results of nationwide EAS tests. FCC establishes technical requirements participating wireless carriers must follow for delivering WEA messages to WEA-capable mobile devices.

Federal alerting authorities. Authorized federal alerting authorities may create alerts and use IPAWS to send alerts to the public. For example, the National Weather Service (NWS), within NOAA, uses software NWS developed to issue WEAs for severe weather risks such as flash floods and tornadoes. USGS intends to send earthquake-related alerts through IPAWS but as of September 2019, had yet to send such an alert through IPAWS. USGS has partnered with Washington, Oregon, and California to test and implement a West Coast earthquake early warning system called “ShakeAlert” that is intended to send WEA messages to mobile devices several seconds after the initiation of an earthquake.

State, territorial, tribal, and local alerting authorities. According to FEMA policy, state, territorial, tribal, and local government agencies first complete FEMA’s application process to gain access to IPAWS and obtain the proper authorization to issue alerts for specific geographic jurisdictions. As discussed earlier, government agencies that issue alerts through IPAWS can include emergency management or law enforcement agencies at the state, county, or city government level. Non-governmental organizations such as a local emergency management association may be granted an authority to issue alerts through IPAWS with approval from FEMA or an alerting authority. For information on FEMA’s IPAWS application process, see figure 2 below.

Industry. Industry develops and owns the infrastructure that enables alerts to be created, authenticated, and delivered to the public. Alerting software companies provide software tools that allow alerting authorities to create and send alert messages via the internet to IPAWS. Alerting software companies also provide “opt-in” or subscription-based alerting services to public safety agencies that allow the public to sign up to receive alerts. EAS participants that transmit EAS messages include radio and television broadcasters, cable operators, wireline video service providers, satellite radio providers, and direct broadcast satellite providers. Wireless carriers operate wireless networks that allow alerting authorities to send one- way geographically targeted WEA messages to WEA-capable mobile devices. Manufacturers develop, test, and provide WEA-capable mobile devices, in coordination with participating wireless carriers, to consumers. Internet and web services companies may also distribute alert information from an IPAWS alert feed to internet applications, websites, or social media.

We have previously reviewed FEMA’s progress in implementing IPAWS. In 2013, we found that FEMA had taken actions to improve the capabilities for IPAWS and to increase federal, state, and local capabilities to alert the public, but barriers remained to fully implement an integrated system. We made six recommendations, including that FEMA work with FCC to establish guidance for states to fully implement and test IPAWS components and implement a strategy for regular nationwide testing. The agencies implemented all of the report’s recommendations.

IPAWS Usage and Testing Have Increased but Parts of the Country Lack IPAWS Access at the Local Level

Substantially More Local Authorities Have Access to IPAWS since 2013, but Gaps Remain at the Local Level

Our analysis of FEMA data found 1,401 alerting authorities at the federal, state, local, territorial, and tribal levels had access to IPAWS to send alerts as of September 2019, a substantial increase from 2013 (soon after it became operational) when fewer than 100 authorities had access. According to FEMA officials, nearly 70 percent of the nation’s population is covered by a local alerting authority that can use IPAWS to send alerts. Further, according to FEMA documentation, from a state authority perspective, all 50 states, the District of Columbia, Puerto Rico, and the U.S. Virgin Islands have at least one state-level authority that can use IPAWS to send alerts to any locality within that state or territory.

Local authority access to IPAWS to send alerts varies, however, as FEMA officials stated that about two-thirds of the nation’s 3,000 counties do not have access to IPAWS to send alerts. Although access to IPAWS at the state level enables alerts to be sent, for example, to jurisdictions that may have lost their capability during an emergency, gaps in access to IPAWS for local officials could limit the timeliness of alerts as emergencies occur. For example, officials from an alerting authority told us that with the exception of alerts issued by NWS, all emergencies start locally. If a locality does not have access to issue an alert through IPAWS, information must be communicated from the locality to an authorized state official to issue the alert, which could result in delays in getting critical information to the public.

Reasons for this gap at the local level could be related to a variety of factors. For example, some counties may still be in the process of applying for access. Other counties may not be able to gain access to IPAWS due to state or local laws, or a state’s EAS communications plan may specify that only certain types of agencies can issue alerts. For example, state EAS communications plans may authorize the governor of the state, an emergency management office, state law enforcement agency, or a non-governmental organization as the authorized agencies for sending alerts. In addition, an academic who specializes in rural emergency management told us that unfunded staff positions in emergency management are commonplace in rural areas and the areas may lack funding to apply for IPAWS access. Figure 3 highlights areas of the country that were covered by a local or tribal alerting authority as of September 2019.

Wireless Emergency Alerts Have Become the Primary Alerting Method and Usage Has Increased

Alerting authorities at the state, territorial, and local levels have increasingly used WEA messages since 2012 (see fig. 4). In addition, these authorities used more WEA messages than EAS alerts each year, with a large difference occurring between 2017 to 2018, when WEA messages increased by 89 percent while EAS alerts increased by 35 percent.

While usage of WEA and EAS by state, territorial, and local authorities has generally increased since 2012, our analysis of FEMA data found that this increase was driven by a small group of alerting authorities in certain parts of the country. Some locations may be more prone to experience certain types of emergencies, particularly weather related emergencies such as hurricanes. However, the potential exists in any location for an alert to be sent to the public if an alerting authority determines an imminent threat to public safety exists. Specifically, our analysis of WEA alert data from April 1, 2012 to October 1, 2019 found:

A total of 236 of the 1,372 state, territorial, and local alerting authorities sent a WEA message.

A total of 69 of the 1,372 state, territorial, and local alerting authorities accounted for nearly 80 percent of WEA messages sent at those levels.

Most of the country has received a low number (fewer than 10) or no alerts sent by state, territorial, and local authorities, while limited parts of the country have received higher numbers of alerts (see fig. 5).

At the federal level, our analysis of FEMA data found that NWS sends the vast majority of WEA messages sent through IPAWS, a number that from April 1, 2012 to October 1, 2019 totaled more than 46,000. The most common WEA messages sent by NWS were related to flash flooding (28,640), tornadoes (15,985), hurricanes (571), and dust storms (386). An academic we interviewed said it is important to note that local alerting authorities use the NWS warnings to issue alerts instructing the public to take specific protective actions, for example, to evacuate using certain roads. For more information on when a person may receive a WEA message on a WEA-capable mobile device through IPAWS, see appendix II.

NWS uses multiple alerting mechanisms to send alert messages to people around the country. As one of its mechanisms, NWS uses the Integrated Public Alert and Warning System to send Wireless Emergency Alert messages to mobile devices in areas facing w eather risks, such as this geographically targeted message to a cell phone in Washington D.C. in July 2019.

To test the capability and effectiveness of IPAWS, FEMA, FCC, NWS, and state and local public safety agencies have carried out nationwide and localized alert tests since 2016.

Nationwide EAS Tests: FEMA, in coordination with FCC, conducted four annual nationwide EAS tests from 2016 to 2019. The tests assessed how well EAS alerts were received and retransmitted using the two ways an EAS alert can be delivered: (1) over the internet via IPAWS and (2) through the legacy “over the air” radio and television broadcast stations. According to FCC’s analysis, about 76 percent of an estimated 26,000 EAS participants took part in the 2018 test, with about 96 percent of participants reporting they received the test alert. While the vast majority of EAS participants reported no complications, FCC’s analysis identified some problems with the 2018 test, including EAS participants reporting audio quality issues (less than 2 percent), EAS equipment issues, out-of- date software, user error, and complications accessing IPAWS (less than 1 percent each). To help address such issues, FCC provided advisories in advance of the next nationwide EAS test. In addition, FEMA has publicly identified how FCC could improve future nationwide tests, including improving the accuracy of reporting and other audio and visual technical issues. FEMA officials said they are working with FCC to resolve technical issues found in recent tests.

Nationwide WEA Tests: FEMA, in coordination with FCC, carried out the first nationwide WEA test in October 2018. FEMA sent a test alert through IPAWS to participating wireless carriers, which then transmitted the alert to their subscribers’ WEA-capable devices across the country. FEMA officials viewed the first nationwide WEA test as a success with regard to the technical execution of delivering a nationwide WEA message via IPAWS. However, officials acknowledged a main lesson from the test was a need to collect data on how effectively WEA messages are being received.

While FCC collects EAS test data to assess how well the EAS test was received and retransmitted, a similar mechanism does not exist for the WEA pathway. According to wireless industry representatives we interviewed, the WEA system was designed to use a one-way broadcast cellular technology that prevents the wireless network from collecting data from mobile devices. Instead, FCC has used voluntary public responses, media reports, and informal surveys conducted by state and local public safety agencies to assess results. For example, FCC’s report on the 2018 WEA test found that media sources reported inconsistent WEA delivery in different parts of the country and that informal surveys conducted by state and local agencies showed variability in WEA delivery. FCC also reported that issues were found during the WEA test related to duplicate messages and audio and vibration cadence that could have affected individuals with disabilities. At the time of our review, FEMA officials said they were preparing to conduct the next nationwide WEA test in late 2020 and developing a survey to accompany the test to collect data on WEA message delivery.

The District of Columbia Homeland Security and Emergency Management Agency used the Integrated Public Alert and Warning System to send a geographically targeted WEA test to mobile devices in Washington, D.C. in June 2019. owned cell phones received the test alert within a range of 6 seconds and several minutes.

In May 2019, FCC rules initially went into effect that will allow alerting authorities to send WEA tests to the public without FCC approval—called State/Local WEA Test. Participating wireless carriers are required to provide the capability, but subscribers must manually opt-in to receive these alerts on their mobile devices. In November 2019, a major wireless carrier obtained a waiver from FCC to conduct two WEA tests under these rules to assess the carriers’ ability to perform enhanced geo- targeting for WEA messages.

Alerting Authorities Cited Benefits and Limitations of Using IPAWS during Recent Emergencies

Officials from alerting authorities we contacted for seven case studies on the use of IPAWS cited benefits and limitations of using the system during recent disasters such as wildfires, a hurricane, a flood, an earthquake, a chemical fire, a power shortage, and a law enforcement event.

Benefits. Officials from authorities we interviewed said that IPAWS has a wide reach because most people have mobile devices to receive WEA messages, and WEA can also reach visitors to their area. Compared with opt-in alerting systems that can have a low percentage of subscribers, officials from alerting authorities we interviewed said that IPAWS provides an opportunity to reach more people during emergencies. In addition, they said that states can act as back-ups for local authorities that have lost their alerting capabilities to help ensure that alerts can be sent. Our analysis found that state and local alerting authorities used IPAWS to send alerts regarding a variety of emergencies, examples of which are shown in table 1. Alerting authority officials also said they plan to use IPAWS in a variety of ways in the future, including for mudslides, rip currents, hazardous materials incidents, and law enforcement emergencies such as terrorism or active shootings.

Limitations. Officials from alerting authorities we contacted cited three main limitations. First, they said it was difficult to write effective WEAs within the 90 character limit. For example, officials from an authority said that within the character limit it is difficult to explain the risk, who the alert is from, and what the public should do. As we discuss later, FCC has expanded the character limit. Second, officials expressed concerns about the ability to target WEAs to specific geographic areas, which caused some to lack confidence in the system or not use it at all. Third, officials from alerting authorities said that because WEA is a one-way communication system, they do not know if the alerts reached the intended public. For example, officials from one authority described sending an evacuation order but not knowing whether people in the intended area received it. In another example, while an alert was helpful in alerting the public about a suspicious package, officials from one authority said the alert was received 4 miles beyond its intended target, which led them to speculate about the number of people who received the alert.

More information about the use of IPAWS during events we selected as case studies is provided in appendix III.

FEMA and FCC Have Taken Steps to Improve Alerting but Face Challenges Monitoring New Capabilities and Managing Pending Applications FEMA and FCC Have Taken Actions to Improve Alerting Capabilities

FEMA has taken recent steps to modernize IPAWS by implementing various improvements and exploring new technologies. For example, FEMA is moving IPAWS to a cloud-based data center to enhance the system’s availability and is modernizing the stations that serve as the main broadcast source for national emergency alerts, according to FEMA’s 2018 performance report for IPAWS. In addition, officials described how FEMA has assisted with developing technical standards for new IPAWS capabilities and engaged the private sector to explore possibilities for integrating alerts into technologies such as digital billboards, Braille reader devices, and internet-connected devices in homes and vehicles.

FCC has published rules that require participating carriers to implement new or improved capabilities for wireless alerts sent through IPAWS.

Improved alert message content and capabilities. FCC required wireless carriers to support several capabilities to help alerting authorities communicate clearly and effectively, including the ability to send longer messages (expanding the limit from 90 to 360 characters) and the ability to send alerts in Spanish. Initially, FCC set a May 1, 2019, deadline for carriers to support all of these capabilities but later extended it to December 19, 2019, to allow time for carriers to complete testing with IPAWS. FEMA completed the necessary updates to support formal testing with the IPAWS gateway in mid- November 2019. Two academics we interviewed who have researched emergency alerting told us that alerts with expanded character length are more effective in prompting people to take protective actions, compared with shorter ones. Other new capabilities required include “alert prioritization,” meaning that alerts must be displayed as soon as they are received and a new “public safety message” category for advisories that prescribe one or more actions likely to save lives or safeguard property during an emergency (e.g., boil water notices, emergency shelter locations). As discussed earlier, a state/local WEA test option was also required to allow alerting authorities to send test messages to a subset of the public without prior approval from FCC.

More precise geographic targeting. FCC required carriers to deliver alerts to areas that match the targeted geographic area, to within one- tenth of a mile, a capability that FCC calls enhanced geo-targeting.

FCC initially required carriers to implement enhanced geo-targeting by November 30, 2019, but later extended it to December 19, 2019, to allow time for carriers to complete testing with IPAWS, as with the capabilities discussed above. FEMA completed the necessary updates to support formal testing with the IPAWS gateway in mid- November 2019. Previously carriers have been required to transmit alerts to the geographic area that best approximates the emergency area identified by the alerting authority. As FCC’s chairman has explained, these less precise geographic targeting capabilities can result in overbroad alerting, where people may receive the alert even though they are located well outside of the target area. Several local WEA tests in 2018 found overbroad alerting when targeting specific geographic locations. Officials from many alerting authorities we interviewed told us they are concerned about the inability to geographically target alerts with accuracy, which can make some reluctant to send WEA messages. According to several wireless and device industry representatives we interviewed and letters that wireless carriers have sent FCC, enhanced geo-targeting is a particularly challenging capability to implement because changes must also be made by different sectors of industry—such as manufacturers of cell phone handsets and chipsets. Some industry representatives also told us that only some, mostly newer model cell phones will be able to receive the more precise geo-targeted alerts and that many older devices currently in the population will not support this new capability because it requires a new chipset.

Other recent improvements. FCC has also required implementation of new alert content and categories, such as: “Clickable” links—Embedded links in alerts so people receiving them can click on the link to see a photo of a suspect, for example. This capability has been implemented.

Blue Alert—A new type of alert to notify the public of threats to law enforcement and to request help apprehending dangerous suspects. This capability has been implemented.

24-hour alert retrieval—Alerts must remain available on devices for 24 hours after receipt, or until the consumer chooses to delete them. FCC required carriers to implement this capability by November 30, 2019, and FEMA officials told us this capability did not require technical changes to the IPAWS gateway.

Although FEMA and FCC are taking actions to improve alerting capabilities, developments in technology are changing the alerting landscape. Our analysis of agency documents and interviews with public- safety stakeholders indicated two emerging and unresolved areas.

Multimedia. In 2018, an FCC advisory committee recommended that alerting systems should carry graphics and other multimedia. For example, four public-safety stakeholders told us it would be helpful to include multimedia (e.g., photos and maps) directly within WEA messages. Doing so would allow the public to see the information without clicking an embedded link. In 2015 and 2016, FCC sought comment on the technical feasibility of including multimedia and in 2018 issued another public notice on the topic to refresh the record. The proceeding remains open and FCC has not taken additional action.

Internet streaming. The public may not receive broadcast EAS alerts when watching television that is streamed through an internet connection. A 2017 Pew Research Center survey found that 28 percent of American adults and 61 percent of adults age 18 to 29 said that streaming is their primary way of watching television. Representatives from two internet service providers told us they have developed solutions that enable customers to receive EAS alerts when the customers are using their applications to stream content. However, EAS alerts may not override other streaming services such as video and gaming because of technical limitations and the limited information that content service providers maintain about a user’s location, according to industry representatives. For example, representatives from an association representing internet companies told us that providing emergency alerts through internet streaming services presents technological challenges and that its members would have concerns about collecting locational information about their customers. The effect of potentially not receiving an EAS alert while streaming is unclear. While more Americans are streaming their television and multimedia, many use a second screen such as a cell phone while watching television and could receive any relevant alert as a WEA message. A 2018 Nielsen survey found that 45 percent of respondents very often or always use a second screen such as a smartphone while watching television. FCC has sought comment about this issue in general. FCC officials told us that extending EAS to new technologies for viewing video content raises legal and technical considerations and that they continue to evaluate the efficacy, costs, and benefits of doing so.

FCC Lacks Goals and Measures for Monitoring Performance of Required Capabilities

Pursuant to statute, FCC is responsible for establishing technical standards and requirements for WEA, as discussed earlier. Further, FCC’s 2018–2022 strategic plan identified a performance goal to facilitate the effectiveness and reliability of EAS and WEA, and following a nationwide test in 2018 FCC’s Public Safety and Homeland Security Bureau recommended that additional measures be taken to improve the reliability and effectiveness of WEA. Developing goals and performance measures is consistent with leading practices for performance management. GPRA, as amended and expanded by GPRAMA, creates a framework for articulating goals and measures that can provide federal agencies with a clear direction for successful implementation of activities and improve the efficiency and accountability of agencies’ efforts.

Goals explain the purpose and intended results that a program seeks to achieve in its work.

Performance measures that are linked to goals allow a program to track the progress it is making toward achieving its goals.

While GPRA and GPRAMA apply to the department or agency level, we have previously reported that their provisions can serve as leading practices at other organizational levels, such as component agencies, offices, programs, and projects. Additionally, federal internal control standards discuss the importance of goals, stating that management should define objectives clearly. This involves defining objectives in specific and measurable terms so that they can be easily understood and performance toward achieving those objectives can be assessed. Federal internal control standards also state that measurable objectives should be specific and stated in quantitative or qualitative form.

FCC has required carriers to implement new WEA capabilities and taken steps to understand more about WEA performance, but FCC has not developed goals and performance measures to help monitor how well the new capabilities perform during emergencies. Instead, we found FCC has taken an ad-hoc approach to monitoring WEA performance. In particular, when we asked whether FCC planned to develop standards or benchmarks to measure WEA performance, FCC officials said they intend to use certain test results, as discussed below, to understand more about WEA performance.

Partnered geo-targeting tests. FCC intends to partner with localities to test the accuracy of participating wireless providers’ enhanced geo- targeting capabilities starting in early 2020. Four localities have applied to participate as of November 2019, according to FCC officials. To perform each test, FCC and its partner at each given location intend to use online surveys to collect information on which individuals receive the test alert and under what circumstances. However, at the time of our review we found that while FCC has broadly identified the purpose of the tests as testing the accuracy of enhanced geo-targeting, it has not defined specific, measurable goals that are specific to this testing effort. For example, FCC has not stated what would be an appropriate success rate for enhanced geo- targeting accuracy. We also found that FCC has not connected its survey questions to specific performance measures that could be compared across test locations. According to FCC officials, FCC has not announced whether it will compare results across localities or use specific performance measures to assess geo-targeting performance. FCC officials said they have no plans to test other new WEA capabilities, including the expanded message length, and that at the time of our review it was too early to say how results from the partnered tests would be analyzed and shared more broadly with public-safety stakeholders.

State and local tests. As discussed earlier, FCC officials told us that 39 alerting authorities at the state and local level received approval from FCC to conduct their own WEA tests as of November 2019. FCC officials also told us that that they encourage alerting authorities that seek approval for WEA tests to share performance data with FCC. According to FCC officials, FCC has received data from nine localities as of November 2019 and will use the test results internally to develop a broader understanding of WEA performance. When we asked what FCC has learned from the data, FCC officials said they have received some results but are still in the process of analyzing them.

By developing goals and performance measures for its efforts to monitor the new WEA capabilities, FCC would have clearer direction for what it plans to achieve and more specific means to assess the performance of the capabilities. For example, performance measures related to FCC’s planned survey questions for geo-targeting could include the percentage of participants who received the alert and the percentage who received the alert within the target geographic area. Another performance measure for the new capabilities could include the extent to which messages of up to 360 characters are fully or partially displayed on a mobile device, or not at all, for example. Without specific goals and performance measures, FCC will have difficulty knowing if it is making progress toward its stated strategic goal of ensuring the effectiveness of WEA. The results of data collected on performance measures could provide assurance that new WEA capabilities are working as intended during emergencies, or could point to areas where performance is lacking and where FCC might need to take other actions such as working with industry to resolve issues, updating WEA requirements, or conducting additional analysis. Monitoring performance is all the more important because of uncertainty about the extent to which all cell phones will be able to receive WEA messages with the new capabilities.

In addition, new capabilities have the potential to make WEA a more powerful tool and possibly further increase its use. Our analysis shows that WEA has outpaced the use of EAS as an alerting method, and according to the Pew Research Center, Americans are increasingly connected to digital devices, with 96 percent of American adults owning cell phones in 2019 and 81 percent owning smartphones. However, as discussed earlier, officials from many alerting authorities we interviewed had concerns with WEA performance. Many officials from alerting authorities told us that they were looking forward to the new capabilities— including enhanced geo-targeting and expanded message length—which will improve their ability to alert the public. Having specific performance information about the effectiveness of these capabilities could increase alerting authorities’ confidence in the system and help make these authorities more informed users of IPAWS.

FEMA Provides Training and Resources but Lacks Documented Next Steps to Address Authorities’ Challenges

The IPAWS Modernization Act requires FEMA to instruct and educate federal, state, tribal, and local government officials in the use of IPAWS. FEMA has multiple efforts underway to educate and train alerting authorities about IPAWS.

Training. Through FEMA’s Emergency Management Institute, FEMA offers training courses on IPAWS, including a mandatory course that IPAWS applicants must take before they can become authorized users of the system. FEMA is revising its training, according to FEMA officials, and they estimated that the new courses will be available midway through 2020.

Online resources. On a regular basis, FEMA emails tips and conducts webinars, which are recorded and made available online. FEMA has developed a library of IPAWS resources, including a toolkit and fact sheets. FEMA also created an online collaboration group for IPAWS users to share information and best practices and plans to expand the capabilities of this group, according to FEMA.

Testing environment. FEMA created a controlled testing environment called the IPAWS lab that alerting authorities can use to send test alerts and receive hands-on or remote assistance from FEMA staff. According to FEMA, demand for IPAWS lab support has increased, and FEMA hosted more than 200 sessions with IPAWS users in calendar year 2018. FEMA implemented a new requirement in October 2019 for all alerting authorities to send a monthly test alert using the IPAWS lab and upgraded the IPAWS lab environment to support the increased testing.

In-person presentations. FEMA officials regularly present at public safety conferences and other events and use these opportunities to share information about IPAWS and encourage potential new users.

FEMA has also assessed alerting authorities’ educational needs, but it has not fully addressed the recommendations it identified to support these needs or developed plans for ongoing assessments. In 2017 FEMA conducted an analysis—interviewing a sample of alerting authorities and assessing their responses to identify common challenges in using IPAWS. FEMA found that alerting authorities need more training and practice in using IPAWS and experience challenges with using their alerting software, among other things. Our interviews with selected alerting authorities and software providers revealed similar concerns, including that for some a lack of confidence is a potential barrier in using IPAWS. For example, representatives from two of the three alerting software providers we interviewed told us they have issued alerts through IPAWS at the request of their customers. According to these representatives, alerting authorities turn to their software providers as experienced users of the system because authorities have limited local staff, or if they cannot send an alert because of a technical reason. Four academics we interviewed said that FEMA should provide additional training for alerting authorities that is focused on drafting effective messages. Less than 20 percent of state, territorial, and local alerting authorities have sent WEA messages as of October 1, 2019. The limited use of IPAWS could lead to decreased proficiency or confidence. For example, an official from one alerting authority told us the jurisdiction did not use IPAWS at first because officials were not confident about using it.

Our analysis of available information found that FEMA has addressed 4 of the 31 recommendations in its 2017 analysis. For example, FEMA revised its IPAWS training and added software requirements to its memorandum of agreement with alerting authorities. However, the extent to which FEMA has addressed other potentially useful recommendations is not clear because FEMA has not developed a plan to address the recommended actions. For example, one priority recommendation was to create skills checklists that provide a complete inventory of the types of skills alerting authorities need to use IPAWS. FEMA officials told us they had addressed many of the challenges identified in the 2017 analysis, including developing some timelines. However, FEMA did not provide documentation about how all the recommendations would be addressed.

FEMA officials also told us they intend to periodically obtain information from alerting authorities about their needs and have begun another round of interviews with alerting authorities. However, these plans have not been documented. FEMA officials said they also use other methods to keep abreast of educational needs and challenges, such as attending conferences and reaching out to their contacts at emergency management associations that represent alerting authorities. In addition, alerting authorities send comments and feedback via email, according to FEMA officials. However, FEMA did not provide documentation about how it uses information obtained from these methods.

As discussed earlier, FEMA is required by statute to educate federal, state, tribal, and local government officials. FEMA’s IPAWS strategic plan also includes a goal to make emergency alerting more effective, which as the plan explains, requires FEMA to engage non-federal alerting authorities to build competence and promote hands-on familiarity with IPAWS. The FEMA National Advisory Council has emphasized these points, recommending that FEMA improve alerting authorities’ ability to transmit effective alerts by developing and providing education, guidance, and best practices on how to use IPAWS as effective emergency messaging. Federal standards for internal control state that management should externally communicate necessary quality information. Open two-way external reporting lines allow for this communication. For example, management obtains quality information from external stakeholders—which in FEMA’s case would be information from alerting authorities—using established reporting lines. Additionally, federal internal control standards state that documentation provides a means to retain organizational knowledge and mitigate the risk of having that knowledge limited to a few personnel.

Documenting how FEMA plans to address key recommendations from its 2017 analysis could help guide its efforts to educate alerting authorities and hold it accountable for addressing identified needs. Without a documented plan, FEMA may not systematically implement each recommendation, which could result in alerting authorities continuing to struggle with known challenges. In addition, by continuing its analytical efforts and implementing a mechanism to regularly obtain and analyze alerting authorities’ needs, FEMA could learn if these needs are changing and develop educational efforts to address them. Taking such actions will help FEMA enhance alerting authorities’ proficiency with, and confidence in, using IPAWS.

FEMA Has Taken Steps to Increase IPAWS Adoption but Faces Challenges Addressing Pending Applications

FEMA has identified increasing adoption of the system and assisting authorities in gaining access to IPAWS as strategic goals. In addition, in June 2019 the FEMA Administrator issued a “call to action” policy memorandum to FEMA’s regional offices to help improve IPAWS adoption at the local level. As described earlier, FEMA has taken various steps in recent years to increase the adoption of IPAWS, for example, by informing local public safety agencies about IPAWS at conferences and encouraging them to apply for access to the system. In addition, FEMA has developed resources that are available on the IPAWS website that describe the expectations and steps for how a public safety agency may apply to become an IPAWS alerting authority. The number of authorities completing an initial step in the application process to obtain access to IPAWS has increased in recent years from 52 applicants in 2017 to 104 applicants in 2018 and to 122 applicants from January 2019 to September 2019.

However, while more agencies are starting the application process, our analysis of FEMA data found that 430 IPAWS applications were pending as of September 2019, some of which dated back to 2012. Our analysis found that 152 applicants, or about one-third of the 430 applications, began the process (initiated the memorandum of agreement process) from 2012 to 2016. In addition, some applicants had yet to complete the key initial steps in the process. For example, after completing the required IPAWS web-based training and procuring IPAWS compatible software, public safety agencies must return a signed memorandum of agreement to FEMA before the application can move forward.

We found that FEMA sent a draft memorandum of agreement to 108 applicants between 2014 and 2017 that had not yet returned the agreement to the agency as of September 2019. This could indicate that several applicants may be stalled in the early stages of the process and may benefit from FEMA’s assistance in completing the application or answering questions. FEMA officials said that once a completed application is received, approving it should take about 30 days but that factors outside FEMA’s control can contribute to processing delays and thereby increase the number of pending applications. For example, FEMA officials said it is out of their control when applicants do not return signed memorandums of agreement because that step of the process is handled at the state and local level. Representatives from an IPAWS applicant we interviewed said the amount of time it took to receive approval from the state authority was one of the reasons that their application was delayed.

Although delays involving certain applications may be out of FEMA’s control, FEMA may be able to help other applicants. However, FEMA provided no evidence that it had followed up with applicants, when it had last contacted them, or how follow up should be prioritized. FEMA officials said one employee serves as a primary lead for managing the entire application process, which would require a labor-intensive process of following up with hundreds of applicants. FEMA officials said that managing pending applications is a challenge for the IPAWS office due to resource constraints.

To help address these constraints, in 2019, FEMA awarded a contract to begin developing a new tool with the goal of streamlining FEMA’s management of applications. Officials said they anticipate the tool, estimated to be available in early 2020, will help them better manage the pending applications and conduct outreach as well as move new applications through the process. In 2016 FEMA conducted a study of the IPAWS application process and highlighted certain factors that contributed to an increasing backlog and response time, including FEMA officials not knowing that a follow up task for an applicant was waiting to be addressed. The study further indicated that determining the next step was manual and often reactive. Officials also said that staff will be able to run an aging report on applications to help them prioritize follow-up efforts. However, the agency has not established procedures to prioritize and follow up with applicants. FEMA officials acknowledged that establishing procedures to prioritize and follow up on the in-process applications would be beneficial. While these applications are pending, people in areas that are not covered by IPAWS authorities may not receive critical alerts and warnings from local authorities through IPAWS.

Conclusions

Effective emergency alerting is vital to helping save lives and property during natural disasters and other threats to public safety, highlighting the importance of IPAWS as a way to disseminate critical information. However, FCC lacks specific goals and performance measures and FEMA lacks plans and processes, which may contribute to decreased confidence in and use of IPAWS by alerting authorities. In particular, because FCC does not have specific goals and performance measures to monitor WEA improvements, FCC will have difficulty assuring that these improvements are working as intended during emergencies and identifying areas where performance is lacking, which could undermine authorities’ confidence in using IPAWS. In addition, because FEMA has not documented next steps or plans for educating alerting authorities and establishing a process to regularly assess their educational needs, some authorities may continue to lack proficiency and confidence in using IPAWS. Furthermore, absent a strategy to address the substantial number of pending IPAWS applications, FEMA’s efforts to increase IPAWS adoption and expand alerting coverage are hindered.

Recommendations for Executive Action

We are making a total of three recommendations, including one to FCC and two to FEMA. Specifically: The Chairman of FCC should develop specific, measurable goals and performance measures for its efforts to monitor the performance of new WEA capabilities, such as enhanced geo-targeting and expanded alert message length. (Recommendation 1)

The Director of the IPAWS program should document how it plans to address key actions needed to educate alerting authorities in their use of IPAWS and implement a mechanism that will allow FEMA to regularly and systematically obtain and analyze feedback on alerting authorities’ educational needs. (Recommendation 2)

The Director of the IPAWS program should establish procedures to prioritize pending IPAWS applications and to follow up with applicants to address these applications. (Recommendation 3)

Agency Comments

We provided a draft of this report to FCC, the Departments of Homeland Security (FEMA), Commerce (NOAA), and the Interior (USGS) for review and comment. FCC and the Department of Homeland Security provided written comments, reprinted in appendixes IV and V respectively. FCC, FEMA, and NOAA provided technical comments, which we incorporated as appropriate.

In its written comments, FCC stated that it agreed with us on the importance of gathering and assessing specific performance information about the effectiveness of WEA capabilities. Separately, FCC officials noted that FCC was taking steps to gather this data, which will help inform the development of metrics, as we recommended. In its written comments, DHS concurred with our two recommendations to FEMA and provided information about activities that FEMA would undertake to implement them.

We are sending copies of this report to the appropriate congressional committees, Chairman of FCC, Secretaries of Homeland Security, Commerce, and the Interior, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-2834 or Vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

This report examines (1) the trends in how alerting authorities use and test IPAWS and their experiences using IPAWS, and (2) actions that FEMA and FCC have taken to modernize IPAWS and increase its adoption, and the challenges they face.

For background information on emergency alerting, we identified key issues and federal roles and responsibilities by reviewing applicable laws and regulations, our prior work, and reports prepared by FEMA, FCC, the Department of Homeland Security’s (DHS) Office of Inspector General, the Congressional Research Service, and academics. We also identified recent trends regarding natural disasters and the use of digital devices and the internet that could affect the use and frequency of emergency alerting. To identify natural disaster trends, we reviewed our prior work, a 2018 report prepared by the U.S. Global Change Research Program, and information on wildfires prepared by the California Department of Forestry and Fire Protection. We identified trends about the use of digital devices and the internet by reviewing surveys conducted from 2017 to 2019 by the Pew Research Center and The Nielsen Company, which regularly conduct national surveys on those topics. We also reviewed proposed federal legislation on emergency alerting.

To examine the use of IPAWS and selected alerting authorities’ experiences using IPAWS, we analyzed IPAWS access and usage throughout the country from 2012 to 2019. We focused on identifying the authorities that used IPAWS from 2017 to 2019 following the passage of the IPAWS Modernization Act of 2015 (enacted in 2016). We analyzed IPAWS testing by judgmentally selecting samples of authorities conducting tests. In our calculations of the number of alerts issued by state, territorial, tribal, and local authorities, we focused on alerts for disasters and threats to public safety and excluded test alerts and alerts for missing persons and child abductions. We reviewed FEMA’s processes for ensuring the completeness and reliability of these alerting data and determined that they were sufficiently reliable for the purposes of examining trends in the use of emergency alerts. We also reviewed reports by FCC and local authorities on EAS and WEA test results. To obtain information on alerting authorities’ experiences using IPAWS, we conducted seven case studies of emergency events. To select them, we analyzed alerts that local authorities issued through IPAWS, FEMA’s list of federally declared disasters from 2017 to 2019, NOAA’s list of billion dollar disasters from 2017 to 2019, and our prior work on natural disaster preparedness and recovery from 2017 to 2019; considered recommendations from stakeholders; conducted internet searches; and reviewed news reports. We selected these case studies to include various areas of the country that experienced different types of disasters and threats to public safety during this time. These included natural disasters (wildfires and an earthquake), weather events (a hurricane and a flood), manmade disasters (a chemical fire and a power shortage), and a law enforcement event (a suspicious package). We then interviewed local alerting authorities in those areas. As a test case study, we interviewed District of Columbia emergency management officials. We conducted site visits with state and local emergency management officials in Los Angeles and Ventura, California; Bristol, Panama City, and Tallahassee, Florida; and Washington, D.C.

To examine the actions that FEMA and FCC have taken to modernize IPAWS and increase its adoption, and the challenges they face, we reviewed FEMA documents such as IPAWS strategic plans and a performance report; FCC regulations, notices, and comments on FCC proposed rulemakings regarding EAS and WEA; and assessed the information against statutory requirements contained in the IPAWS Modernization Act and federal internal control standards. We focused on four areas of the Act that were key in the implementation of the program. These areas required FEMA, in consultation and coordination with FCC, to: ensure that IPAWS is capable of distributing alerts on the basis of geographic location, risks, and technologies; educate state, tribal, and local governments to understand how IPAWS works, and how and when to use IPAWS; establish training opportunities for alerting authorities; and conduct nationwide tests of IPAWS alerts.

We compared FCC’s actions to leading practices based on the Government Performance and Results Act of 1993 (GPRA) as enhanced by the GPRA Modernization Act of 2010 (GPRAMA), which create a framework of goal setting and performance management for federal agencies. While GPRA and GPRAMA apply to the department or agency level, we have previously reported that their provisions can serve as leading practices at other organizational levels, such as component agencies, offices, programs, and projects. We also reviewed recommendations in reports prepared by the FEMA National Advisory Council IPAWS Subcommittee and FCC’s Communications Security, Reliability, and Interoperability Council, and disaster after-action reports prepared by FEMA and state and local governments.

As an additional step in assessing the challenges that FEMA faces in increasing IPAWS adoption, we analyzed FEMA’s pending IPAWS applications as of September 2019 to determine which steps in the application and approval process had been completed and how long the applications were in process. We also interviewed four selected IPAWS applicants to obtain their views on the application process. To obtain a variety of perspectives, we selected applicants that were different types of organizations (an airport, a university, a local government, and a federal agency) in different areas of the country. In addition, for both objectives, we interviewed officials from FEMA, FCC, NOAA, USGS, and 18 state, local and territorial alerting authorities; representatives from 4 industry associations, 2 advocacy groups, and 15 companies, including wireless carriers, internet service providers, internet content providers, IPAWS software providers, and mobile device manufacturers; and 7 academics. To obtain a variety of perspectives, we selected industry associations and companies that represented different telecommunications industry sectors and have different roles in emergency alerting (broadcasting, cable, wireless, internet service, and application developers) and academics with different areas of expertise (public health, engineering, natural hazards, disaster preparedness, rural emergency management, and communication). We also interviewed staff from a county board that oversees emergency management activities in that jurisdiction and officials from a city that is planning to apply for IPAWS access. The results of these interviews are not generalizable to all stakeholders, but provide insight on the use of IPAWS and related emergency alerting issues. Our interviewees are listed in table 2 below.

Appendix II: Types of Wireless Emergency Alerts

Appendix III: Case Studies Regarding the Use of IPAWS

Selected Alert Sent by the Florida Division of Emergency Management for Bay County during Hurricane Michael: October 10, 2018: GOVERNOR EVAC ALERT 6 to 13 FT STORM SURGE EXPECTED IN BAY COUNTY Zones A,B,C EVACUATE NOW Selected alerts sent by Bay County Emergency Services during Hurricane Michael: October 10, 2018: Dangerous w inds are beginning to occur in Bay County Shelter in place now October 15, 2018: Bay County remains under a boil w ater notice. Please boil or use bottled w ater for consumption. October 15, 2018: FOOD AND OR WATER ARE AVAILABLE NORTH OF 15th AND CR 386 AT 1011 CR 386 SOUTH Bay County officials said the county lost its ability to issue alerts at this point.

Hurricane Michael, October 2018: Hurricane Michael was a category 5 storm that NWS reported made a catastrophic landfall near Mexico Beach and Tyndall Air Force Base, Florida, producing devastating winds and storm surge near the coast, and rain and wind inland (see fig. 6). According to a State of Florida report, Hurricane Michael was the most powerful storm to hit the Panhandle region and the third most intense storm to make landfall in the mainland United States in recorded history. During the storm, several counties could not send alerts because of power outages and inoperable cellular towers.

Officials from an alerting authority we interviewed in Florida commended the state’s ability to send IPAWS alerts on behalf of the county, which had lost its communications capabilities during the storm. Authorities also said IPAWS provides an ability to warn the public about approaching hurricanes and share critical lifesaving information such as the location of food, water, and shelter. However, authorities expressed frustration about the inability to accurately geo-target WEA messages to evacuation zones and about how the WEA text character limit forced them to issue multiple WEA messages regarding the same alert. Some officials said they were frustrated when certain local EAS alerts were not delivered by broadcasters, which could prevent some people from receiving them.

Alert issued by the California Governor’s Office of Emergency Services: Dec. 6, 2017: Strong w inds over night creating extreme fire danger. Stay alert. Listen to authorities. Alert issued by the Ventura County Sheriff’s Office – Office of Emergency Services: Dec. 7, 2017: VENTURA COUNTY-FAST MOVING BRUSH FIRE NORTH OF OJAI.GO TO READYVENTURACOUNTY.ORG FOR INFO Selected alerts issued by the City of Los Angeles: Dec. 6, 2017: For information regarding the Skirball Fire in Los Angeles please go to Tw itter.com/LAFD Nov. 9, 2018: MANDATORY EVACUATION in West Hills: W of Valley Circle, N to Roscoe Blvd, S to Vanowen. Selected alert issued by Santa Barbara County: Dec. 16, 2017: EVAC ORDER: SB City: east of Hw y 154 to Mission Canyon Rd and north of 192. Leave now .

Southern California Wildfires, December 2017 and November 2018: The southern California area experienced large wildfires in recent years, including the Thomas fire in December 2017 and the Woolsey fire in November 2018 (see fig. 7). The California Department of Forestry and Fire Protection reported in August 2019 that the Thomas fire, which affected Santa Barbara and Ventura Counties, was the second-largest wildfire in the state’s history and destroyed more than 1,000 structures. The Woolsey fire, which affected Los Angeles and Ventura Counties, had a footprint over 150 square miles and resulted in the evacuation of about a quarter-million people. According to Los Angeles County, the Woolsey fire was the most destructive fire in the county’s history.

California officials we interviewed said IPAWS is an effective tool for wildfire evacuations and that because most people have cell phones, they do not have to subscribe to receive WEA messages. Officials also praised the capability of IPAWS to allow a state alerting authority to send alerts to at-risk counties ahead of potential wildfires. However, officials said it is a challenge to know when and where other alerting authorities in the area are sending alerts and that there may be little time. For example, an official told us that the Thomas fire moved at 60 miles per hour. Officials also said that even though WEA messages were targeted to an area during the fires, they did not know whether people received them because geo-targeting was not precise and because cell towers may have been damaged.

Alerts issued by the New York City Emergency Management Department on Oct. 24, 2018: Police Activity: Residents on W 58th St btw Columbus & 8th Av shelter in place immediately The suspicious device on W 58 St & 8th Ave w as safely removed by NYPD Bomb Squad.

Suspicious Package in New York City, October 24, 2018: On October 24, 2018, the New York City Emergency Management Department issued a WEA shelter-in-place order regarding a suspicious package at the Time Warner Center in Manhattan that was found to contain an improvised explosive device (see fig. 8). According to officials, police removed the device and determined it was no longer a threat. About an hour after the initial alert was issued, the city issued another WEA canceling the shelter- in-place alert.

New York City officials said IPAWS is the city’s most effective alert and warning tool, compared with its own alerting system to which about 9 percent of the population has subscribed. Regarding the suspicious package, authorities were able to draw an alerting area covering a 3 to 4 block radius. The officials also said that WEA messages were instrumental in helping to capture a suspect in a bombing incident in the city’s Chelsea neighborhood in 2016. However, officials said the October 2018 alert was received as far as 4 miles from the targeted area, which led them to speculate about the number of people who received the alert. Officials also said they would like IPAWS to incorporate more languages for use in alerts and provide them with the ability to use photographs or maps in future alerts.

Alerts issued by the Douglas County Emergency Management Agency: March 14, 2019: From Douglas County Sheriffs Office. Record flooding on Elkhorn River. Evacuate Now . March 15, 2019: From Douglas County Sheriff: Evacuate City of Valley NOW. Use Q Street. Hw y 275 closed. Alert Issued by Platte County Emergency Management: March 14, 2019: Travel is not advised in and around Columbus and Platte County due to extensive flooding.

Flood in Nebraska, March 2019: In March 2019, Nebraska experienced one of the most devastating floods in recent history, according to the state government (see fig. 9). We interviewed officials in Douglas County and Platte County, areas that experienced torrential rain and flooding. One county sent a geo-targeted WEA evacuation alert to people living near a river while the other county sent a WEA advising the public to not travel within the county.

A county official in Nebraska said that if the planned future enhancements to WEA take place and are found to be successful, WEA will ultimately be of greater value than other means of notification such as the county’s previous subscription system, which had a low participation rate. The official stressed the difficulty in explaining the threat, the source of the alert, and a protective action within the 90 character WEA limit. The official also noted that some local broadcasters were not equipped to recognize an EAS law enforcement alert for further transmission. An official in another county said that some people did not receive the WEA messages.

Power shortage in Michigan, January 2019: On January 30, 2019, Consumers Energy, a primary energy supplier in Michigan, experienced a fire at a natural gas storage facility at a time when there was high energy demand because of extreme cold temperatures (see fig. 10). According to NWS, Michigan’s Lower Peninsula experienced the lowest temperatures in decades—down to minus 20 degrees with wind chills of down to minus 40 degrees. As a result, the state’s Emergency Operations Center asked the Michigan State Police, an IPAWS alerting authority, to issue WEA and EAS alerts asking people to lower their thermostats to conserve natural gas.

Michigan State Police officials said that IPAWS allowed the alerting authority to send a WEA message to 68 counties, which was an effective and quick way to reach many people. However, officials said they attempted to send an EAS alert to all 68 counties in Michigan’s Lower Peninsula but were limited to a total of 31 counties per alert, per FCC regulations. They said that after the EAS alert was sent, the actual EAS broadcast message was not displayed on television because the entire list of the 31 county names, which must be read first according to FCC regulations, took up the allowable 2-minute time span for an EAS broadcast.

Alerts issued by the City of Houston on Sept. 20, 2017: Shelter in Place in northw est Spring Branch due to hazardous fire. Check local media. Shelter in place is CANCELLED for Northw est Spring Branch after hazardous fire.

Chemical Fire in Houston, Texas, September 20, 2017: The Houston Fire Department requested that the Houston Office of Emergency Management issue a WEA shelter-in-place order following a chemical fire at a bearing supply company that resulted in the release of potentially hazardous smoke (see fig. 11).

Houston officials said they believe that IPAWS allowed the alerting authority to reach a broad area at risk using the WEA message. However, officials said it is possible that a lack of training on behalf of the alerting authority, among other things, limited their ability to issue the alert in a timely fashion. They said it took the alerting authority 43 minutes and multiple attempts to properly prepare and send the message using its IPAWS-compatible software before the message was successfully sent to the public.

Earthquake in Alaska, November 30, 2018: A magnitude 7.0 earthquake struck north of Anchorage, Alaska, on November 30, 2018 (see fig. 12). We interviewed officials from three local governments that were affected by the earthquake. Officials at one borough said they did not issue an IPAWS alert because the earthquake had a short intensity and they did not receive reports of fatalities or widespread damage. However, the officials said that if the earthquake’s intensity had been greater, they would have issued used IPAWS to alert people about shelter locations. NWS used IPAWS to issue a tsunami warning but local officials did not issue any alerts through IPAWS.

Officials in Alaska said that it is helpful that another government agency can be a backup alerting authority and provide alerts through IPAWS on behalf of the local government. However, an official said the inability to precisely geo-target alerts about tsunami risks to coastal areas prevented the official from sending out an alert due to concerns that people who were not affected by the earthquake would receive the alert. Another official said the cost of procuring alerting software that is compatible with IPAWS may be a challenge for some local governments.

Appendix IV: Comments from the Federal Communications Commission

Appendix V: Comments from the U.S. Department of Homeland Security

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, Sally Moino (Assistant Director); Michael Sweet (Analyst in Charge); David Aja; Melissa Bodeau; Mark Goldstein; Bob Homan; Kate Perl; Cheryl Peterson; Sam Portnow; Malika Rice; and Andrew Stavisky made key contributions to this report.