Coastal Zone Management: Opportunities Exist for NOAA to Enhance Its Use of Performance Information
Released: 2014-07-16T00:00:00

Letter

Background

Congress enacted the Coastal Zone Management Act in 1972 to balance the often competing demands for economic growth and development with the need to protect coastal resources. To accomplish the goals of the act, Congress established a framework for a voluntary federal and state coastal management partnership, the CZMP. The CZMP represents a unique federal-state partnership for protecting, restoring, and responsibly developing the nation’s coastal communities and resources, according to program documents.

The act identifies specific goals for state programs that fall into six broad focus areas ranging from protecting and restoring coastal habitat to assisting with coastal community development efforts and improving government coordination and decision making (see table 1).

States must submit comprehensive descriptions of their coastal management programs—which must be approved by the states’ governors—to NOAA for review and approval. As specified in the act, states must meet the following requirements to receive NOAA’s approval for their state programs, among others: designate coastal zone boundaries that will be subject to state define what constitutes permissible land and water uses in coastal propose an organizational structure for implementing the state program, including the responsibilities of and relationships among local, state, regional, and interstate agencies; and demonstrate sufficient legal authorities to carry out the objectives and policies of the state program, including the means by which a state will regulate land and water uses, control development, and resolve conflicts among competing activities in coastal zones to ensure their wise use.

The act provides states the flexibility to design programs that best address states’ unique coastal challenges, laws, and regulations, and participating states have taken various approaches to developing and carrying out their programs. For instance, there are generally two organizational structures used by states to implement their programs: (1) networked programs, which rely on multiple state and local agencies to implement their programs, and (2) non-networked, or comprehensive state programs that administer all aspects of the program through a single centralized agency. The coastal management activities carried out also vary across states with some states focusing on permitting, mitigation, and enforcement activities, while other states focus on providing technical and financial assistance to local governments and nonprofits for local coastal protection and management projects. If states make changes to their programs, such as changes in their coastal zone boundaries or organizational structures, the states must submit those changes to NOAA for review and approval.

The act includes two primary incentives to encourage states to develop coastal management programs and participate in the CZMP. First, participating states are eligible to receive federal funding from NOAA to support the implementation and management of their programs, which the agency receives annually through congressional appropriations. In fiscal year 2013, NOAA awarded participating states a total of approximately $61.3 million, a 9 percent decline from fiscal year 2008 awards, when it awarded just over $67.5 million across participating states. NOAA awards CZMP funding to individual states across three fund types—administrative, enhancement, and coastal nonpoint program—according to requirements in the act (see table 2). The majority of funding NOAA awards through the CZMP is administrative funding. Administrative funding, which requires state matching funds, supports general implementation of the state’s coastal management program. Under the act, NOAA may also award a maximum of $10 million annually in enhancement program funding to participating states. Enhancement funding is to be used by states to develop program changes, or enhancements, to their NOAA-approved programs in one or more of nine enhancement objectives specified in the act, as listed in table 2. In addition, Congress has generally provided direction on the total amount of funds to be awarded through the coastal nonpoint program to assist with states’ coastal nonpoint pollution control programs, which are programs to ensure states have necessary tools and enforceable authorities to prevent and control polluted runoff in coastal areas. According to NOAA officials, funding has not been provided for this program since fiscal year 2009, when nearly $3.4 million was awarded to states. States may also use other sources of funding for their coastal nonpoint pollution control programs, including administrative and enhancement funding.

Second, federal agency activities in or affecting the uses or resources of a participating state’s defined coastal zone are required to be consistent to the maximum extent practicable with enforceable policies of the state’s program. Under this provision, known as federal consistency, states with approved programs must have the opportunity to review proposed federal actions for consistency with enforceable policies of their state programs. Types of federal actions that may be reviewed by states include federal agency activities, such as improvements made to a military base; licenses or permits to nonfederal applicants; financial assistance to state and local governments; and outer continental shelf activities, such as oil and gas development. If a state finds that a federal activity is not consistent with the state’s enforceable policies, the state can object to the activity and work with the federal agency to resolve any differences between the proposed activity and state policies. All participating state programs have developed federal consistency review processes.

Thirty-four out of 35 eligible states have federally approved coastal management programs (see fig. 1). Most state programs have been in existence for more than 30 years, with the earliest program approved in 1976, and 29 states having received federal approval for their programs by 1986. The most recent state to begin participating in the program is Illinois, which received federal approval in January 2012.

NOAA’s Office of Ocean and Coastal Resource Management (OCRM) is responsible for general administration and oversight of the CZMP. NOAA plans to merge the OCRM with its Coastal Services Center—an office that provides coastal-related mapping tools and data; training on various coastal management issues such as climate adaptation and coastal restoration design and evaluation; and technical and other assistance to local, state, and regional coastal organizations—into a single office by the end of 2014. Under the current and planned office structure, NOAA officials are responsible for approving state programs and any program changes, administering federal funding to the states, providing technical assistance to states such as on the development of 5-year assessment and strategy reports that identify states’ priority needs and projects to address one or more of nine enhancement objectives required for enhancement funding, among other topics, and managing the CZMP performance measurement system. NOAA assigns coastal management specialists to work with individual state programs. As part of its administration of the program, NOAA evaluates program performance using its CZMP performance measurement system. NOAA began developing a framework for this performance measurement system in 2001, started piloting it in 2004, and fully implemented the system by 2008. The system consists of 15 performance measures that generally correspond with the goals of the act, and two additional measures to track state financial expenditures. The 17 total performance measures incorporate individual data elements, plus additional subcategories of information that state programs collect and report into the system annually (see app. II).

In addition, NOAA evaluators, who are in a different NOAA division than specialists, are responsible for conducting individual state program evaluations, which are required under the act. State program evaluations are designed to examine the extent to which states have: (1) implemented their approved programs,(2) addressed coastal management needs identified in the act, and (3) adhered to the terms of CZMP funds awarded through cooperative agreements. NOAA’s state program evaluation reports identify state accomplishments and make recommendations for improving states’ programs. NOAA’s recommendations are classified as either necessary actions—actions a state must take by a specific date such as the next regularly scheduled evaluation—or program suggestions—actions it believes a state should take to improve its program. NOAA may withdraw approval for a state’s program and financial assistance in cases where states do not address necessary actions. NOAA has not withdrawn approval for a state program as of the end of fiscal year 2013 and, according to NOAA officials, few necessary actions have been identified in past state evaluations.

In 2008, we examined NOAA’s process for awarding financial assistance to states and how the agency evaluated the effectiveness of the CZMP. Of the seven recommendations we made in 2008, NOAA disagreed with one recommendation that the agency develop performance measures to evaluate the effectiveness of state programs in improving processes; NOAA agreed with the other six recommendations and has taken some actions to address them as described in table 3.

States Allocated Nearly $400 Million in CZMP Funds during Fiscal Years 2008 through 2013 for a Wide Range of Coastal Management Activities

During fiscal years 2008 through 2013, the 34 participating states allocated a total of nearly $400 million in CZMP funds for a variety of activities, generally related to the broad goals for state programs outlined in the Coastal Zone Management Act. Each year, NOAA analyzes its cooperative agreements with states for CZMP funding, and categorizes the states’ CZMP funding allocations as they correspond with the six focus areas based on the broad goals in the act, along with a seventh category to capture state program administrative costs, such as general program operations, supplies, and rent. According to NOAA’s analysis, during fiscal years 2008 through 2013, states’ allocations of CZMP funds varied across the seven categories, with about half concentrated in support of activities related to two focus areas, government coordination and coastal habitat (see fig. 2).

NOAA officials told us that, while states have the flexibility to design and implement programs that best meet their unique needs, the agency does influence how states allocate CZMP funds through (1) NOAA’s review and approval of states’ 5-year assessment and strategy reports required for enhancement funding in which participating states prioritize projects that support program improvements and (2) NOAA’s periodic state program evaluations in which NOAA outlines necessary actions or makes program suggestions that can influence state program activities. NOAA officials said that they also informally shape or influence state program activities through ongoing discussions with state program officials about funding proposals or specific projects, such as how projects might be adjusted to address NOAA priorities.

Examples of activities for which participating states allocated CZMP funds during fiscal years 2008 through 2013 in each of the six focus areas include the following:

Government coordination. States allocated CZMP funds for activities including state and regional planning efforts that involve coordination among multiple levels of government and stakeholders to address complex and controversial coastal issues, such as comprehensive planning of ocean and nearshore areas, energy facility siting or special area management planning; federal consistency activities; technical assistance to local governments; and public outreach and education on coastal issues including website development and publications about a state program’s activities. According to NOAA’s analysis of cooperative agreements with states for CZMP funding, states allocated the largest amount of CZMP funding during the 6-year period—about 27 percent of total funding— to government coordination activities. We found that a number of state programs use CZMP funds to support participation in regional organizations involving ocean planning activities that entail coordination across federal, state, and local governments. For example, state program officials in some Northeast and Mid-Atlantic states participate in regional organizations, such as the Northeast Regional Ocean Council and Mid-Atlantic Regional Council on the Ocean, that have ocean resource data collection and planning efforts under way. We also found that most states we reviewed provide some type of technical or financial assistance to local governments to support local level coastal management activities and projects.

Protecting Coastal Habitat in Texas The Texas state program used coastal zone funds to support a multiyear marsh restoration project on the Texas Gulf Coast near Corpus Christi. Over the past 60 years, about 340 acres of coastal marsh habitat were lost due to the construction of an adjacent highway and subsequent erosion. A local nonprofit organization began restoring the marsh in 2005. The project involved scooping sand, clay, and shells from the bay bottom and piling the material into terraces and mounds; planting native grasses on the terraces to stabilize the structures and provide habitat; and constructing an outer rock berm to protect the new marsh area from strong waves in the bay, as shown below. Project officials told us Texas’s state program provided about $1 million in coastal zone funding, about 20 percent of the project’s total cost, to the nonprofit organization responsible for the project. Other funding to carry out the project was provided by the EPA, U.S. Fish and Wildlife Service, state government sources, and grants from private foundations. According to project officials, the project was completed in spring 2014 and has resulted in 160 acres of restored marsh that provide habitat for fish, crabs, shrimp, nesting birds, sea grass, and other plants and animals. The project also resulted in the creation of new opportunities for public recreation, such as fishing and kayaking, and the marsh protects the adjacent highway from coastal hazards, such as storms, according to project officials.

Coastal habitat. States allocated CZMP funds for coastal habitat protection and restoration activities including the acquisition or placement of easements on coastal lands; restoration of coastal habitats; data collection and mapping of coastal habitats; development of plans for habitat acquisition, restoration, and other habitat management needs; implementation of permitting and enforcement programs that protect coastal habitat through planning and regulation of development; or support of land management programs such as those for coastal preserves and parks. States also allocated CZMP funds for public outreach and education activities that focused on coastal habitat protection and restoration. According to NOAA’s analysis, approximately 24 percent of CZMP funds awarded during fiscal years 2008 through 2013 were allocated to coastal habitat protection and restoration activities. According to NOAA’s CZMP performance measurement system data from 2008 through 2013, states reported that they used CZMP funds to protect nearly 23,300 acres of coastal habitat through acquisition or easement, restore nearly 37,400 acres of coastal habitat, and through regulatory programs protect more than 123,000 net acres of coastal habitat.

Coastal hazards. States allocated CZMP funds for activities that help coastal communities minimize risks from coastal hazards, such as storms, tsunamis, and sea-level rise, and improve hazard awareness and understanding. Such activities include assessment and planning efforts, such as developing mitigation plans, risk and vulnerability assessments, and data collection and mapping to identify and manage development in areas vulnerable to coastal hazards; implementation of hazard mitigation projects; implementation and enforcement of hazard policies, regulations, and requirements; and education and training on coastal hazard topics. According to NOAA’s analysis of cooperative agreements with states for CZMP funding, about 13 percent of CZMP funds awarded in fiscal years 2008 through 2013 were allocated for coastal hazards projects. The coastal hazards focus area was the one focus area where the share of CZMP funds allocated steadily increased over the 6-year period, from roughly 7 percent in fiscal year 2008 to about 16 percent in fiscal year 2013. Most state program officials we spoke with identified their work to help communities reduce future damage from hazardous events and impacts from sea-level rise related to climate change as among their more significant projects. NOAA also identified coastal hazards work as a priority area, and in 2011, through the agency’s funding guidance, began encouraging states to use CZMP funding for projects that improve the resiliency of coastal communities to adapt to the impacts of coastal hazards and climate change. In addition, many of the projects that were awarded funding under the competitive Projects of Special Merit Program in fiscal years 2012 and 2013 were identified by states as addressing, at least in part, coastal hazards, according to NOAA officials. For example, South Carolina’s project to study tidal inlet dynamics and erosion and Maine’s adaptation planning project for its coastal parks both addressed coastal hazard issues. NOAA’s CZMP performance measurement system data for 2008 through 2013 show that states reported working with more than 410 communities to reduce risks from coastal hazards and nearly 230 communities to improve public awareness of coastal hazards issues.

A Coastal Water Quality Monitoring and Modeling Project in Florida Estuaries—such as Sarasota Bay, that spans about 56 miles along the southwest Florida coast—are important productive ecosystems that provide habitat for a diversity of species. Nonpoint source pollution carried through runoff influences the health of the Sarasota Bay, which has limited tidal flushing, no major tributary, and receives most of its freshwater from rainfall and associated runoff. Florida’s coastal management program provided nearly $150,000 in coastal zone funds to support a multiyear water quality monitoring and modeling study in Sarasota Bay led by the Florida Fish and Wildlife Research Institute. The study was designed to help determine major factors affecting the ecological health of the bay. Specifically, coastal zone funding was used for statistical modeling to differentiate between the effects of polluted runoff into the bay during storm events from the effects of natural algal, or other natural sources of nutrients, in the bay. Florida state program officials told us that understanding ecological responses in estuaries can facilitate planning to minimize potential impacts and help maintain overall ecosystem health. Continued water quality monitoring and modeling is being completed in the bay with other funding sources, according to Florida officials.

Coastal water quality. States allocated CZMP funds for water quality permitting and enforcement activities such as permitting of storm water discharges; activities and projects related to water quality management including vegetative plantings or other nonstructural shoreline erosion control projects; water quality monitoring; activities and projects for local governments to improve water quality management; technical assistance, data collection, mapping, planning, and policy development to address water quality issues; marine debris and other coastal cleanup or pollution prevention programs; and projects and activities that provide technical assistance to marinas to reduce nonpoint source pollution; and public outreach and education on water quality issues. Activities include those that support states in implementing their coastal nonpoint source pollution control programs. According to NOAA’s CZMP performance measurement system data, from 2008 through 2013, states reported that they worked with more than 680 communities to develop nonpoint source pollution management policies and plans, or complete related projects, and removed 27 million pounds of marine debris through coastal cleanup activities.

Coastal community development. States allocated CZMP funds for activities including planning and construction to support the redevelopment of urban waterfronts, ports, and harbors; technical assistance to local governments related to waterfront redevelopment; community planning, land-use planning, green infrastructure planning, and other sustainable development efforts; and public outreach and education activities specific to coastal community development issues. According to CZMP performance measurement system data from 2008 through 2013, states reported that they worked with more than 580 coastal communities to promote development and growth in ways that protect coastal resources and with more than 250 communities to redevelop ports and waterfronts.

Public access. States allocated CZMP funds for activities including creating new public access sites through easements or right of ways; enhancing existing public access through trails, handicap features, or educational signage; developing plans, collecting data, and providing technical assistance to local governments on public access planning; and conducting public outreach and education activities on public access issues. According to NOAA’s analysis, states allocated the least amount of CZMP funding (about 6 percent of total CZMP funding) for activities that improve public access to the coast. Unlike other focus areas, a number of states did not allocate funds for public access. According to NOAA officials, some states may not need to use CZMP funding to support public access projects, for example, because they already have sufficient public access to coastal areas. In total, according to CZMP performance measurement system data from 2008 through 2013, states reported that with CZMP funds and through regulatory programs they helped create nearly 700 new public coastal access sites and helped enhance nearly 1,500 existing sites.

State program officials told us that CZMP funding is important because it can help leverage other financial resources and provides sustained, multiyear funding for projects. We found that CZMP-funded projects and activities often involved partnerships with various entities and used multiple sources of funding. According to state program officials, CZMP funds were often the catalyst for obtaining additional financial assistance or other resources. For example, we visited a $5.2 million, multiyear marsh restoration project along the Texas Gulf coast that received nearly 20 percent of overall project funding through the CZMP and additional financial support from eight other federal, state, and private sources. Representatives from the nonprofit organization responsible for managing the project told us that CZMP funds received during the initial stages helped attract other funding partners needed for such a large-scale restoration project. Similarly, Virginia’s program used $6,000 of its CZMP funding to leverage staff from six partner organizations to plan and conduct a Marine Debris Summit that laid the groundwork for developing a marine debris plan and establish priorities for future work, which state program officials expect will serve as a model for other Mid-Atlantic states. Most of the state programs we reviewed also provide competitive grants or offer other assistance to leverage local resources to address coastal issues. For example, Florida’s program competitively awards a portion of its administrative funds annually through grants to coastal counties and municipalities for projects that help communities address a wide range of coastal issues, and these grants require local entities to match the state grants. Similarly, Maine’s program uses CZMP funds annually to provide competitive grants to coastal communities for planning activities that support harbor management and development or improve shoreline access, but actual implementation of the projects must be funded through other sources.

Limitations Exist with NOAA’s Performance Assessment Tools, and NOAA Makes Limited Use of Performance Information

NOAA’s two primary performance assessment tools, the CZMP performance measurement system and its state program evaluations, have limitations, even with changes NOAA has made since 2008, and NOAA uses the performance information it collects to a limited extent in managing the CZMP. We found that NOAA’s CZMP performance measurement system does not align with some key attributes of successful performance measures. In addition, in its method for selecting stakeholders to survey during state program evaluations, NOAA may be susceptible to collecting incomplete and biased information because, in part, it uses a single criterion to select stakeholders to survey. Furthermore, NOAA makes limited use of the performance information it collects—for instance, NOAA does not use data from its performance measurement system or its evaluations of state programs to improve implementation of the CZMP at the national level—and, as a result, may not be realizing the full benefit of collecting such information.

NOAA’s CZMP Performance Measurement System Does Not Align with Some Key Attributes of Successful Performance Measures

NOAA’s CZMP performance measurement system, which the agency developed in response to congressional direction to assess the national impact of the CZMP, has limitations, even with changes the agency made to the system since our 2008 report. Specifically, NOAA has made changes to several aspects of the data collection and review components of its system, including the following: establishing a requirement, in 2010, that state programs submit documentation of source information to support their data submissions, such as documentation of the public access sites being reported for public access performance measures; refining, in 2009, 2010, and 2011, the names and definitions of some performance measures with the intention of clarifying the activities that a given measure is intended to capture; and issuing internal guidance, in 2010, for NOAA staff to review state- submitted data and accompanying documentation to ensure that only eligible activities are reported by the states, among other things.

With these changes, the system aligns with some key attributes of successful performance measures. In our past work, we found that successful performance measures typically align with key attributes including reliability, clarity, balance, numerical targets, and limited overlap, among others (see app. III for a complete list of key attributes we identified). In our current review, we found that some of the changes NOAA made to its CZMP performance measurement system since 2008 are consistent with such key attributes. For example, NOAA’s requirement that state programs submit documentation of source information and its internal guidance for how staff are to review this documentation correspond with the key attribute of ensuring the reliability of performance measures. In addition, NOAA’s steps to refine the names and definitions of certain performance measures are demonstrative of the key attribute of clarity, meaning that measures are clearly stated and have names and definitions consistent with the methodology used to calculate them.

On the other hand, we found limitations in the CZMP performance measurement system that did not align with the key attributes. For instance, in 2011, NOAA eliminated its coastal water quality focus area— corresponding to one of the six focus areas based on goals of the CZMP outlined in the act. In eliminating this focus area, NOAA removed five related performance measures; states continue to report on one measure related to coastal water quality, but do so under another focus area on coastal community development. Balance, or having a set of measures that cover a program’s various goals, is a key attribute of successful performance measures. We found that having measures that correspond to various program goals provided agencies with a complete picture of performance. NOAA officials indicated that they eliminated the coastal water quality focus area based on a 2011 performance measurement system workgroup’s recommendation to streamline the measurement system. They further explained that they took this action because state programs were no longer receiving coastal nonpoint program funding, which often funded activities in support of coastal water quality, and that activities under this focus area were often tied to the coastal community development focus area. In speaking with some state program officials, however, we found that improving coastal water quality remains a priority for their programs even without coastal nonpoint program funding.

Similarly, representatives from the Coastal States Organization’s coastal water quality workgroup indicated that many state programs have made progress in developing and implementing coastal nonpoint pollution control programs, but that these results are not quantified by NOAA.

In addition, NOAA has not established numerical targets for the measures in its CZMP performance measurement system for the purpose of tracking progress or assessing performance of the CZMP. Our past work found that numerical targets are a key attribute of successful performance measures because they allow managers to compare planned performance with actual results. In 2008, we recommended that NOAA establish numerical targets for performance measures that would help track progress toward meeting program goals and help assess the overall CZMP effectiveness. NOAA’s 2011 performance measurement system workgroup also recommended that NOAA set targets to help it more effectively measure and communicate CZMP performance. NOAA agreed with these recommendations, but it has not established numerical targets for the measures in its CZMP performance measurement system to assess CZMP performance. NOAA officials explained that state programs vary widely, making it difficult to set targets at the national level. Officials also said that they first need to review the performance measures before they assess the feasibility of developing numerical targets. NOAA officials added that NOAA has set numerical targets for four CZMP performance measures, which are included in Commerce’s department-wide goals related to environmental stewardship. NOAA officials told us that they considered historical performance measure data and state programs’ planned strategies when establishing these targets, but they do not use them to assess CZMP performance. We continue to believe that, without setting numerical targets for the CZMP performance measurement system, NOAA will not have a benchmark to help it determine the extent to which the CZMP may be meeting expectations.

Finally, the CZMP performance measurement system includes performance measures that involve the collection of data by state programs that are already available to NOAA from other sources. Limited overlap, another key attribute of successful performance measures, notes that measures should produce new information beyond what is provided by other data sources and that redundant or unnecessary performance information costs resources and clouds the bottom line by making managers sort through excess information. We found that the CZMP performance measurement system includes at least two financial measures whereby states collect and submit financial expenditure data similar to data states already provide NOAA through their cooperative agreements. NOAA officials told us that, in developing the CZMP performance measurement system, they anticipated that including such measures would be useful for tracking the amount of CZMP funding used in different focus areas each year. However, NOAA used the financial information from its CZMP performance measurement system to prepare a one-time summary of performance measure data published in 2013. In contrast, it uses financial information drawn from cooperative agreements on an annual basis to analyze states’ planned uses of CZMP funding. NOAA officials acknowledged that they may need to review the utility of requiring state programs to collect financial expenditure data for the performance measurement system. By requiring states to collect and submit financial data similar to data that they already provide in their cooperative agreements and making limited use of these data, NOAA may be unnecessarily burdening state programs with data collection requirements.

Several state program officials we interviewed told us that collecting data for the numerous data elements under the 17 performance measures is a time- and resource-intensive activity, with a few stating that this is particularly true relative to the amount of CZMP funds they receive. Some indicated, for instance, that they spend 30 staff days or more per year collecting these data. State officials said that, in particular, data for the financial measures are among the most time-consuming to collect and report to NOAA. Other state officials told us that collecting data on the number of educational and training events and participants for each focus area is especially time-consuming, with one official noting that collecting data on number of participants is particularly burdensome when events are hosted by parties other than the program itself.

NOAA officials told us they recognized the need to continue to review and potentially streamline or revise the CZMP performance measurement system, and that they intend to do so once the merger of OCRM and the Coastal Services Center is complete, which they expect to occur by the end of 2014. In the interim, NOAA officials said they initiated at the beginning of fiscal year 2014 an effort to assess all performance measures collected by the various programs within the two offices, including the CZMP, to determine which measures may be most effective in tracking and communicating progress toward goals identified in the merged office’s strategic plan. NOAA officials said they are committed to developing a strong framework for evaluating the performance of all programs under its merged coastal management office. However, the agency has not documented the approach it plans to take for these efforts. Federal internal control standards state the need for federal agencies to establish plans that encompass actions the agency will take to help ensure goals and objectives can be met. Without a documented approach for how it plans to assess its CZMP performance measurement system—including the scope and criteria it will use, such as how it will ensure its measures align with key attributes of successful performance measures—NOAA cannot demonstrate that its intended effort will improve its CZMP performance measurement system.

NOAA Cannot Ensure That Its Revised Process for Surveying Stakeholders to Inform State Program Evaluations Will Produce Complete and Unbiased Information

In 2013, NOAA revised its process for conducting state program evaluations, which are required under the Coastal Zone Management Act to assess state programs’ adherence to the act’s requirements, but we identified a limitation in NOAA’s method for sampling stakeholders under the revised process. According to NOAA documents, the purpose of the revisions was to conduct evaluations more efficiently, at a reduced cost, while continuing to meet evaluation requirements outlined in the act. In revising its state program evaluations, NOAA made changes in the timing and methods for collecting information from participating states (see table 4). A NOAA official estimates that the agency’s revised evaluation process will save the agency approximately $236,000 annually. NOAA began evaluating state programs using its revised process at the beginning of fiscal year 2014 with evaluations of seven state programs. We did not evaluate NOAA’s implementation of its revised state program evaluations because NOAA had not completed its first cycle at the time of our review and, therefore, it was too early to assess the effectiveness of its revisions. However, we did assess NOAA’s revised evaluation design against our and others’ work on program evaluations to identify standards for strong evaluation design. We were unable to evaluate the qualitative components of its revised evaluation design—including the change in the scope of the evaluations from NOAA’s review of all aspects of each state program to a review of a few areas determined by NOAA—because the results of using these methods cannot be fully assessed until the evaluations have been conducted. But, we did evaluate the steps NOAA laid out in its guidance on its methods for collecting information and identified a limitation in its method for sampling stakeholders to survey.

Under its revised evaluation process, NOAA relies in part on information obtained through stakeholder surveys, but we found that through its method of sampling stakeholders to survey, the agency may be susceptible to collecting incomplete and biased information. According to NOAA guidance on its revised evaluations, stakeholder surveys are intended to provide information about stakeholders’ perspectives and opinions across a range of topics, from a state program’s top three strengths and weaknesses to opportunities for improving a program’s federal consistency and permitting processes. The guidance states that NOAA will use stakeholder survey responses to identify evaluation target areas, as well as obtain information about the extent to which a state program is performing effectively in areas outside of the target areas. NOAA officials indicated that they plan to analyze survey results by collating respondents’ answers to identify common themes. NOAA evaluators will identify a sample of stakeholders to survey from 12 categories of organizations that stakeholders represent, including federal agencies, state agencies, nonprofit organizations, academic institutions, and local businesses and industries. According to NOAA officials, they adopted the criterion of stakeholder categories to ensure that stakeholders whose views were not consistently represented in the former evaluations—such as those from local businesses and industries—are included in evaluations conducted under the revised process. NOAA evaluators will select stakeholders to survey from these 12 categories from a list of potential stakeholders to survey compiled by state program officials and NOAA specialists working with the state.

According to the Office of Management and Budget’s Standards and Guidelines for Statistical Surveys, a survey sampling method should yield the data required to meet the objectives of the survey. Our previous work has found that strong program evaluations rely on data that sufficiently reflect the activities and conditions a program is expected to address. Because NOAA’s stakeholder sampling method is guided by one criterion—categories of stakeholder organizations—NOAA may not collect information that reflects the various activities and aspects of the state programs. Specifically, under the act, NOAA is required to evaluate the extent to which state programs have addressed coastal management needs reflecting the six focus areas based on the goals identified in the act. In the absence of additional criteria for selecting stakeholders to survey, NOAA may select a sample of stakeholders whose work with a state program does not span all of the act’s goals, potentially leaving NOAA without information to inform its evaluation of a state’s performance on one or more goals. Such an information gap could be significant because stakeholder surveys are intended to be a main source of information on how well a program is performing in areas beyond those identified as target areas.

Furthermore, when using a nonprobabilistic sampling method, such as that being employed by NOAA for its stakeholder surveys, the Office of Management and Budget’s survey guidelines state that agencies should demonstrate that they used an impartial, objective method to include or exclude people or organizations from a sample. Our previous work on program evaluation also found that evaluation data should be sufficiently free of bias or other errors that could lead to inaccurate conclusions. Because state program officials responsible for identifying potential stakeholders to survey have a vested interest in their programs, NOAA’s process is susceptible to collecting biased information. NOAA specialists who work with state programs also contribute to the selection process. However, we found that some NOAA specialists are not regionally located or have worked with a state program for a short period of time and, therefore, their knowledge or experience to inform the selection process may be limited. NOAA’s evaluation guidance recognizes the need to assess its revised process in the future and states that the agency plans to evaluate the effectiveness and efficiency of its revised state program evaluation process after conducting 8 to 10 evaluations.

NOAA Makes Limited Use of Performance Data It Collects in Managing the CZMP

We found that in managing the CZMP, NOAA makes limited use of the performance information it collects. Our past work has found that performance information can be used across a range of management functions to improve programs and results, including to (1) identify problems or weaknesses in programs and take corrective actions, (2) set program priorities and develop strategies, (3) recognize and reward organizations who meet or exceed expectations, and (4) identify and share effective approaches to program implementation. For example, our previous work found that the Department of Labor effectively used performance measure data to identify technical assistance needs of state programs and to then provide assistance to try to improve performance. The department also used performance measure data as a basis for providing financial incentives to state programs that receive federal grants. We found that agencies realize the full benefit of collecting performance information only when they use such information to make decisions designed to improve results.

NOAA collects performance information through its CZMP performance measurement system, state program evaluations, and other sources, but we found that the agency generally does not use the information it collects to help manage the CZMP at a national level. Specifically, we found the following:

NOAA uses its CZMP performance measurement system data to report on national program accomplishments on a limited basis. In particular, in 2013, NOAA produced one report summarizing performance measurement system data from 2008 through 2011. However, NOAA has not published additional similar reports, and has not used performance measurement system data for other purposes. For example, the agency has not used the performance measurement system data to identify potential problems or weaknesses in the CZMP, set program priorities or strategies, or recognize and reward high-performing state programs—which may limit the usefulness of collecting such data.

NOAA does not use its state program evaluations to assess the performance or improve the implementation of the CZMP at the national level. NOAA uses its state program evaluations to identify state-specific accomplishments and encourage or require the state under evaluation to make improvements or take corrective actions. But, according to NOAA officials, the agency does not regularly analyze findings from individual state evaluations to identify and share effective approaches across states or to identify common performance weaknesses that may warrant national focus or assistance. Our analysis of recent NOAA evaluations of the seven state programs we reviewed found that NOAA recommended the states undertake similar actions. In five of the seven state program evaluations, for example, NOAA recommended that programs undertake strategic planning, and for four of the seven programs, NOAA recommended that programs improve their coordination with local governments or other partners who help carry out coastal management activities. Yet NOAA has not analyzed these evaluations to identify common findings. One NOAA specialist we spoke with suggested that NOAA could also use the results of its state program evaluations to recognize and reward high-performing state programs. For instance, the NOAA specialist suggested that NOAA could modify its eligibility requirements for its Projects of Special Merit funding such that only high-performing programs, with any necessary actions from past state program evaluations fully implemented, would be eligible to receive funding.

NOAA does not use performance-related information from other sources to support its management of the CZMP. NOAA uses state programs’ semiannual progress reports—which contain, among other things, “success stories,” or examples of a state program successfully addressing coastal management issues—to track states’ progress in implementing their cooperative agreements. However, NOAA does not use information from these reports to identify and promote effective approaches to coastal management by regularly sharing states’ success stories across states or with other stakeholders. The 2011 performance measurement system workgroup composed of NOAA and state program officials recommended that NOAA develop a website to share success stories on an annual basis. NOAA did not implement this recommendation because, according to NOAA officials, at that time it was incorporating success stories into a quarterly newsletter. According to a NOAA document, the agency produced the newsletter in response to requests from states for more information about how other state programs address coastal management issues. NOAA stopped issuing this newsletter in 2012, when its office merger began, and NOAA officials said they are now evaluating how the merged office might best share information about the CZMP across state programs and with other stakeholders.

NOAA’s strategic plan for its merged coastal management office recognizes the importance of using and reporting performance information. According to this plan, NOAA is committed to maintaining a culture of monitoring and evaluation to improve the implementation of its programs. We found, however, that the strategic plan does not include a documented strategy for using the performance data NOAA collects through its CZMP performance measurement system, state program evaluations, or other sources of information, such as states’ semiannual progress reports, to manage the CZMP. NOAA officials told us that because the office merger is under way, they have not formulated a strategy for how the merged office will use performance data to inform and manage the CZMP, but they recognized the need to do so once the merger is complete. Federal control standards state the need for federal agencies to document management approaches to ensure goals and objectives can be met. Without a documented strategy for using the full range of performance information it collects, NOAA may not be taking full advantage of the performance information that its specialists, evaluators, and state program officials spend time and resources collecting, and it cannot ensure that it is realizing the full benefit of collecting such information, such as identifying common problems in state programs and taking corrective actions, setting national program priorities and developing strategies, recognizing state programs that exceed expectations, or identifying and sharing effective approaches to program implementation.

Finally, NOAA has not taken steps to integrate data from its CZMP performance measurement system with information from its state program evaluations to develop a complete picture of the CZMP’s performance, as we recommended in our 2008 report. In 2008, we found that NOAA was not integrating quantitative national performance measure data with qualitative information from state program evaluations to develop a more comprehensive assessment of the CZMP’s performance. NOAA agreed with our recommendation to develop an approach for integrating the two types of information and, in response, tasked the 2011 performance measurement system workgroup with developing a method for better communicating performance measure data. The workgroup recommended a template for communicating program results that includes quantitative national performance measure data and qualitative success stories from states’ semiannual progress reports. However, NOAA has not drawn on this quantitative and qualitative information for purposes other than producing a report in 2013 summarizing performance measurement system data. Specifically, NOAA has not integrated quantitative and qualitative information to better understand program performance, improve its assessment of difficult-to-measure activities, or validate its assessments of program progress. We have previously found that agencies that used multiple sources of data to assess performance had information that covered more aspects of program performance than those that relied on a single source. We also found that agencies can improve their performance assessments by using program evaluation information to validate performance measurement system data. We continue to believe that developing an approach to combine performance information from its CZMP performance measurement system and state program evaluations could help NOAA obtain a more complete picture of CZMP performance.

Conclusions

The CZMP plays an integral role in helping states protect, restore, and manage the development of the nation’s coastal resources and habitats. In managing the CZMP, NOAA is challenged with the task of assessing the performance of the program, composed of partnerships with 34 individual states, each with unique coastal habitats, and differing laws, organizational structures, and funding priorities. NOAA is to be commended for its progress in improving its two primary performance assessment tools—its CZMP performance measurement system and state program evaluations—since we last reviewed the agency’s performance assessment processes in 2008. We are encouraged by NOAA’s recognition of the importance of using performance information to improve the implementation of the CZMP. However, NOAA does not use or have a documented strategy for how it will use the performance information it collects from its CZMP performance measurement system, state program evaluations, or other sources of performance-related information, as appropriate, to aid its management of the CZMP. Without a documented strategy for using the range of its performance information, NOAA cannot ensure that it is collecting the most meaningful information and realizing the full benefit of the significant amount of information it and the states collect, such as identifying common problems in state programs and taking corrective actions, setting national program priorities and developing strategies, recognizing state programs that exceed expectations, or identifying and sharing effective approaches to program implementation.

We also are encouraged by NOAA’s intentions to review and possibly revise the CZMP performance measurement system once its new coastal office is in place, but the agency has yet to document the approach it plans to take—including the scope and criteria it will use for this effort. In the absence of a documented approach indicating how it will review its performance measurement system, NOAA cannot ensure that its upcoming effort will take into consideration key attributes of successful performance measures, including balance and limited overlap, or result in a system that provides meaningful information that can be used by NOAA to determine how effectively the CZMP is performing relative to its goals. We are further encouraged by NOAA’s commitment to evaluate the effectiveness and efficiency of its revised state program evaluation process and to modify it, as needed, as it moves forward with its implementation. In the interim, however, NOAA’s method for selecting stakeholders to survey during state program evaluations—which relies on a single criterion and on state program officials who have a vested interest in the program—may result in the collection of incomplete or biased information that does not ensure perspectives are gathered from stakeholders representing a variety of program goals and are collected in an objective manner, potentially undermining the sufficiency and credibility of the data the produces. In the absence of additional criteria for selecting stakeholders to survey, NOAA may select a sample of stakeholders whose work with a state program does not span the act’s six focus areas or who present less-than-objective assessments of a state program.

Recommendations for Executive Action

To ensure that NOAA collects and uses meaningful performance information to help manage the CZMP, including continuing to improve its CZMP performance measurement system and its state program evaluations, we are recommending that the Secretary of Commerce direct the Administrator of NOAA to take the following three actions:

Develop a documented strategy to use the range of performance information the agency collects, as appropriate, to aid its management of the CZMP, such as to identify potential problems or weaknesses in the CZMP; set program priorities or strategies; or recognize and reward high-performing state programs.

As part of its intended review of the CZMP performance measurement system and in consideration of how it intends to use the performance information, document the approach it plans to take to analyze and revise, as appropriate, the performance measures, and in so doing ensure the analysis considers key attributes of successful performance measures, such as balance and limited overlap.

Revise the sampling methodology for selecting stakeholders to survey—included as part of its state program evaluation process—to ensure perspectives are gathered from stakeholders representing a variety of program goals and are collected in an objective manner.

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Commerce for review and comment. In written comments provided by NOAA through Commerce (reproduced in appendix IV), NOAA generally agreed with our findings and concurred with our recommendations. NOAA also provided technical comments that we incorporated, as appropriate. In its comment letter, NOAA stated that while it found GAO’s evaluation of the CZMP performance measurement system accurate, the agency did not agree with GAO’s assessment that eliminating a stand-alone category for coastal water quality could negatively affect the system’s ability to reflect the goals of the CZMA in a balanced way. NOAA stated that removal of the coastal water quality focus area did not impair its ability to track progress in meeting the water quality goal of the CZMA, explaining that it retained one measure composed of two data elements related to coastal water quality, but housed under a different focus area. We agree that the two-part measure NOAA maintained related to coastal water quality may provide important information on performance in this area. However, we continue to believe that the information it is collecting related to coastal water quality may not be balanced in comparison to the information it is collecting for the other five focus areas, which could in turn result in inconsistent performance information when looking across the six focus areas of the program. NOAA concurred with the three recommendations in the report and described actions it plans to address them. With regard to the first recommendation, NOAA stated that it plans to develop a strategy for using performance information it collects, including information from its performance measurement system, evaluations of state programs, performance reports, and other sources, and noted that it will build upon existing efforts to share lessons-learned regarding successful approaches or shared challenges across the national program. In addressing our second recommendation, on documenting its approach for analyzing and revising, as appropriate, the performance measures, NOAA stated that it plans to conduct a review of CZMP performance measures in fiscal year 2015 as part of its ongoing analysis of performance measures for programs under its new coastal office. In response to our third recommendation, NOAA stated that it will revise its sampling methodology to ensure stakeholders representing a variety of program goals are selected.

We are sending copies of this report to the Secretary of Commerce, the appropriate congressional committees, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or fennella@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

Focusing on National Coastal Zone Management Program (CZMP) activities since our 2008 report, our objectives were to examine (1) how participating states allocated CZMP funds awarded in fiscal years 2008 through 2013 and (2) how the National Oceanic and Atmospheric Administration’s (NOAA) primary performance assessment tools have changed and the extent to which NOAA uses performance information to help manage the CZMP.

To examine how participating states allocated CZMP funds awarded in fiscal years 2008 through 2013, we reviewed the Coastal Zone Management Act and related regulations and guidance, including NOAA funding guidance and allocation memos. We analyzed NOAA data on federal funds awarded by state and by funding type from fiscal years 2008 to 2013, and we compared this data against annual NOAA funding guidance and allocation memorandums to states. Based on our analysis, and interviews with NOAA officials, we found the data to be sufficiently reliable. We reviewed NOAA’s analysis of states’ allocations of CZMP funding for fiscal years 2008 through 2013, which was based on NOAA’s review of its cooperative agreements for federal funding with states. NOAA’s analysis involved the categorization of states’ funding allocations for projects into six focus areas based on the goals of the act and an additional state program management category as defined by NOAA to cover administrative costs, such as general program operations, supplies, and rent. NOAA officials noted that total funding allocation amounts are approximate and that many CZMP funded activities could address more than one focus area. For example, Maine state program officials told us their activities to conserve and enhance properties that provide commercial fishing access address both coastal community development and public access focus areas. To address this challenge, NOAA developed written guidance for NOAA specialists who conduct the analysis that specifies the types of activities to include in each focus area and the state program management category, as well as direction on how to categorize funds in cases where a project or activity may fall in more than one category. For instance, NOAA defined funds in the government coordination focus area to include, among others, activities that involved coordination with other government agencies and stakeholders, technical assistance to local governments, or public outreach and education activities only if they did not correspond to other focus areas. To determine the reliability of NOAA’s analysis, we interviewed knowledgeable NOAA officials, reviewed NOAA’s process for categorizing proposed activities and projects, including its written guidance on categorizing CZMP-funded activities and its steps to compare funding amounts to ensure that the double-counting of funds did not take place.

We did not independently verify the results of NOAA’s analysis, but we verified major categories used in NOAA’s analysis for consistency across years, checked the total allocated funds in NOAA’s analysis against total federal funding award data, and reviewed NOAA’s categorization of a small sample of projects. We concluded the data to be sufficiently reliable for our purposes of reporting states’ allocated uses of CZMP funds.

We also reviewed data from NOAA’s CZMP performance measurement system from 2008 through 2013 (the most recent years for which data was available) to further illustrate how CZMP funds were used. To assess the reliability of NOAA’s CZMP performance measurement system data, we interviewed NOAA officials about reliability of the data and reviewed corresponding documentation including performance measures guidance to states and internal guidance to NOAA specialists about their required reviews of data submitted. We did not independently verify performance measure data submitted by state programs, but based on our review of steps taken by NOAA to review state-submitted data, we found the data sufficiently reliable for the purposes of our report.

To examine how NOAA’s primary performance assessment tools have changed since 2008, and the extent to which NOAA uses performance information to help manage the CZMP, we analyzed applicable laws and guidance including the act, and NOAA’s guidance on its CZMP performance measurement system and state program evaluations. We reviewed documentation on changes NOAA has made to these two performance tools, including steps taken to address our 2008 report recommendations, and we interviewed NOAA officials about the changes they made and their use of performance information. We reviewed GAO’s work on performance measurement to identify key attributes associated with successful performance measures and assessed NOAA’s CZMP performance measurement system against these attributes by reviewing the agency’s performance measures and guidance on the system and interviewing NOAA and state program officials. We also analyzed NOAA’s CZMP performance measurement system data from 2011, 2012, and 2013. We reviewed our and others’ work on program evaluations to identify standards for strong evaluation design and assessed NOAA’s process for evaluating state coastal programs against these standards by examining NOAA’s evaluation guidance and interviewing NOAA officials. We examined information NOAA maintains on CZMP performance including fact sheets, states’ cooperative agreements, semiannual progress reports, performance measurement system data submitted by states, and state program evaluation reports.

In conducting our work on both objectives, we interviewed representatives of the Coastal States Organization, a nonprofit organization that represents coastal states on legislative and policy issues, as well as state program officials from the seven states that received the most fiscal year 2012 CZMP funding in each of NOAA’s seven regions (California, Florida, Hawaii, Maine, Michigan, Texas, and Virginia) about how states used CZMP funds and for their perspectives on NOAA’s management and assessment of the overall national program. We also reviewed the seven states’ cooperative agreements and semiannual progress reports for fiscal years 2011 and 2012 (the most recent years for which reports were available) to learn about projects undertaken by these seven states. We selected one CZMP-funded project in each of the seven states to further determine and illustrate how states used funds on a project-level basis and to learn about how the results of a select project are captured by NOAA’s performance assessment tools. In selecting projects to review, we considered the amount of CZMP funds allocated to specific projects, funding type, project type (e.g., projects that provide financial and technical assistance to local governments, planning projects, construction-related projects, permitting activities), and focus area (e.g., coastal habitat, government coordination). Our review of the states’ information cannot be generalized across all states or projects. We also interviewed coastal program officials from American Samoa and the Northern Mariana Islands to obtain perspectives from territories on NOAA’s performance assessment tools and territories’ use of this performance information.

We conducted two site visits to observe and learn more about CZMP projects—one to a coastal habitat restoration project in Texas and one to an ocean planning project in Virginia. We selected these projects for site visits considering project type, focus area addressed, and geographic location. During our site visits, we met with state program officials and also interviewed stakeholders involved in the selected projects, as well as stakeholders involved in other CZMP-funded projects. In Texas, we met with the nonprofit organization managing the coastal habitat restoration project and toured the restoration site; in Virginia, we visited a public access enhancement project that received CZMP funding.

We conducted this performance audit from June 2013 to July 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: National Coastal Zone Management Program (CZMP) Performance Measurement System Summary Data

The National Oceanic and Atmospheric Administration’s (NOAA) CZMP performance measurement system is organized by broad focus areas that are related to five of the six primary focus areas based on the goals of the CZMP as outlined in the Coastal Zone Management Act. The system consists of 17 performance measures—15 of the 17 measures are organized under the five broad focus areas (NOAA removed the sixth focus area, coastal water quality, from its performance measurement system in 2011 in response to a performance measurement system workgroup’s recommendation to streamline the system), and the remaining 2 measures are to track state financial expenditures. Each of the 17 measures is composed of several individual data elements. For example, the performance measure on federal consistency is composed of two data elements that track the number of projects reviewed and the number of projects modified under states’ federal consistency review processes. In addition, some data elements are further broken down into specific categories, such as types of federal consistency projects modified. See table 5 for a list of the performance measures and supporting data elements and categories, as reported by participating state programs for 2011 through 2013.

Appendix III: Key Attributes of Successful Performance Measures Previously Identified by GAO

Attributes Linkage

Definition Measure is aligned with division and agency-wide goals and mission and clearly communicated throughout the organization.

Potentially adverse consequences of not meeting attribute Behaviors and incentives created by measures may not support achieving division or agency-wide goals or mission.

Measure is clearly stated and the name and definition are consistent with the methodology used to calculate it.

Data may confuse or mislead users.

Measure has a numerical target.

Managers may not be able to determine whether performance is meeting expectations.

Measure is reasonably free from significant bias or manipulation.

Performance assessments may be systematically over- or understated.

Measure produces the same result under similar conditions.

Reported performance data may be inconsistent and add uncertainty.

Measures cover the activities that an entity is expected to perform to support the intent of the program.

Information available to managers and stakeholders in core program areas may be insufficient.

Measure provides new information beyond that provided by other data sources.

Manager may have to sort through redundant, costly information that does not add value.

Taken together, measures ensure that an organization’s various priorities are covered.

Measures may over emphasize some goals and skew incentives.

Each measure should cover a priority such as quality, timeliness, and cost of service.

A program’s overall success is at risk if all priorities are not addressed.

Appendix IV: Comments from the Department of Commerce

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, Alyssa M. Hundrup (Assistant Director), Elizabeth Beardsley, Mark A. Braza, Elizabeth Curda, John Delicath, Tom James, Katherine Killebrew, Patricia Moye, Dan Royer, Kiki Theodoropoulos, and Swati Sheladia Thomas made key contributions to this report.