Coastal Zone Management: Opportunities Exist for NOAA to Enhance Its Use of Performance Information
Published: 2014-07-16T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The U.S. coast is home to more than half the U.S. population and integral to the nation's economy. Under the Coastal Zone Management Act, NOAA administers the CZMP, a federal-state partnership that encourages states to balance development with protection of coastal zones in exchange for federal financial assistance and other incentives. In 2008, GAO reviewed the CZMP and recommended improvements for CZMP performance assessment tools.
A fiscal year 2013 appropriations committee report mandated GAO to review NOAA's implementation of the act. This report examines (1) how states allocated CZMP funds awarded in fiscal years 2008 through 2013 and (2) how NOAA's primary performance assessment tools have changed since GAO's 2008 report and the extent to which NOAA uses performance information in managing the CZMP. GAO reviewed laws, guidance, and performance-related reports; analyzed CZMP funding data for fiscal years 2008-2013; and interviewed NOAA officials and a nongeneralizeable sample of officials from seven states selected for receiving the most fiscal year 2012 funding in each of NOAA's regions.
section_title: What GAO Found
paragraphs: During fiscal years 2008 through 2013, the 34 states participating in the National Oceanic and Atmospheric Administration's (NOAA) National Coastal Zone Management Program (CZMP) allocated nearly $400 million in CZMP funds for a variety of activities. States allocated this funding for activities spanning six broad focus areas based on goals outlined in the Coastal Zone Management Act. For example, states allocated about a quarter of their CZMP funding to the coastal habitat focus area, according to NOAA's analysis. Coastal habitat activities encompassed a variety of actions to protect, restore, or enhance coastal habitat areas, such as habitat mapping or restoration planning efforts of marsh habitats for fish and wildlife and enhanced recreational opportunities.
NOAA's two primary performance assessment tools—its CZMP performance measurement system and state program evaluations—have limitations, even with changes NOAA made since 2008, and NOAA makes limited use of the performance information it collects. Regarding the performance measurement system, NOAA has made changes such as taking steps intended to improve the reliability of data it collects. However, its current measurement system does not align with some key attributes of successful performance measures, including the following:
Balance: a balanced set of measures ensures that a program's various goals are covered. NOAA removed the coastal water quality focus area, one of six focus areas based on goals in the act, to streamline the performance measurement system. As a result, the system may not provide a complete picture of states' overall performance across all focus areas based on goals in the act.
Limited overlap: measures should produce new information beyond what is provided by other data sources . NOAA's system includes measures that overlap with financial data provided in cooperative agreements. By requiring states to submit financial data available through other sources, NOAA may be unnecessarily burdening states with data collection requirements.
NOAA plans to review and potentially revise its measurement system, but it has not documented the approach it plans to take, including how the measures will align with key attributes of successful performance measures. Regarding state program evaluations, in 2013, NOAA revised its process to conduct evaluations more efficiently, at a reduced cost. However, GAO identified a limitation in NOAA's method for sampling stakeholders to survey under its revised process that may result in the selection of stakeholders that do not span all six focus areas based on goals of the act. Finally, NOAA makes limited use of the performance information it collects from these tools. For example, since it began collecting performance measurement data in 2008, NOAA used the data once to report on accomplishments. NOAA recognizes the importance of using performance information to improve program implementation, but it has not documented a strategy for how it will use its performance information to manage the program. As a result, NOAA may not be realizing the full benefit of collecting performance information.
section_title: What GAO Recommends
paragraphs: GAO recommends that NOAA document an approach to analyze and revise, as appropriate, its performance measures against key attributes, revise its process for selecting stakeholders to survey in its state program evaluations, and document a strategy for using the performance information it collects. NOAA concurred with the recommendations.

section_title: Letter
paragraphs: The U.S. coastal zone—which includes areas along the Atlantic, Pacific,  and Arctic oceans, the Gulf of Mexico, and the Great Lakes—is home to  more than half of the population of the United States and plays an integral  role in our nation’s economy, contributing to over half of the U.S. gross  domestic product and supporting more than 66 million jobs. The coastal  zone provides harbors for ports, shipping, and navigation; opportunities  for energy production from traditional sources such as offshore oil, and  alternative sources, including wind, tidal, and wave; beaches and  shorelines for recreation and tourism; and wetlands and estuaries that are  critical for sustained fisheries. The coastal zone also provides important  environmental benefits including filtering pollution from runoff; buffering  shoreline communities against storms; and providing habitat including  spawning grounds, shelter, and food for marine life, and other threatened,  endangered, and commercially important species. Increased population  density and economic activity in the coastal zone, however, puts pressure  on coastal habitats.
In 1972, Congress enacted the Coastal Zone Management Act to balance  the often competing demands for economic growth and development with  the need to protect coastal resources. In doing so, Congress recognized  that states have primary responsibility for planning and managing their  coastal zones. On this basis and to help achieve the goals of the act, the  Department of Commerce’s National Oceanic and Atmospheric  Administration (NOAA), which has responsibility for implementing the act,  administers a voluntary, cooperative program established under the act  known as the National Coastal Zone Management Program (CZMP).   The CZMP involves federal and state partnerships that encourage coastal  states, Great Lakes states, and U.S. territories and commonwealths— hereafter referred to as states—to balance and manage economic  development and coastal protection. To participate, states must develop  and implement a comprehensive coastal management program that  addresses specific goals established in the act and meets other federal  requirements, but states have the flexibility to design programs that best  meet their own coastal needs. NOAA provides funding and technical  assistance to support administrative and project-specific costs for state  coastal management programs (state programs) it approves to participate  in the CZMP. In fiscal year 2013, 34 of 35 eligible states had state  programs and received CZMP funding.
NOAA evaluates the CZMP’s performance using two primary tools—its  CZMP performance measurement system and state program evaluations.  NOAA, working with state programs, developed the performance  measurement system in the mid-2000s, in response to congressional  direction to assess the national impact of the CZMP and report on  progress in meeting the act’s goals. In addition, NOAA is required under  the act to conduct evaluations of each state program to assess its  adherence to the act’s requirements. NOAA also collects performance- related information from other sources, such as semiannual progress  reports submitted by participating states describing progress in  implementing their cooperative agreements. Cooperative agreements are  the mechanisms by which NOAA provides congressionally appropriated  financial assistance annually to states for the program activities they have  agreed to undertake under the CZMP. We reviewed the CZMP in 2008  and made several recommendations to address weaknesses we found in  NOAA’s use of its tools to assess the performance of the CZMP, among  others. NOAA agreed with all but one of our recommendations, and it  has taken steps to address the majority of our recommendations.
The House Appropriations Committee report for the Department of  Commerce Fiscal Year 2013 appropriation mandated us to examine  NOAA’s implementation of the Coastal Zone Management Act. Focusing  on CZMP activities since our 2008 report, this report examines (1) how  participating states allocated CZMP funds awarded in fiscal years 2008  through 2013 and (2) how NOAA’s primary performance assessment  tools have changed and the extent to which NOAA uses performance  information to help manage the CZMP.
To conduct our work, we reviewed applicable laws, regulations, and  NOAA guidance, including guidance on its CZMP performance  measurement system and state program evaluations. We analyzed  NOAA data on financial assistance awards for fiscal years 2008 through  2013, NOAA analyses of states’ allocations of CZMP funds, and its  performance measurement system data. To assess the reliability of these  data, we interviewed NOAA officials and reviewed related documentation,  and we determined the data we used to be sufficiently reliable for our  purposes. We interviewed NOAA officials responsible for administering  the CZMP and conducting state program evaluations about the  approaches they use to assess program performance, how they use  performance information, and any changes they have made since 2008,  including steps taken to address recommendations in our 2008 report.  We also reviewed our and others’ work on performance assessment,  including our work on key attributes of successful performance  measurement systems and elements of strong evaluation designs. We  examined NOAA’s publicly available reports and documents on CZMP  performance, including program fact sheets, program reports, and state  program evaluation results. We interviewed state program officials from  the seven states that received the most fiscal year 2012 CZMP funding in  each of NOAA’s seven regions. These include California, Florida, Hawaii,  Maine, Michigan, Texas, and Virginia. We obtained information about  states’ use of CZMP funds, their perspectives on NOAA’s performance  assessment tools, and how the results of a select project in each state we  reviewed were reported to NOAA. We selected projects considering such  factors as amount of CZMP funding, types of financial assistance and  projects, and goals established in the act. We also conducted two site  visits to observe and learn more about CZMP projects—one to a coastal  habitat restoration project in Texas, and one to an ocean planning project  in Virginia. We selected these projects for site visits considering project  type, goals of the act the project addressed, and geographic location.  Appendix I presents a more detailed description of our objectives, scope,  and methodology.
We conducted this performance audit from June 2013 to July 2014 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Congress enacted the Coastal Zone Management Act in 1972 to balance  the often competing demands for economic growth and development with  the need to protect coastal resources. To accomplish the goals of the act,  Congress established a framework for a voluntary federal and state  coastal management partnership, the CZMP. The CZMP represents a  unique federal-state partnership for protecting, restoring, and responsibly  developing the nation’s coastal communities and resources, according to  program documents.
The act identifies specific goals for state programs that fall into six broad  focus areas ranging from protecting and restoring coastal habitat to  assisting with coastal community development efforts and improving  government coordination and decision making (see table 1).
States must submit comprehensive descriptions of their coastal  management programs—which must be approved by the states’  governors—to NOAA for review and approval. As specified in the act,  states must meet the following requirements to receive NOAA’s approval  for their state programs, among others: designate coastal zone boundaries that will be subject to state define what constitutes permissible land and water uses in coastal propose an organizational structure for implementing the state  program, including the responsibilities of and relationships among  local, state, regional, and interstate agencies; and demonstrate sufficient legal authorities to carry out the objectives  and policies of the state program, including the means by which a  state will regulate land and water uses, control development, and  resolve conflicts among competing activities in coastal zones to  ensure their wise use.
The act provides states the flexibility to design programs that best  address states’ unique coastal challenges, laws, and regulations, and  participating states have taken various approaches to developing and  carrying out their programs. For instance, there are generally two  organizational structures used by states to implement their programs: (1)  networked programs, which rely on multiple state and local agencies to  implement their programs, and (2) non-networked, or comprehensive  state programs that administer all aspects of the program through a single  centralized agency. The coastal management activities carried out also  vary across states with some states focusing on permitting, mitigation,  and enforcement activities, while other states focus on providing technical  and financial assistance to local governments and nonprofits for local  coastal protection and management projects. If states make changes to  their programs, such as changes in their coastal zone boundaries or  organizational structures, the states must submit those changes to NOAA  for review and approval.
The act includes two primary incentives to encourage states to develop  coastal management programs and participate in the CZMP. First,  participating states are eligible to receive federal funding from NOAA to  support the implementation and management of their programs, which  the agency receives annually through congressional appropriations. In  fiscal year 2013, NOAA awarded participating states a total of  approximately $61.3 million, a 9 percent decline from fiscal year 2008  awards, when it awarded just over $67.5 million across participating  states.  NOAA awards CZMP funding to individual states across three  fund types—administrative, enhancement, and coastal nonpoint  program—according to requirements in the act (see table 2). The majority  of funding NOAA awards through the CZMP is administrative funding.  Administrative funding, which requires state matching funds, supports  general implementation of the state’s coastal management program.  Under the act, NOAA may also award a maximum of $10 million annually  in enhancement program funding to participating states. Enhancement  funding is to be used by states to develop program changes, or  enhancements, to their NOAA-approved programs in one or more of nine  enhancement objectives specified in the act, as listed in table 2. In  addition, Congress has generally provided direction on the total amount of  funds to be awarded through the coastal nonpoint program to assist with  states’ coastal nonpoint pollution control programs, which are programs to  ensure states have necessary tools and enforceable authorities to  prevent and control polluted runoff in coastal areas. According to NOAA  officials, funding has not been provided for this program since fiscal year  2009, when nearly $3.4 million was awarded to states. States may also  use other sources of funding for their coastal nonpoint pollution control  programs, including administrative and enhancement funding.
Second, federal agency activities in or affecting the uses or resources of  a participating state’s defined coastal zone are required to be consistent  to the maximum extent practicable with enforceable policies of the state’s  program. Under this provision, known as federal consistency, states with  approved programs must have the opportunity to review proposed federal  actions for consistency with enforceable policies of their state programs.  Types of federal actions that may be reviewed by states include federal  agency activities, such as improvements made to a military base; licenses  or permits to nonfederal applicants; financial assistance to state and local  governments; and outer continental shelf activities, such as oil and gas  development. If a state finds that a federal activity is not consistent with  the state’s enforceable policies, the state can object to the activity and  work with the federal agency to resolve any differences between the  proposed activity and state policies. All participating state programs have  developed federal consistency review processes.
Thirty-four out of 35 eligible states have federally approved coastal  management programs (see fig. 1). Most state programs have been in  existence for more than 30 years, with the earliest program approved in  1976, and 29 states having received federal approval for their programs  by 1986. The most recent state to begin participating in the program is  Illinois, which received federal approval in January 2012.
NOAA’s Office of Ocean and Coastal Resource Management (OCRM) is  responsible for general administration and oversight of the CZMP. NOAA  plans to merge the OCRM with its Coastal Services Center—an office  that provides coastal-related mapping tools and data; training on various  coastal management issues such as climate adaptation and coastal  restoration design and evaluation; and technical and other assistance to  local, state, and regional coastal organizations—into a single office by the  end of 2014. Under the current and planned office structure, NOAA  officials are responsible for approving state programs and any program  changes, administering federal funding to the states, providing technical  assistance to states such as on the development of 5-year assessment  and strategy reports that identify states’ priority needs and projects to  address one or more of nine enhancement objectives required for  enhancement funding, among other topics, and managing the CZMP  performance measurement system. NOAA assigns coastal management  specialists to work with individual state programs. As part of its  administration of the program, NOAA evaluates program performance  using its CZMP performance measurement system. NOAA began  developing a framework for this performance measurement system in  2001, started piloting it in 2004, and fully implemented the system by  2008. The system consists of 15 performance measures that generally  correspond with the goals of the act, and two additional measures to track  state financial expenditures. The 17 total performance measures  incorporate individual data elements, plus additional subcategories of  information that state programs collect and report into the system  annually (see app. II).
In addition, NOAA evaluators, who are in a different NOAA division than  specialists, are responsible for conducting individual state program  evaluations, which are required under the act. State program evaluations  are designed to examine the extent to which states have: (1)  implemented their approved programs,(2) addressed coastal  management needs identified in the act, and (3) adhered to the terms of  CZMP funds awarded through cooperative agreements. NOAA’s state  program evaluation reports identify state accomplishments and make  recommendations for improving states’ programs. NOAA’s  recommendations are classified as either necessary actions—actions a  state must take by a specific date such as the next regularly scheduled  evaluation—or program suggestions—actions it believes a state should  take to improve its program. NOAA may withdraw approval for a state’s  program and financial assistance in cases where states do not address  necessary actions. NOAA has not withdrawn approval for a state program  as of the end of fiscal year 2013 and, according to NOAA officials, few  necessary actions have been identified in past state evaluations.
In 2008, we examined NOAA’s process for awarding financial assistance  to states and how the agency evaluated the effectiveness of the CZMP.  Of the seven recommendations we made in 2008, NOAA disagreed with  one recommendation that the agency develop performance measures to  evaluate the effectiveness of state programs in improving processes;  NOAA agreed with the other six recommendations and has taken some  actions to address them as described in table 3.
section_title: States Allocated Nearly $400 Million in CZMP Funds during Fiscal Years 2008 through 2013 for a Wide Range of Coastal Management Activities
paragraphs: During fiscal years 2008 through 2013, the 34 participating states  allocated a total of nearly $400 million in CZMP funds for a variety of  activities, generally related to the broad goals for state programs outlined  in the Coastal Zone Management Act. Each year, NOAA analyzes its  cooperative agreements with states for CZMP funding, and categorizes  the states’ CZMP funding allocations as they correspond with the six  focus areas based on the broad goals in the act, along with a seventh  category to capture state program administrative costs, such as general  program operations, supplies, and rent. According to NOAA’s analysis,  during fiscal years 2008 through 2013, states’ allocations of CZMP funds  varied across the seven categories, with about half concentrated in  support of activities related to two focus areas, government coordination  and coastal habitat (see fig. 2).
NOAA officials told us that, while states have the flexibility to design and  implement programs that best meet their unique needs, the agency does  influence how states allocate CZMP funds through (1) NOAA’s review  and approval of states’ 5-year assessment and strategy reports required  for enhancement funding in which participating states prioritize projects  that support program improvements and (2) NOAA’s periodic state  program evaluations in which NOAA outlines necessary actions or makes  program suggestions that can influence state program activities. NOAA  officials said that they also informally shape or influence state program  activities through ongoing discussions with state program officials about  funding proposals or specific projects, such as how projects might be  adjusted to address NOAA priorities.
Examples of activities for which participating states allocated CZMP funds  during fiscal years 2008 through 2013 in each of the six focus areas  include the following:
Government coordination. States allocated CZMP funds for  activities including state and regional planning efforts that involve  coordination among multiple levels of government and stakeholders to  address complex and controversial coastal issues, such as  comprehensive planning of ocean and nearshore areas, energy  facility siting or special area management planning; federal  consistency activities; technical assistance to local governments; and  public outreach and education on coastal issues including website  development and publications about a state program’s activities.  According to NOAA’s analysis of cooperative agreements with states  for CZMP funding, states allocated the largest amount of CZMP  funding during the 6-year period—about 27 percent of total funding— to government coordination activities. We found that a number of state  programs use CZMP funds to support participation in regional  organizations involving ocean planning activities that entail  coordination across federal, state, and local governments. For  example, state program officials in some Northeast and Mid-Atlantic  states participate in regional organizations, such as the Northeast  Regional Ocean Council and Mid-Atlantic Regional Council on the  Ocean, that have ocean resource data collection and planning efforts  under way. We also found that most states we reviewed provide some  type of technical or financial assistance to local governments to  support local level coastal management activities and projects.
Protecting Coastal Habitat in Texas  The Texas state program used coastal zone  funds to support a multiyear marsh restoration  project on the Texas Gulf Coast near Corpus  Christi. Over the past 60 years, about 340  acres of coastal marsh habitat were lost due  to the construction of an adjacent highway and  subsequent erosion. A local nonprofit  organization began restoring the marsh in  2005. The project involved scooping sand,  clay, and shells from the bay bottom and piling  the material into terraces and mounds;  planting native grasses on the terraces to  stabilize the structures and provide habitat;  and constructing an outer rock berm to protect  the new marsh area from strong waves in the  bay, as shown below.   Project officials told us Texas’s state program  provided about $1 million in coastal zone  funding, about 20 percent of the project’s total  cost, to the nonprofit organization responsible  for the project. Other funding to carry out the  project was provided by the EPA, U.S. Fish  and Wildlife Service, state government  sources, and grants from private foundations.   According  to project officials, the project was  completed in spring 2014 and has resulted in  160 acres of restored marsh that provide  habitat for fish, crabs, shrimp, nesting birds,  sea grass, and other plants and animals. The  project also resulted in the creation of new  opportunities for public recreation, such as  fishing and kayaking, and the marsh protects  the adjacent highway from coastal hazards,  such as storms, according to project officials.
Coastal habitat. States allocated CZMP funds for coastal habitat  protection and restoration activities including the acquisition or  placement of easements on coastal lands; restoration of coastal  habitats; data collection and mapping of coastal habitats;  development of plans for habitat acquisition, restoration, and other  habitat management needs; implementation of permitting and  enforcement programs that protect coastal habitat through planning  and regulation of development; or support of  land management  programs such as those for coastal preserves and parks. States also  allocated CZMP funds for public outreach and education activities that  focused on coastal habitat protection and restoration. According to  NOAA’s analysis, approximately 24 percent of CZMP funds awarded  during fiscal years 2008 through 2013 were allocated to coastal   habitat protection and restoration activities. According to NOAA’s  CZMP performance measurement system data from 2008 through  2013, states reported that they used CZMP funds to protect nearly  23,300 acres of coastal habitat through acquisition or easement,  restore nearly 37,400 acres of coastal habitat, and through regulatory  programs protect more than 123,000 net acres of coastal habitat.
Coastal hazards. States allocated CZMP funds for activities that help  coastal communities minimize risks from coastal hazards, such as  storms, tsunamis, and sea-level rise, and improve hazard awareness  and understanding. Such activities include assessment and planning  efforts, such as developing mitigation plans, risk and vulnerability  assessments, and data collection and mapping to identify and  manage development in areas vulnerable to coastal hazards;  implementation of hazard mitigation projects; implementation and  enforcement of hazard policies, regulations, and requirements; and  education and training on coastal hazard topics. According to NOAA’s  analysis of cooperative agreements with states for CZMP funding,  about 13 percent of CZMP funds awarded in fiscal years 2008 through  2013 were allocated for coastal hazards projects. The coastal hazards  focus area was the one focus area where the share of CZMP funds  allocated steadily increased over the 6-year period, from roughly 7  percent in fiscal year 2008 to about 16 percent in fiscal year 2013.  Most state program officials we spoke with identified their work to help  communities reduce future damage from hazardous events and  impacts from sea-level rise related to climate change as among their  more significant projects. NOAA also identified coastal hazards work  as a priority area, and in 2011, through the agency’s funding  guidance, began encouraging states to use CZMP funding for projects  that improve the resiliency of coastal communities to adapt to the  impacts of coastal hazards and climate change. In addition, many of  the projects that were awarded funding under the competitive Projects  of Special Merit Program in fiscal years 2012 and 2013 were identified  by states as addressing, at least in part, coastal hazards, according to  NOAA officials. For example, South Carolina’s project to study tidal  inlet dynamics and erosion and Maine’s adaptation planning project  for its coastal parks both addressed coastal hazard issues. NOAA’s  CZMP performance measurement system data for 2008 through 2013  show that states reported working with more than 410 communities to  reduce risks from coastal hazards and nearly 230 communities to  improve public awareness of coastal hazards issues.
A Coastal Water Quality Monitoring and  Modeling Project in Florida   Estuaries—such as Sarasota Bay, that spans  about 56 miles along the southwest Florida  coast—are important productive ecosystems  that provide habitat for a diversity of species.  Nonpoint source pollution carried through  runoff influences the health of the Sarasota  Bay, which has limited tidal flushing, no major  tributary, and receives most of its freshwater  from rainfall and associated runoff.  Florida’s coastal management program  provided nearly $150,000 in coastal zone  funds to support a multiyear water quality  monitoring and modeling study in Sarasota  Bay led by the Florida Fish and Wildlife  Research Institute. The study was designed to  help determine major factors affecting the  ecological health of the bay. Specifically,  coastal zone funding was used for statistical  modeling to differentiate between the effects  of polluted runoff into the bay during storm  events from the effects of natural algal, or  other natural sources of nutrients, in the bay.  Florida state program officials told us that  understanding ecological responses in  estuaries can facilitate planning to minimize  potential impacts and help maintain overall  ecosystem health. Continued water quality  monitoring and modeling is being completed in  the bay with other funding sources, according  to Florida officials.
Coastal water quality. States allocated CZMP funds for water quality  permitting and enforcement activities such as permitting of storm  water discharges; activities and projects related to water quality  management including vegetative plantings or other nonstructural  shoreline erosion control projects; water quality monitoring; activities  and projects for local governments to improve water quality  management; technical assistance, data collection, mapping,  planning, and policy development to address water quality issues;  marine debris and other coastal cleanup or pollution prevention  programs; and projects and activities that provide technical assistance  to marinas to reduce nonpoint source pollution; and public outreach  and education on water quality issues. Activities include those that  support states in implementing their coastal nonpoint source pollution  control programs. According to NOAA’s CZMP performance  measurement system data, from 2008 through 2013, states reported  that they worked with more than 680 communities to develop nonpoint  source pollution management policies and plans, or complete related  projects, and removed 27 million pounds of marine debris through  coastal cleanup activities.
Coastal community development. States allocated CZMP funds for  activities including planning and construction to support the  redevelopment of urban waterfronts, ports, and harbors; technical  assistance to local governments related to waterfront redevelopment;  community planning, land-use planning, green infrastructure planning,  and other sustainable development efforts; and public outreach and  education activities specific to coastal community development  issues. According to CZMP performance measurement system data  from 2008 through 2013, states reported that they worked with more  than 580 coastal communities to promote development and growth in  ways that protect coastal resources and with more than 250  communities to redevelop ports and waterfronts.
Public access. States allocated CZMP funds for activities including  creating new public access sites through easements or right of ways;  enhancing existing public access through trails, handicap features, or  educational signage; developing plans, collecting data, and providing  technical assistance to local governments on public access planning;  and conducting public outreach and education activities on public  access issues. According to NOAA’s analysis, states allocated the  least amount of CZMP funding (about 6 percent of total CZMP  funding) for activities that improve public access to the coast. Unlike  other focus areas, a number of states did not allocate funds for public  access. According to NOAA officials, some states may not need to   use CZMP funding to support public access projects, for example,  because they already have sufficient public access to coastal areas.  In total, according to CZMP performance measurement system data  from 2008 through 2013, states reported that with CZMP funds and  through regulatory programs they helped create nearly 700 new public  coastal access sites and helped enhance nearly 1,500 existing sites.
State program officials told us that CZMP funding is important because it  can help leverage other financial resources and provides sustained,  multiyear funding for projects. We found that CZMP-funded projects and  activities often involved partnerships with various entities and used  multiple sources of funding. According to state program officials, CZMP  funds were often the catalyst for obtaining additional financial assistance  or other resources. For example, we visited a $5.2 million, multiyear  marsh restoration project along the Texas Gulf coast that received nearly  20 percent of overall project funding through the CZMP and additional  financial support from eight other federal, state, and private sources.  Representatives from the nonprofit organization responsible for managing  the project told us that CZMP funds received during the initial stages  helped attract other funding partners needed for such a large-scale  restoration project. Similarly, Virginia’s program used $6,000 of its CZMP  funding to leverage staff from six partner organizations to plan and  conduct a Marine Debris Summit that laid the groundwork for developing  a marine debris plan and establish priorities for future work, which state  program officials expect will serve as a model for other Mid-Atlantic  states. Most of the state programs we reviewed also provide competitive  grants or offer other assistance to leverage local resources to address  coastal issues. For example, Florida’s program competitively awards a  portion of its administrative funds annually through grants to coastal  counties and municipalities for projects that help communities address a  wide range of coastal issues, and these grants require local entities to  match the state grants. Similarly, Maine’s program uses CZMP funds  annually to provide competitive grants to coastal communities for  planning activities that support harbor management and development or  improve shoreline access, but actual implementation of the projects must  be funded through other sources.
section_title: Limitations Exist with NOAA’s Performance Assessment Tools, and NOAA Makes Limited Use of Performance Information
paragraphs: NOAA’s two primary performance assessment tools, the CZMP  performance measurement system and its state program evaluations,  have limitations, even with changes NOAA has made since 2008, and  NOAA uses the performance information it collects to a limited extent in  managing the CZMP. We found that NOAA’s CZMP performance  measurement system does not align with some key attributes of  successful performance measures. In addition, in its method for selecting  stakeholders to survey during state program evaluations, NOAA may be  susceptible to collecting incomplete and biased information because, in  part, it uses a single criterion to select stakeholders to survey.  Furthermore, NOAA makes limited use of the performance information it  collects—for instance, NOAA does not use data from its performance  measurement system or its evaluations of state programs to improve  implementation of the CZMP at the national level—and, as a result, may  not be realizing the full benefit of collecting such information.
subsections: section_title: NOAA’s CZMP Performance Measurement System Does Not Align with Some Key Attributes of Successful Performance Measures
paragraphs: NOAA’s CZMP performance measurement system, which the agency  developed in response to congressional direction to assess the national  impact of the CZMP, has limitations, even with changes the agency made  to the system since our 2008 report. Specifically, NOAA has made  changes to several aspects of the data collection and review components  of its system, including the following: establishing a requirement, in 2010, that state programs submit  documentation of source information to support their data  submissions, such as documentation of the public access sites being  reported for public access performance measures;   refining, in 2009, 2010, and 2011, the names and definitions of some  performance measures with the intention of clarifying the activities  that a given measure is intended to capture; and     issuing internal guidance, in 2010, for NOAA staff to review state- submitted data and accompanying documentation to ensure that only  eligible activities are reported by the states, among other things.
With these changes, the system aligns with some key attributes of  successful performance measures. In our past work, we found that  successful performance measures typically align with key attributes  including reliability, clarity, balance, numerical targets, and limited  overlap, among others (see app. III for a complete list of key attributes we  identified). In our current review, we found that some of the changes  NOAA made to its CZMP performance measurement system since 2008  are consistent with such key attributes. For example, NOAA’s  requirement that state programs submit documentation of source  information and its internal guidance for how staff are to review this  documentation correspond with the key attribute of ensuring the reliability  of performance measures. In addition, NOAA’s steps to refine the  names and definitions of certain performance measures are  demonstrative of the key attribute of clarity, meaning that measures are  clearly stated and have names and definitions consistent with the  methodology used to calculate them.
On the other hand, we found limitations in the CZMP performance  measurement system that did not align with the key attributes. For  instance, in 2011, NOAA eliminated its coastal water quality focus area— corresponding to one of the six focus areas based on goals of the CZMP  outlined in the act. In eliminating this focus area, NOAA removed five  related performance measures; states continue to report on one measure  related to coastal water quality, but do so under another focus area on  coastal community development. Balance, or having a set of measures  that cover a program’s various goals, is a key attribute of successful  performance measures. We found that having measures that correspond  to various program goals provided agencies with a complete picture of  performance. NOAA officials indicated that they eliminated the coastal  water quality focus area based on a 2011 performance measurement  system workgroup’s recommendation to streamline the measurement  system. They further explained that they took this action because state  programs were no longer receiving coastal nonpoint program funding,  which often funded activities in support of coastal water quality, and that  activities under this focus area were often tied to the coastal community  development focus area. In speaking with some state program officials,  however, we found that improving coastal water quality remains a priority  for their programs even without coastal nonpoint program funding.
Similarly, representatives from the Coastal States Organization’s coastal  water quality workgroup indicated that many state programs have made  progress in developing and implementing coastal nonpoint pollution  control programs, but that these results are not quantified by NOAA.
In addition, NOAA has not established numerical targets for the measures  in its CZMP performance measurement system for the purpose of  tracking progress or assessing performance of the CZMP. Our past work  found that numerical targets are a key attribute of successful performance  measures because they allow managers to compare planned  performance with actual results. In 2008, we recommended that NOAA  establish numerical targets for performance measures that would help  track progress toward meeting program goals and help assess the overall  CZMP effectiveness. NOAA’s 2011 performance measurement system  workgroup also recommended that NOAA set targets to help it more  effectively measure and communicate CZMP performance. NOAA agreed  with these recommendations, but it has not established numerical targets  for the measures in its CZMP performance measurement system to  assess CZMP performance. NOAA officials explained that state programs  vary widely, making it difficult to set targets at the national level. Officials  also said that they first need to review the performance measures before  they assess the feasibility of developing numerical targets. NOAA officials  added that NOAA has set numerical targets for four CZMP performance  measures, which are included in Commerce’s department-wide goals  related to environmental stewardship. NOAA officials told us that they  considered historical performance measure data and state programs’  planned strategies when establishing these targets, but they do not use  them to assess CZMP performance. We continue to believe that, without  setting numerical targets for the CZMP performance measurement  system, NOAA will not have a benchmark to help it determine the extent  to which the CZMP may be meeting expectations.
Finally, the CZMP performance measurement system includes  performance measures that involve the collection of data by state  programs that are already available to NOAA from other sources. Limited  overlap, another key attribute of successful performance measures, notes  that measures should produce new information beyond what is provided  by other data sources and that redundant or unnecessary performance  information costs resources and clouds the bottom line by making  managers sort through excess information. We found that the CZMP  performance measurement system includes at least two financial  measures whereby states collect and submit financial expenditure data  similar to data states already provide NOAA through their cooperative  agreements. NOAA officials told us that, in developing the CZMP  performance measurement system, they anticipated that including such  measures would be useful for tracking the amount of CZMP funding used  in different focus areas each year. However, NOAA used the financial  information from its CZMP performance measurement system to prepare  a one-time summary of performance measure data published in 2013. In  contrast, it uses financial information drawn from cooperative agreements  on an annual basis to analyze states’ planned uses of CZMP funding.  NOAA officials acknowledged that they may need to review the utility of  requiring state programs to collect financial expenditure data for the  performance measurement system. By requiring states to collect and  submit financial data similar to data that they already provide in their  cooperative agreements and making limited use of these data, NOAA  may be unnecessarily burdening state programs with data collection  requirements.
Several state program officials we interviewed told us that collecting data  for the numerous data elements under the 17 performance measures is a  time- and resource-intensive activity, with a few stating that this is  particularly true relative to the amount of CZMP funds they receive. Some  indicated, for instance, that they spend 30 staff days or more per year  collecting these data. State officials said that, in particular, data for the  financial measures are among the most time-consuming to collect and  report to NOAA. Other state officials told us that collecting data on the  number of educational and training events and participants for each focus  area is especially time-consuming, with one official noting that collecting  data on number of participants is particularly burdensome when events  are hosted by parties other than the program itself.
NOAA officials told us they recognized the need to continue to review and  potentially streamline or revise the CZMP performance measurement  system, and that they intend to do so once the merger of OCRM and the  Coastal Services Center is complete, which they expect to occur by the  end of 2014. In the interim, NOAA officials said they initiated at the  beginning of fiscal year 2014 an effort to assess all performance  measures collected by the various programs within the two offices,  including the CZMP, to determine which measures may be most effective  in tracking and communicating progress toward goals identified in the  merged office’s strategic plan. NOAA officials said they are committed to  developing a strong framework for evaluating the performance of all  programs under its merged coastal management office. However, the  agency has not documented the approach it plans to take for these  efforts. Federal internal control standards state the need for federal  agencies to establish plans that encompass actions the agency will take  to help ensure goals and objectives can be met. Without a documented  approach for how it plans to assess its CZMP performance measurement  system—including the scope and criteria it will use, such as how it will  ensure its measures align with key attributes of successful performance  measures—NOAA cannot demonstrate that its intended effort will  improve its CZMP performance measurement system.
section_title: NOAA Cannot Ensure That Its Revised Process for Surveying Stakeholders to Inform State Program Evaluations Will Produce Complete and Unbiased Information
paragraphs: In 2013, NOAA revised its process for conducting state program  evaluations, which are required under the Coastal Zone Management Act  to assess state programs’ adherence to the act’s requirements, but we  identified a limitation in NOAA’s method for sampling stakeholders under  the revised process. According to NOAA documents, the purpose of the  revisions was to conduct evaluations more efficiently, at a reduced cost,  while continuing to meet evaluation requirements outlined in the act. In  revising its state program evaluations, NOAA made changes in the timing  and methods for collecting information from participating states (see table  4). A NOAA official estimates that the agency’s revised evaluation  process will save the agency approximately $236,000 annually. NOAA  began evaluating state programs using its revised process at the  beginning of fiscal year 2014 with evaluations of seven state programs.   We did not evaluate NOAA’s implementation of its revised state program  evaluations because NOAA had not completed its first cycle at the time of  our review and, therefore, it was too early to assess the effectiveness of  its revisions. However, we did assess NOAA’s revised evaluation design  against our and others’ work on program evaluations to identify standards  for strong evaluation design. We were unable to evaluate the qualitative  components of its revised evaluation design—including the change in the  scope of the evaluations from NOAA’s review of all aspects of each state  program to a review of a few areas determined by NOAA—because the  results of using these methods cannot be fully assessed until the  evaluations have been conducted. But, we did evaluate the steps NOAA  laid out in its guidance on its methods for collecting information and  identified a limitation in its method for sampling stakeholders to survey.
Under its revised evaluation process, NOAA relies in part on information  obtained through stakeholder surveys, but we found that through its  method of sampling stakeholders to survey, the agency may be  susceptible to collecting incomplete and biased information. According to  NOAA guidance on its revised evaluations, stakeholder surveys are  intended to provide information about stakeholders’ perspectives and  opinions across a range of topics, from a state program’s top three  strengths and weaknesses to opportunities for improving a program’s  federal consistency and permitting processes. The guidance states that  NOAA will use stakeholder survey responses to identify evaluation target  areas, as well as obtain information about the extent to which a state  program is performing effectively in areas outside of the target areas.  NOAA officials indicated that they plan to analyze survey results by  collating respondents’ answers to identify common themes. NOAA  evaluators will identify a sample of stakeholders to survey from 12  categories of organizations that stakeholders represent, including federal  agencies, state agencies, nonprofit organizations, academic institutions,  and local businesses and industries. According to NOAA officials, they  adopted the criterion of stakeholder categories to ensure that  stakeholders whose views were not consistently represented in the  former evaluations—such as those from local businesses and  industries—are included in evaluations conducted under the revised  process. NOAA evaluators will select stakeholders to survey from these  12 categories from a list of potential stakeholders to survey compiled by  state program officials and NOAA specialists working with the state.
According to the Office of Management and Budget’s Standards and  Guidelines for Statistical Surveys, a survey sampling method should yield  the data required to meet the objectives of the survey. Our previous  work has found that strong program evaluations rely on data that  sufficiently reflect the activities and conditions a program is expected to  address. Because NOAA’s stakeholder sampling method is guided by  one criterion—categories of stakeholder organizations—NOAA may not  collect information that reflects the various activities and aspects of the  state programs. Specifically, under the act, NOAA is required to evaluate  the extent to which state programs have addressed coastal management  needs reflecting the six focus areas based on the goals identified in the  act. In the absence of additional criteria for selecting stakeholders to  survey, NOAA may select a sample of stakeholders whose work with a  state program does not span all of the act’s goals, potentially leaving  NOAA without information to inform its evaluation of a state’s  performance on one or more goals. Such an information gap could be  significant because stakeholder surveys are intended to be a main source  of information on how well a program is performing in areas beyond those  identified as target areas.
Furthermore, when using a nonprobabilistic sampling method, such as  that being employed by NOAA for its stakeholder surveys, the Office of  Management and Budget’s survey guidelines state that agencies should  demonstrate that they used an impartial, objective method to include or  exclude people or organizations from a sample. Our previous work on  program evaluation also found that evaluation data should be sufficiently  free of bias or other errors that could lead to inaccurate conclusions.  Because state program officials responsible for identifying potential  stakeholders to survey have a vested interest in their programs, NOAA’s  process is susceptible to collecting biased information. NOAA specialists  who work with state programs also contribute to the selection process.  However, we found that some NOAA specialists are not regionally located  or have worked with a state program for a short period of time and,  therefore, their knowledge or experience to inform the selection process  may be limited. NOAA’s evaluation guidance recognizes the need to  assess its revised process in the future and states that the agency plans  to evaluate the effectiveness and efficiency of its revised state program  evaluation process after conducting 8 to 10 evaluations.
section_title: NOAA Makes Limited Use of Performance Data It Collects in Managing the CZMP
paragraphs: We found that in managing the CZMP, NOAA makes limited use of the  performance information it collects. Our past work has found that  performance information can be used across a range of management  functions to improve programs and results, including to (1) identify  problems or weaknesses in programs and take corrective actions, (2) set  program priorities and develop strategies, (3) recognize and reward  organizations who meet or exceed expectations, and (4) identify and  share effective approaches to program implementation. For example,  our previous work found that the Department of Labor effectively used  performance measure data to identify technical assistance needs of state  programs and to then provide assistance to try to improve performance.  The department also used performance measure data as a basis for  providing financial incentives to state programs that receive federal  grants. We found that agencies realize the full benefit of collecting  performance information only when they use such information to make  decisions designed to improve results.
NOAA collects performance information through its CZMP performance  measurement system, state program evaluations, and other sources, but  we found that the agency generally does not use the information it  collects to help manage the CZMP at a national level. Specifically, we  found the following:
NOAA uses its CZMP performance measurement system data to  report on national program accomplishments on a limited basis. In  particular, in 2013, NOAA produced one report summarizing  performance measurement system data from 2008 through 2011.  However, NOAA has not published additional similar reports, and has  not used performance measurement system data for other purposes.  For example, the agency has not used the performance measurement  system data to identify potential problems or weaknesses in the  CZMP, set program priorities or strategies, or recognize and reward  high-performing state programs—which may limit the usefulness of  collecting such data.
NOAA does not use its state program evaluations to assess the  performance or improve the implementation of the CZMP at the  national level. NOAA uses its state program evaluations to identify  state-specific accomplishments and encourage or require the state  under evaluation to make improvements or take corrective actions.  But, according to NOAA officials, the agency does not regularly  analyze findings from individual state evaluations to identify and share  effective approaches across states or to identify common  performance weaknesses that may warrant national focus or  assistance. Our analysis of recent NOAA evaluations of the seven  state programs we reviewed found that NOAA recommended the  states undertake similar actions. In five of the seven state program  evaluations, for example, NOAA recommended that programs  undertake strategic planning, and for four of the seven programs,  NOAA recommended that programs improve their coordination with  local governments or other partners who help carry out coastal  management activities. Yet NOAA has not analyzed these evaluations  to identify common findings. One NOAA specialist we spoke with  suggested that NOAA could also use the results of its state program  evaluations to recognize and reward high-performing state programs.  For instance, the NOAA specialist suggested that NOAA could modify  its eligibility requirements for its Projects of Special Merit funding such  that only high-performing programs, with any necessary actions from  past state program evaluations fully implemented, would be eligible to  receive funding.
NOAA does not use performance-related information from other  sources to support its management of the CZMP. NOAA uses state  programs’ semiannual progress reports—which contain, among other  things, “success stories,” or examples of a state program successfully  addressing coastal management issues—to track states’ progress in  implementing their cooperative agreements. However, NOAA does  not use information from these reports to identify and promote  effective approaches to coastal management by regularly sharing  states’ success stories across states or with other stakeholders. The  2011 performance measurement system workgroup composed of  NOAA and state program officials recommended that NOAA develop  a website to share success stories on an annual basis. NOAA did not  implement this recommendation because, according to NOAA  officials, at that time it was incorporating success stories into a  quarterly newsletter. According to a NOAA document, the agency  produced the newsletter in response to requests from states for more  information about how other state programs address coastal  management issues. NOAA stopped issuing this newsletter in 2012,  when its office merger began, and NOAA officials said they are now  evaluating how the merged office might best share information about  the CZMP across state programs and with other stakeholders.
NOAA’s strategic plan for its merged coastal management office  recognizes the importance of using and reporting performance  information. According to this plan, NOAA is committed to maintaining a  culture of monitoring and evaluation to improve the implementation of its  programs. We found, however, that the strategic plan does not include a  documented strategy for using the performance data NOAA collects  through its CZMP performance measurement system, state program  evaluations, or other sources of information, such as states’ semiannual  progress reports, to manage the CZMP. NOAA officials told us that  because the office merger is under way, they have not formulated a  strategy for how the merged office will use performance data to inform  and manage the CZMP, but they recognized the need to do so once the  merger is complete. Federal control standards state the need for federal  agencies to document management approaches to ensure goals and  objectives can be met. Without a documented strategy for using the full  range of performance information it collects, NOAA may not be taking full  advantage of the performance information that its specialists, evaluators,  and state program officials spend time and resources collecting, and it  cannot ensure that it is realizing the full benefit of collecting such  information, such as identifying common problems in state programs and  taking corrective actions, setting national program priorities and  developing strategies, recognizing state programs that exceed  expectations, or identifying and sharing effective approaches to program  implementation.
Finally, NOAA has not taken steps to integrate data from its CZMP  performance measurement system with information from its state  program evaluations to develop a complete picture of the CZMP’s  performance, as we recommended in our 2008 report. In 2008, we found  that NOAA was not integrating quantitative national performance measure  data with qualitative information from state program evaluations to  develop a more comprehensive assessment of the CZMP’s performance.  NOAA agreed with our recommendation to develop an approach for  integrating the two types of information and, in response, tasked the 2011  performance measurement system workgroup with developing a method  for better communicating performance measure data. The workgroup  recommended a template for communicating program results that  includes quantitative national performance measure data and qualitative  success stories from states’ semiannual progress reports. However,  NOAA has not drawn on this quantitative and qualitative information for  purposes other than producing a report in 2013 summarizing performance  measurement system data. Specifically, NOAA has not integrated  quantitative and qualitative information to better understand program  performance, improve its assessment of difficult-to-measure activities, or  validate its assessments of program progress. We have previously found  that agencies that used multiple sources of data to assess performance  had information that covered more aspects of program performance than  those that relied on a single source. We also found that agencies can  improve their performance assessments by using program evaluation  information to validate performance measurement system data. We  continue to believe that developing an approach to combine performance  information from its CZMP performance measurement system and state  program evaluations could help NOAA obtain a more complete picture of  CZMP performance.
section_title: Conclusions
paragraphs: The CZMP plays an integral role in helping states protect, restore, and  manage the development of the nation’s coastal resources and habitats.  In managing the CZMP, NOAA is challenged with the task of assessing  the performance of the program, composed of partnerships with 34  individual states, each with unique coastal habitats, and differing laws,  organizational structures, and funding priorities. NOAA is to be  commended for its progress in improving its two primary performance  assessment tools—its CZMP performance measurement system and  state program evaluations—since we last reviewed the agency’s  performance assessment processes in 2008. We are encouraged by  NOAA’s recognition of the importance of using performance information  to improve the implementation of the CZMP. However, NOAA does not  use or have a documented strategy for how it will use the performance  information it collects from its CZMP performance measurement system,  state program evaluations, or other sources of performance-related  information, as appropriate, to aid its management of the CZMP. Without  a documented strategy for using the range of its performance information,  NOAA cannot ensure that it is collecting the most meaningful information  and realizing the full benefit of the significant amount of information it and  the states collect, such as identifying common problems in state  programs and taking corrective actions, setting national program priorities  and developing strategies, recognizing state programs that exceed  expectations, or identifying and sharing effective approaches to program  implementation.
We also are encouraged by NOAA’s intentions to review and possibly  revise the CZMP performance measurement system once its new coastal  office is in place, but the agency has yet to document the approach it  plans to take—including the scope and criteria it will use for this effort. In  the absence of a documented approach indicating how it will review its  performance measurement system, NOAA cannot ensure that its  upcoming effort will take into consideration key attributes of successful  performance measures, including balance and limited overlap, or result in  a system that provides meaningful information that can be used by NOAA  to determine how effectively the CZMP is performing relative to its goals.  We are further encouraged by NOAA’s commitment to evaluate the  effectiveness and efficiency of its revised state program evaluation  process and to modify it, as needed, as it moves forward with its  implementation. In the interim, however, NOAA’s method for selecting  stakeholders to survey during state program evaluations—which relies on  a single criterion and on state program officials who have a vested  interest in the program—may result in the collection of incomplete or  biased information that does not ensure perspectives are gathered from  stakeholders representing a variety of program goals and are collected in  an objective manner, potentially undermining the sufficiency and  credibility of the data the produces. In the absence of additional criteria  for selecting stakeholders to survey, NOAA may select a sample of  stakeholders whose work with a state program does not span the act’s six  focus areas or who present less-than-objective assessments of a state  program.
section_title: Recommendations for Executive Action
paragraphs: To ensure that NOAA collects and uses meaningful performance  information to help manage the CZMP, including continuing to improve its  CZMP performance measurement system and its state program  evaluations, we are recommending that the Secretary of Commerce direct  the Administrator of NOAA to take the following three actions:
Develop a documented strategy to use the range of performance  information the agency collects, as appropriate, to aid its management  of the CZMP, such as to identify potential problems or weaknesses in  the CZMP; set program priorities or strategies; or recognize and  reward high-performing state programs.
As part of its intended review of the CZMP performance measurement  system and in consideration of how it intends to use the performance  information, document the approach it plans to take to analyze and  revise, as appropriate, the performance measures, and in so doing  ensure the analysis considers key attributes of successful  performance measures, such as balance and limited overlap.
Revise the sampling methodology for selecting stakeholders to  survey—included as part of its state program evaluation process—to  ensure perspectives are gathered from stakeholders representing a  variety of program goals and are collected in an objective manner.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Department of Commerce for  review and comment. In written comments provided by NOAA through  Commerce (reproduced in appendix IV), NOAA generally agreed with our  findings and concurred with our recommendations. NOAA also provided  technical comments that we incorporated, as appropriate. In its comment  letter, NOAA stated that while it found GAO’s evaluation of the CZMP  performance measurement system accurate, the agency did not agree  with GAO’s assessment that eliminating a stand-alone category for  coastal water quality could negatively affect the system’s ability to reflect  the goals of the CZMA in a balanced way. NOAA stated that removal of  the coastal water quality focus area did not impair its ability to track  progress in meeting the water quality goal of the CZMA, explaining that it  retained one measure composed of two data elements related to coastal  water quality, but housed under a different focus area. We agree that the  two-part measure NOAA maintained related to coastal water quality may  provide important information on performance in this area. However, we  continue to believe that the information it is collecting related to coastal  water quality may not be balanced in comparison to the information it is  collecting for the other five focus areas, which could in turn result in  inconsistent performance information when looking across the six focus  areas of the program. NOAA concurred with the three recommendations  in the report and described actions it plans to address them. With regard  to the first recommendation, NOAA stated that it plans to develop a  strategy for using performance information it collects, including  information from its performance measurement system, evaluations of  state programs, performance reports, and other sources, and noted that it  will build upon existing efforts to share lessons-learned regarding  successful approaches or shared challenges across the national program.  In addressing our second recommendation, on documenting its approach  for analyzing and revising, as appropriate, the performance measures,  NOAA stated that it plans to conduct a review of CZMP performance  measures in fiscal year 2015 as part of its ongoing analysis of  performance measures for programs under its new coastal office. In  response to our third recommendation, NOAA stated that it will revise its  sampling methodology to ensure stakeholders representing a variety of  program goals are selected.
We are sending copies of this report to the Secretary of Commerce, the  appropriate congressional committees, and other interested parties. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-3841 or fennella@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. GAO staff who made major contributions to  this report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Focusing on National Coastal Zone Management Program (CZMP)  activities since our 2008 report, our objectives were to examine (1) how  participating states allocated CZMP funds awarded in fiscal years 2008  through 2013 and (2) how the National Oceanic and Atmospheric  Administration’s (NOAA) primary performance assessment tools have  changed and the extent to which NOAA uses performance information to  help manage the CZMP.
To examine how participating states allocated CZMP funds awarded in  fiscal years 2008 through 2013, we reviewed the Coastal Zone  Management Act and related regulations and guidance, including NOAA  funding guidance and allocation memos. We analyzed NOAA data on  federal funds awarded by state and by funding type from fiscal years  2008 to 2013, and we compared this data against annual NOAA funding  guidance and allocation memorandums to states. Based on our analysis,  and interviews with NOAA officials, we found the data to be sufficiently  reliable. We reviewed NOAA’s analysis of states’ allocations of CZMP  funding for fiscal years 2008 through 2013, which was based on NOAA’s  review of its cooperative agreements for federal funding with states.  NOAA’s analysis involved the categorization of states’ funding allocations  for projects into six focus areas based on the goals of the act and an  additional state program management category as defined by NOAA to  cover administrative costs, such as general program operations, supplies,  and rent. NOAA officials noted that total funding allocation amounts are  approximate and that many CZMP funded activities could address more  than one focus area. For example, Maine state program officials told us  their activities to conserve and enhance properties that provide  commercial fishing access address both coastal community development  and public access focus areas. To address this challenge, NOAA  developed written guidance for NOAA specialists who conduct the  analysis that specifies the types of activities to include in each focus area  and the state program management category, as well as direction on how  to categorize funds in cases where a project or activity may fall in more  than one category. For instance, NOAA defined funds in the government  coordination focus area to include, among others, activities that involved  coordination with other government agencies and stakeholders, technical  assistance to local governments, or public outreach and education  activities only if they did not correspond to other focus areas. To  determine the reliability of NOAA’s analysis, we interviewed  knowledgeable NOAA officials, reviewed NOAA’s process for categorizing  proposed activities and projects, including its written guidance on  categorizing CZMP-funded activities and its steps to compare funding  amounts to ensure that the double-counting of funds did not take place.
We did not independently verify the results of NOAA’s analysis, but we  verified major categories used in NOAA’s analysis for consistency across  years, checked the total allocated funds in NOAA’s analysis against total  federal funding award data, and reviewed NOAA’s categorization of a  small sample of projects. We concluded the data to be sufficiently reliable  for our purposes of reporting states’ allocated uses of CZMP funds.
We also reviewed data from NOAA’s CZMP performance measurement  system from 2008 through 2013 (the most recent years for which data  was available) to further illustrate how CZMP funds were used. To assess  the reliability of NOAA’s CZMP performance measurement system data,  we interviewed NOAA officials about reliability of the data and reviewed  corresponding documentation including performance measures guidance  to states and internal guidance to NOAA specialists about their required  reviews of data submitted. We did not independently verify performance  measure data submitted by state programs, but based on our review of  steps taken by NOAA to review state-submitted data, we found the data  sufficiently reliable for the purposes of our report.
To examine how NOAA’s primary performance assessment tools have  changed since 2008, and the extent to which NOAA uses performance  information to help manage the CZMP, we analyzed applicable laws and  guidance including the act, and NOAA’s guidance on its CZMP  performance measurement system and state program evaluations. We  reviewed documentation on changes NOAA has made to these two  performance tools, including steps taken to address our 2008 report  recommendations, and we interviewed NOAA officials about the changes  they made and their use of performance information. We reviewed GAO’s  work on performance measurement to identify key attributes associated  with successful performance measures and assessed NOAA’s CZMP  performance measurement system against these attributes by reviewing  the agency’s performance measures and guidance on the system and  interviewing NOAA and state program officials. We also analyzed NOAA’s  CZMP performance measurement system data from 2011, 2012, and  2013. We reviewed our and others’ work on program evaluations to  identify standards for strong evaluation design and assessed NOAA’s  process for evaluating state coastal programs against these standards by  examining NOAA’s evaluation guidance and interviewing NOAA officials.  We examined information NOAA maintains on CZMP performance  including fact sheets, states’ cooperative agreements, semiannual  progress reports, performance measurement system data submitted by  states, and state program evaluation reports.
In conducting our work on both objectives, we interviewed representatives  of the Coastal States Organization, a nonprofit organization that  represents coastal states on legislative and policy issues, as well as state  program officials from the seven states that received the most  fiscal year  2012 CZMP funding in each of NOAA’s seven regions (California, Florida,  Hawaii, Maine, Michigan, Texas, and Virginia) about how states used  CZMP funds and for their perspectives on NOAA’s management and  assessment of the overall national program. We also reviewed the seven  states’ cooperative agreements and semiannual progress reports for  fiscal years 2011 and 2012 (the most recent years for which reports were  available) to learn about projects undertaken by these seven states. We  selected one CZMP-funded project in each of the seven states to further  determine and illustrate how states used funds on a project-level basis  and to learn about how the results of a select project are captured by  NOAA’s performance assessment tools. In selecting projects to review,  we considered the amount of CZMP funds allocated to specific projects,  funding type, project type (e.g., projects that provide financial and  technical assistance to local governments, planning projects,  construction-related projects, permitting activities), and focus area (e.g.,  coastal habitat, government coordination). Our review of the states’  information cannot be generalized across all states or projects. We also  interviewed coastal program officials from American Samoa and the  Northern Mariana Islands to obtain perspectives from territories on  NOAA’s performance assessment tools and territories’ use of this  performance information.
We conducted two site visits to observe and learn more about CZMP  projects—one to a coastal habitat restoration project in Texas and one to  an ocean planning project in Virginia. We selected these projects for site  visits considering project type, focus area addressed, and geographic  location. During our site visits, we met with state program officials and  also interviewed stakeholders involved in the selected projects, as well as  stakeholders involved in other CZMP-funded projects. In Texas, we met  with the nonprofit organization managing the coastal habitat restoration  project and toured the restoration site; in Virginia, we visited a public  access enhancement project that received CZMP funding.
We conducted this performance audit from June 2013 to July 2014 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: National Coastal Zone Management Program (CZMP) Performance Measurement System Summary Data
paragraphs: The National Oceanic and Atmospheric Administration’s (NOAA) CZMP  performance measurement system is organized by broad focus areas that  are related to five of the six primary focus areas based on the goals of the  CZMP as outlined in the Coastal Zone Management Act. The system  consists of 17 performance measures—15 of the 17 measures are  organized under the five broad focus areas (NOAA removed the sixth  focus area, coastal water quality, from its performance measurement  system in 2011 in response to a performance measurement system  workgroup’s recommendation to streamline the system), and the  remaining 2 measures are to track state financial expenditures. Each of  the 17 measures is composed of several individual data elements. For  example, the performance measure on federal consistency is composed  of two data elements that track the number of projects reviewed and the  number of projects modified under states’ federal consistency review  processes. In addition, some data elements are further broken down into  specific categories, such as types of federal consistency projects  modified. See table 5 for a list of the performance measures and  supporting data elements and categories, as reported by participating  state programs for 2011 through 2013.
section_title: Appendix III: Key Attributes of Successful Performance Measures Previously Identified by GAO
subsections: section_title: Attributes Linkage
paragraphs: Definition  Measure is aligned with division and agency-wide  goals and mission and clearly communicated  throughout the organization.
Potentially adverse consequences of not meeting  attribute  Behaviors and incentives created by measures may  not support achieving division or agency-wide goals or  mission.
Measure is clearly stated and the name and definition  are consistent with the methodology used to calculate  it.
Data may confuse or mislead users.
Measure has a numerical target.
Managers may not be able to determine whether  performance is meeting expectations.
Measure is reasonably free from significant bias or  manipulation.
Performance assessments may be systematically  over- or understated.
Measure produces the same result under similar  conditions.
Reported performance data may be inconsistent and  add uncertainty.
Measures cover the activities that an entity is expected  to perform to support the intent of the program.
Information available to managers and stakeholders in  core program areas may be insufficient.
Measure provides new information beyond that  provided by other data sources.
Manager may have to sort through redundant, costly  information that does not add value.
Taken together, measures ensure that an  organization’s various priorities are covered.
Measures may over emphasize some goals and skew  incentives.
Each measure should cover a priority such as quality,  timeliness, and cost of service.
A program’s overall success is at risk if all priorities are  not addressed.
section_title: Appendix IV: Comments from the Department of Commerce
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Alyssa M. Hundrup (Assistant  Director), Elizabeth Beardsley, Mark A. Braza, Elizabeth Curda, John  Delicath, Tom James, Katherine Killebrew, Patricia Moye,  Dan Royer,  Kiki Theodoropoulos, and Swati Sheladia Thomas made key contributions  to this report.