Pacific Groundfish: Continued Efforts Needed to Improve Reliability of Stock Assessments
Released: 2004-06-14T00:00:00

Letter

Background

The Magnuson-Stevens Fisheries Conservation and Management Act granted responsibility for managing marine resources to the Secretary of Commerce. The Secretary delegated this responsibility to NMFS, which is part of Commerce’s National Oceanic and Atmospheric Administration (NOAA). The act also established eight regional fishery management councils, each responsible for making recommendations to the Secretary of Commerce about managing fisheries in federal waters. The eight fishery management councils—consisting of fishing industry participants, state and federal fishery managers, and other interested parties—and their areas of responsibility include the following: Caribbean Council, covering waters off the U.S. Virgin Islands and the Commonwealth of Puerto Rico; Gulf of Mexico Council, covering waters off Texas, Louisiana, Mississippi, Alabama, and the west coast of Florida; Mid-Atlantic Council, covering waters off New York, New Jersey, Delaware, Maryland, Virginia, and North Carolina; New England Council, covering waters off Maine, New Hampshire, Massachusetts, Rhode Island, and Connecticut; North Pacific Council, covering waters off Alaska; Pacific Council, covering waters off California, Oregon, and Washington; South Atlantic Council, covering waters off North Carolina, South Carolina, Georgia, and the east coast of Florida; and Western Pacific Council, covering waters off Hawaii, American Samoa, Guam, the Commonwealth of the Northern Mariana Islands, and uninhabited U.S. territories in the Western Pacific.

In addition to these eight councils, NMFS has six regional science centers, which are responsible for generating the scientific information necessary for the conservation, management, and use of each region's marine resources. The six fishery science centers and their areas of responsibility are as follows: Alaska Center, covering the coastal oceans off Alaska and parts of the west coast of the United States; Northeast Center, covering waters along the Northeast Continental Shelf from the Gulf of Maine to Cape Hatteras, North Carolina; Northwest Center, covering the northeast Pacific Ocean, primarily waters off the coasts of California, Oregon, Washington, and British Columbia; Pacific Islands Center, covering the central and western Pacific Ocean; Southwest Center, primarily covering waters off the coast of California and areas throughout the Pacific and Antarctic Oceans; and Southeast Center, covering waters along the continental southeastern United States as well as Puerto Rico and the U.S. Virgin Islands.

The Magnuson-Stevens Act, as amended by the Sustainable Fisheries Act, also established national standards for fishery conservation and management. These standards deal with preventing overfishing, using scientific information, using fishery resources efficiently, minimizing bycatch, and minimizing administrative costs. The regional councils use these standards to develop appropriate plans for conserving and managing fisheries under their jurisdiction, including measures to prevent overfishing and to rebuild overfished stocks as well as measures to protect, restore, and promote the long-term health and stability of the fishery.

In 1982, the Pacific Fishery Management Council (Pacific Council) released its initial Fishery Management Plan for groundfish. The Pacific Council’s goal is to have long-range plans for managing groundfish fisheries that will promote a stable planning environment for the seafood industry, including marine recreation interests, and will maintain the health of the resource and environment. To help achieve these goals, stock assessments are conducted on groundfish species to estimate fish populations. Since 1995, the Northwest Center has had lead responsibility for conducting stock assessments on Pacific groundfish. The Northwest Center receives assistance from other NMFS science centers, such as the Southwest Center, which conducted the bocaccio assessment.

Stock assessments are the biological evaluation of the status of fish stocks. Stock assessments provide official estimates in key areas, such as the size of the stock population, the size of the spawning population, the amount of fish that have died (fish mortality), and the estimated number of fish at a particular young age (recruitment). Stock assessments form the scientific basis used by regional councils to determine biologically sustainable harvests and guide the monitoring and rebuilding of overfished and threatened stocks. For example, regional councils use stock assessments and other indicators of biological productivity to recommend to NMFS a maximum, or total allowable catch, in a particular fishery—typically for a year. Stock assessments are a key tool for managing fisheries. Without stock assessments, fishery managers would have limited information about the status of fisheries in making decisions about setting harvest levels and developing plans to rebuild overfished stocks.

For each species, the assessor reviews previous stock assessments, gathers available data about the species being assessed, runs the data through computer-generated models, and estimates the species’ total biomass. Stock assessors use NMFS-collected data, such as stock surveys conducted on NOAA vessels or contracted commercial fishing vessels, as well as data collected by non-NMFS sources, such as commercial and recreational catch data collected by state agencies. The following six key types of data are collected: Stock abundance—surveys of how many fish constitute a stock’s total size or weight.

Commercial and recreational fisheries data—the amount and composition of fish caught from a particular stock, whether caught intentionally by commercial and recreational fishermen or unintentionally caught and discarded. Data sources include fishing logbooks, dockside samples, and onboard observations, among others.

Life history—biological data, such as the age and sex composition of the stock, age at first maturity, fertility, average lifespan, and natural mortality.

Ecosystem relationships—data on the relationship between a fish stock and its physical environment, as well the relationship of a fish stock to other species.

Recruitment research—data on the abundance of juvenile and larval fish (fish at their earliest stage), which helps scientists forecast the size of a particular stock in the future.

Synoptic oceanographic sampling—data on the ocean ecosystem, such as water temperature or salinity, plankton composition, or ocean currents.

For each stock assessment, a review panel, consisting of NMFS scientists and outside experts, independently reviews the methodology of the assessment and works with the assessor to ensure their comments are adequately addressed.

Through 2003, 24 of the 82 species of Pacific groundfish have had a full quantitative stock assessment. Relying on these assessments, NMFS has declared that nine species of Pacific groundfish are overfished, including the five species of Pacific groundfish we reviewed in this report—Pacific hake as well as bocaccio rockfish, canary rockfish, darkblotched rockfish, and yelloweye rockfish. Rockfish are long-lived, late-maturing and slow-growing species, making them particularly susceptible to overfishing. More specifically: Pacific hake, also called Pacific whiting, is generally found off the west coast of North America. It is one of many species of hake distributed in the Atlantic and Pacific oceans. Fishing of Pacific hake primarily takes place off the coasts of northern California, Oregon, Washington, and British Columbia. Fishermen use mid-water trawls and generally fish over the ocean bottom at depths of 100 to 500 meters. Pacific hake was declared overfished in 2002 because the 2002 stock assessment estimated Pacific hake biomass at 700,000 metric tons. By 2004, the biomass was estimated at between 2.7 and 4.2 million metric tons, and Pacific hake is no longer considered overfished. Figure 1 shows a picture of a Pacific hake.

Bocaccio rockfish generally inhabit waters off the coast of northern Baja, Mexico to Alaska. Bocaccio, commonly sold at market as “red snapper,” are commercially fished using trawls, hook-and-line and gillnets. Adult bocaccio commonly live over rocky areas or open areas of the ocean’s floor to about 320 meters. Bocaccio were formally declared to be “overfished” in 1999. The 2003 stock assessment estimated the bocaccio biomass at 7,133 metric tons. Figure 2 shows a picture of a bocaccio rockfish.

Canary rockfish inhabit the northeastern Pacific Ocean, from northern Baja, Mexico to the western Gulf of Alaska. Adult canary rockfish are primarily found along the continental shelf, from 46 to 457 meters deep. Canary rockfish are harvested commercially using trawl nets and hook-and-line and are also considered an important species for recreational fishermen. NMFS declared canary rockfish as overfished in 2000. The 2002 stock assessment estimated the canary rockfish biomass at 6,197 metric tons. Figure 3 shows a picture of a canary rockfish.

Darkblotched rockfish are found in the waters from Santa Catalina Island, California to the Bering Sea on soft bottom areas at about 29 to 549 meters deep. Commercial fishery concentrations are located off the coasts of California and Oregon. Darkblotched rockfish are caught primarily by commercial trawls and contribute to both commercial and recreational fishing. NMFS determined that darkblotched rockfish was overfished in 2000, when the last full stock assessment was conducted; it was updated in 2003. This update estimated the darkblotched rockfish biomass at 7,266 metric tons in 2001. Figure 4 shows a picture of a darkblotched rockfish.

Yelloweye rockfish almost exclusively inhabit rocky areas from northern Baja, Mexico to the Aleutian Islands, Alaska. Yelloweye rockfish, found in depths ranging from 15 to 550 meters, are caught using both trawl nets and line gear and are highly prized by both commercial and recreational fishermen. Stock assessments for yelloweye rockfish were first conducted in 2001, and NMFS determined that the species was overfished in 2002. The 2002 stock assessment estimated the yelloweye biomass at 2,325 metric tons in 2001. Figure 5 shows a picture of a yelloweye rockfish.

Reliability of NMFS’ Pacific Groundfish Stock Assessments Is Questionable

The reliability of NMFS’ stock assessments is questionable for the Pacific hake and four rockfish species we reviewed, although they were based on the best information available at the time the assessments were conducted. The reliability of the stock assessments we reviewed is questionable because (1) four of the assessments did not have at least one NMFS-collected data source of sufficient scope and accuracy; (2) NMFS lacked a standard process for assessing the reliability of non-NMFS data used in all five assessments; and (3) for four of the assessments, the stock assessment reports did not adequately identify the uncertainty of the biomass estimates. (See table 1.) To address these limitations, the Northwest Center plans to increase the scope and accuracy of its collected data, as additional funds become available; is implementing changes that will help ensure the reliability of non-NMFS data; and plans to update the stock assessment model to provide uncertainty ranges for the 2005 stock assessments.

Absence of NMFS Survey Data Adversely Affects the Assessments’ Reliability

Stock assessors use a variety of data, including NMFS data and non-NMFS data, in developing their assessments. Two key pieces of NMFS survey data are the shelf and slope bottom trawl survey and the acoustic survey. Other NMFS data that assessors sometimes use include larval surveys (data for fish in their earliest stage) and recruitment surveys. The non-NMFS data assessors use include commercial catch data and recreational catch data.

A 2002 National Research Council report found that the inclusion of NMFS survey data was the best option for a reliable estimate of abundance because such surveys use an unbiased statistical design, control sampling locations, and provide for quality assurance. According to Northwest Center officials, to obtain reliable results, each stock assessment should include at least one source of NMFS-collected data of sufficient scope and accuracy because such surveys are unbiased and scientifically designed.

Northwest Center officials raised concerns about basing assessments solely on non-NMFS data such as commercial and recreational catch data. Catch data do not provide the species’ relative or absolute biomass, according to NMFS officials. For example, catch data alone is insufficient because fishermen are not randomly sampling the ocean but are fishing areas that they are allowed to fish and they believe have the most fish; fishing restrictions, such as a total allowable catch, can limit the amount of fish being caught; and catch data have often been inaccurate for a variety of reasons, such as imprecise accounting for dead fish tossed back into the ocean.

Although the assessors used several different data sources, four of the five assessments did not use NMFS survey data or the NMFS data used covered only a portion of the species’ habitats. In the yelloweye assessment, no NMFS survey data were available because yelloweye live almost exclusively in the rocky habitat that NMFS trawl surveys cannot cover. As a result, the yelloweye assessment was based solely on non-NMFS data. Similarly, the NMFS survey data used in the bocaccio, canary, and darkblotched assessments were limited in scope because the surveys were conducted only in trawlable waters. Bocaccio, canary, and darkblotched live in both the trawlable and untrawlable habitats. Using data from trawl surveys conducted from 1977 through 1998, NMFS reported in 2003 that 77 percent of the survey area was trawlable and 23 percent was untrawlable. Lacking data on species in the 23 percent of the ocean floor that is untrawlable, the assessors estimated the overall biomass using the NMFS data collected from the trawlable areas. However, the abundance in the trawlable area is not representative of the abundance in the untrawlable area. The 2003 NMFS report also found that darkblotched groundfish are less abundant in untrawlable waters, while canary and bocaccio species are more abundant in untrawlable waters. As a result, some rockfish populations may be understated while others may be overstated. According to stock assessors, relying solely on survey data from trawlable waters increases the uncertainty of stock assessments.

In contrast, the fifth groundfish species, Pacific hake, lives primarily in mid-water habitat; and so the concerns about the lack of NMFS data in rocky, untrawlable habitats are not applicable.

NMFS Lacks a Standard Process for Ensuring the Reliability of Non-NMFS Data Used in the Stock Assessments

NMFS does not have a standard process for evaluating whether the non-NMFS data used in its stock assessments are reliable. We believe that certain internal control activities, such as a standard process for ensuring data reliability, can help ensure that information used to make management decisions is complete and accurate. Lacking a standard process, some assessors reviewed the quality of the raw non-NMFS data, while others did not. Assessors who reviewed for data quality found mistakes that they believed made some of the data unusable or that could have impaired the accuracy of the stock assessments. For example, the assessor for the 2002 yelloweye stock assessment found numerous errors in the recreational catch data, such as attributing the catch from an entire fishing vessel to a single fisherman, and thus did not use the data because doing so could have resulted in overestimating the biomass. According to another stock assessor, commercial catch data frequently have inconsistencies. Specifically, the assessor said California, Oregon, and Washington require fishermen to enter catch and location information into logbooks, but logbooks are often incomplete and inaccurate. While the stock assessment review panels evaluated the assessments, the panels did not evaluate the quality of the raw data used in the assessments.

According to a Northwest Center official, several assessors have raised concerns about data quality and accessibility in feedback meetings. In response to these concerns, the Northwest Center official has recently begun assigning data stewards to each data set used in its assessments. Data stewards are responsible for helping assessors compile relevant data and for conducting quality assurance and quality control checks on the data. The Northwest Center plans to conduct a data quality workshop in July 2004 to formally establish the roles and responsibilities of the data stewards, with the intent of standardizing the data evaluation process.

Stock Assessments for Four Species Did Not Provide Uncertainty Estimates

In 1998, the National Research Council recommended that NMFS include realistic measures of uncertainty in its stock assessments. NMFS’ 2001 stock assessment improvement plan also recognized the need to better quantify and communicate the uncertainty in assessments. In a review of the 2002 canary assessment, the stock assessment review panel recommended that standard estimates of uncertainty be included in future assessments because it is difficult to determine the reliability of the stock assessment without them. Similarly, we believe that estimates based on samples should have a range of uncertainty to show the amount of variability in the estimates. However, the bocaccio, canary, darkblotched, and yelloweye assessments did not present a measure of uncertainty associated with the biomass estimates.

Without uncertainty ranges, it is difficult for regional councils and NMFS to know how much confidence they can have in relying on the estimates for determining stock abundance and hence for setting allowable harvests of the fish. For example, lacking uncertainty ranges, the 2002 bocaccio stock assessment estimated a bocaccio biomass of 2,914 metric tons in 2002. The 2003 assessment of bocaccio biomass, however, estimated 6,506 metric tons in 2002—more than doubling the previous estimate because of additional and updated data. With such wide variations, it is important to provide uncertainty ranges, otherwise management may make inappropriate decisions.

While assessors told us that their stock assessments included some information about differences in estimated biomass when using different data sources (sensitivity analyses), the mathematical model that NMFS uses to estimate biomass (Stock Synthesis model) does not calculate uncertainty ranges. NMFS officials told us that NMFS is updating the model so that it can compute uncertainty ranges; NMFS expects to use the updated model for all 2005 stock assessments. The Pacific hake assessor used a mathematical model (AD Model Builder) that could compute uncertainty ranges and included these ranges in the Pacific hake assessment report.

Some Recommended Stock Improvements Have Been Implemented, but Much Remains to be Done

NMFS has taken some steps recommended in the Marine Fisheries Stock Assessment Improvement Plan to improve the quantity, quality, and type of data used in Pacific groundfish stock assessments, but much remains to be done to make the assessments more reliable. The Northwest Center has concentrated most of its efforts on implementing recommendations aimed at obtaining more data. Recommendations aimed at increasing the types of data and improving their quality have not yet been fully implemented for a variety of reasons, such as staffing and funding limitations. In addition, other program priorities have precluded NMFS from implementing the recommendation to create a comprehensive plan that incorporates the improvement plan and related plans so that it can develop integrated program initiatives to improve stock assessments.

The Northwest Center Has Taken Some Steps Recommended for Improving Stock Assessments

The October 2001 stock assessment improvement plan identified three scenarios (tiers) to consider when analyzing the resources needed to improve stock assessments. The three tiers of assessment improvements are as follows: Tier 1—improve stock assessments using existing data without initiating new data collection programs.

Tier 2—conduct baseline monitoring of species, which in most cases requires sampling the species at least every 1 to 3 years, and preferably at least every 1 to 2 years.

Tier 3—implement “next generation” stock assessments by explicitly incorporating ecosystem considerations, such as multispecies interactions and environmental effects in assessments.

The improvement plan also made a number of recommendations to improve stock assessments. The recommendations fall into the following four categories: Data collection—pursue new initiatives to expand data collection efforts that at a minimum bring stock assessment science to Tier 2. In addition, continue to develop partnerships and cooperative research programs with other entities, such as state agencies, commercial and recreational fishing organizations, and individuals to improve the quantity, quality, and type of data collected.

Communication—educate constituents about NMFS’ strategies for improving stock assessments.

Training—implement comprehensive training and staff development programs for NMFS’ analytical and quantitative staff, and augment existing programs that support graduate students interested in stock assessment science.

Planning—develop integrated program initiatives by preparing a comprehensive plan that combines the improvement plan with its complementary plans.

Improvement in the quantity of data collected for use in stocks assessments is a key component to achieving Tier 2 status. The improvement plan states that as the quantity of the data increases, the assessments become more reliable because the data cover a longer period of time, producing better population trend information. Northwest Center officials said that the quality of the data improves with more frequent surveys and more randomly selected survey locations that, over the long term, provide a better understanding of the variability inherent in the population distribution and abundance. A better sense of trends and variability allow for improved short-term predictions of the status of the species.

The Northwest Center has concentrated most of its efforts on implementing improvements in data quantity, such as more frequent acoustic, and shelf and slope bottom trawl surveys. The following illustrate some of the actions the Northwest Center took in 2003 to improve data quantity: Increased the frequency of the Pacific hake acoustic survey from triennially to biennially. Beginning in 2003, the survey was restructured into a single, integrated survey with Northwest Center and Canadian officials jointly planning all survey elements. Officials from the Northwest Center and Canada now jointly conduct all of the acoustic surveys.

Increased the frequency of the groundfish shelf and slope bottom trawl survey from triennially to annually, leveraging available resources by cooperatively working with the fishing industry. Specifically, contracting with private commercial fishing vessels to conduct the surveys. According to Northwest Center officials, working collaboratively with the fishing industry has afforded fishermen the opportunity to become stakeholders in the data collection process.

Extended the geographic range of the groundfish shelf and slope bottom trawl survey. The surveys are now coastwide from Cape Flattery, Washington to the Mexican border, adding over 300 more miles along the southern California coast. Previous surveys ended at Morro Bay, California.

Efforts continue to communicate the strategies needed to improve stock assessments and to augment existing programs aimed at developing future stock assessment scientists. For example, through the Pacific Fishery Management Council, Northwest Center staff meet with their constituents, such as representatives from state agencies and the fishing industry, to discuss groundfish management issues. In addition, the Northwest Center organized a series of public meetings to discuss new initiatives that affect the stock assessment program. For example, the Northwest Center held public meetings in several communities along the Pacific coast to discuss implementation of the Observer Program—a program designed to collect information about discarded fish for the non-hake west coast groundfish fleets.

Finally, the Northwest Center now participates in NMFS’ National Sea Grant program to augment a Northwest Center-supported graduate study program at the University of Washington that trains stock assessment scientists. The Sea Grant program provides fellowships for students interested in marine research, such as stock assessment methodology and marine resource economics. The Northwest Center plans to employ two Sea Grant students during the summer of 2004.

Further Actions Are Needed to Improve Stock Assessments

While the Northwest Center has implemented some recommendations aimed at improving stock assessments, it has not yet fully implemented many others. These recommendations include collecting additional types of data, such as ecosystem and recruitment information; improving data quality, such as calibrating survey vessel equipment; and increasing training opportunities for Northwest Center staff. Also, NMFS has not acted on the task force recommendation to combine the improvement plan and its complimentary plans into a comprehensive plan that provides integrated program initiatives.

According to the improvement plan, additional types of data will allow NMFS to further test and validate model assumptions, thereby increasing the reliability of the stock assessments. The improvement plan further states that information derived from ecosystem research and recruitment surveys is essential if assessments are to meet the national standards of “next generation” assessments or Tier 3 status. According to Northwest Center officials, ecosystem information and coastwide recruitment surveys are two of the most critical data sets needed to ensure continuous improvement of groundfish stock assessments. The Northwest Center conducts ecosystem research as part of its Science for Ecosystem-based Management Initiative. Understanding the complex ecological relationships between fish and the environment within which they exist provides a better understanding of the effects of the ecosystem on the groundfish fisheries and the scientific knowledge needed to make informed ecosystem-based management decisions. Although research is ongoing to develop ecosystem information, only a limited amount of the data is collected and used in stock assessments. For example, ecosystem data are collected during shelf and slope bottom trawl surveys as time and resources allow. However, this information is not widely incorporated into stock assessments. For the five species we reviewed, only the boccacio assessment used ecosystem data—information on the temperature of the ocean’s surface.

According to Northwest Center officials, the collection of ecosystem data is limited because the relatively small size of the commercial vessels used in the shelf and slope bottom trawl surveys cannot support the number of researchers needed to effectively conduct comprehensive ecosystem research and collection activities. Furthermore, the implementation of comprehensive ecosystem research and data collection programs is contingent upon the funding of a dedicated research vessel for west coast surveys. Northwest Center officials said they are to receive a dedicated research vessel sometime during calendar years 2008 or 2009, at the earliest.

Better recruitment information for Pacific groundfish is also needed because such information provides an early predictor of fish abundance, especially for species such as hake, where there is a great variation in recruitment. Northwest Center officials said that current recruitment surveys are limited because existing funds support only yearly surveys in selected areas. To achieve the best early predictions of stock status, these officials said, recruitment surveys should be coastwide and conducted twice a year. According to Northwest Center officials, 13 full-time staff are needed to expand these and other high-priority data collection efforts, such as surveys in untrawlable habitat and expanded acoustic surveys.

The lack of quality data was identified in the improvement plan as an impediment to producing reliable stock assessments. For example, when equipment on different survey vessels are not calibrated, the data are not comparable, and trends may not be accurately determined. The Northwest Center is continuing its efforts to calibrate survey vessel equipment.

The improvement plan also recommended that NMFS provide additional training to ensure that qualified NMFS staff are available now and in the future to conduct stock assessments and related activities. For example, the plan recommended the development of a comprehensive training program and more professional developmental opportunities for NMFS’ scientific staff. Northwest Center officials said they try to meet the training and professional development needs of their scientific staff. However, to date they have focused on developing external training programs, such as the University of Washington graduate program, to develop stock assessment scientists for the future and have yet to develop a comprehensive training program for in-house stock assessment scientists.

Finally, the improvement plan recommended that NMFS prepare a comprehensive plan that combines the improvement plan with other complementary plans, such as the NOAA Fisheries Data Acquisition Plan and the NMFS Social Sciences Plan. A comprehensive plan would allow NMFS to better integrate and coordinate program initiatives for improving stock assessments. For example, the acquisition plan—the key complementary plan to the improvement plan—identifies the need for fishery research vessels to satisfy NMFS’ data collection needs. Although the improvement plan includes the number of staff that would participate in data collection surveys, it does not contain the capital and operating costs of the research vessels. Similarly, the staffing requirements for augmenting the social sciences capabilities of NMFS to conduct economic analyses is represented in the sciences plan and not in the stock improvement plan. A NMFS official said that other program priorities, such as conducting more stock assessments and improving data collection activities, have precluded them from developing a comprehensive plan.

Remaining Improvements Estimated to Cost at Least $8.9 Million

According to NMFS funding and budget requests, the Northwest Center needs at least $8.9 million to complete ongoing and planned improvements to the stock assessments for Pacific groundfish. However, the actual cost of implementing remaining improvements to Pacific groundfish stock assessments may be even higher because the Northwest Center’s budget requests primarily reflect the amount of money the Center believed it could realistically obtain, rather than the actual cost of the improvements.

Remaining Improvements Are Estimated to Cost at Least $8.9 Million, but Estimate Is Likely Understated

According to NMFS, the Northwest Center needs at least $8.9 million to complete ongoing and planned improvements for Pacific groundfish stock assessments: $2.6 million that NMFS’ Northwest Center requested but did not receive between fiscal years 2001 to 2003 and $6.3 million the Center requested for fiscal years 2004 and 2005. Specifically, as shown in table 2, the Northwest Center records have identified the following funding needs $7.7 million to improve the types of data used, including $2.4 million for surveys of untrawlable waters, $2.1 million to expand acoustic and recruitment surveys, and $3.2 million to collect ecosystem data; and $1.2 million to improve the quality of data used in stock assessments, including $600,000 to enhance the calibration of vessel equipment; $525,000 to develop and implement methods to collect information on stock identification, structure, and movement; and $75,000 to standardize trawl survey procedures.

The Northwest Center did not receive its full funding request, in part, because NMFS did not receive all the funding it had requested. Between fiscal years 2001 and 2003, NMFS received $20.6 million (80 percent of its request) in additional funding to implement improvements for all marine stock assessments. NMFS allocated $3.6 million (58 percent of funds the Northwest Center requested) to the Northwest Center for improving Pacific groundfish stock assessments, resulting in a $2.6 million shortfall in the Center’s request. This shortfall occurred in part because of NMFS’ need to balance the requests of its six science centers against its program priorities and the available funds. According to NMFS officials, their goal is to achieve parity among the science centers in terms of their capability to conduct scientific work, such as stock assessments.

Estimated Costs May Understate Actual Cost of Remaining Improvements

The $8.9 million needed to implement remaining recommended improvements is probably understated because the Northwest Center’s budget requests primarily reflect the amount of money the Center officials believed they could realistically obtain, rather than the amount the improvements would actually cost, according to NMFS officials. The Northwest Center’s budget requests for fiscal years 2004 and 2005 are preliminary requests submitted before the Northwest Center received its fiscal year 2003 funding. Consequently, the Northwest Center will likely submit revised budget requests for fiscal years 2004 and 2005 that account for both its unfunded needs from fiscal years 2001 through 2003 and items that were unexpectedly funded in fiscal year 2003. Moreover, the fiscal year 2004 and 2005 preliminary budget requests do not incorporate any unanticipated problems or data gaps that have developed since the Northwest Center submitted its preliminary requests.

According to NMFS officials, NMFS’ science centers, including the Northwest Center, primarily make and justify their funding requests in response to how much money Congress appropriates. After Congress passes NMFS’ budget, NMFS asks its science centers to reassess and detail how much new money each needs to implement science center programs, such as marine stock assessment improvements. According to NMFS officials, it is unrealistic for a science center to request more funds than are available in its appropriation, even if it needs more. While NMFS’ Northwest Center requested $6.2 million to implement improvements to Pacific groundfish stock assessments between fiscal years 2001 and 2003, NMFS’ 2001 West Coast Groundfish Research Plan estimated that almost twice as much money would be needed—approximately $11.7 million in new funding—to implement top-priority improvements to Pacific groundfish stock assessments.

NMFS is now updating its plan and cost estimates for improving Pacific groundfish stock assessments. Using key findings from its December 2003 review of the groundfish program, the Northwest Center plans to update its groundfish research plan, last published in 2001. According to NMFS, the updated groundfish research plan should be completed in late 2004 and is designed to (1) provide a comprehensive framework for Pacific groundfish, (2) identify some of the greatest information gaps, and (3) provide guidance for setting priorities on work to fill these gaps. In addition, the updated plan will estimate how much such improvements will cost.

Conclusions

Stock assessments are the key to effectively managing fisheries. They provide estimates of the species population, which NMFS uses to set harvest limits that allow for sustainability and/or recovery of the species. While stock assessment results often change from assessment report to assessment report, the more types of information used in the assessments, such as recruitment surveys and ecosystem studies, and the greater the accuracy and quality of the data, such as scientifically designed and collected data, the more reliable the assessment results. However, the Pacific groundfish assessments we reviewed did not (1) use scientifically designed and collected NMFS data of sufficient scope and accuracy, such as survey data on the abundance of groundfish residing in rocky, untrawlable habitats; (2) subject the non-NMFS data used to a standard process for assessing its reliability; and/or (3) identify the uncertainties of the assessments total biomass estimates. As a result, the reliability of the five assessments is questionable. Without reliable assessments, fishery managers may reach erroneous conclusions and take actions that could adversely affect the fishing industry economically or adversely affect the recovery and sustainability of the fishery resources. Moreover, without a comprehensive, integrated improvement plan, funding requests and planned actions to improve the stock assessments may not be coordinated, jeopardizing successful and timely implementation of assessment improvements.

Recommendations for Executive Action

To improve the reliability of Pacific groundfish stock assessments, we recommend that the Secretary of Commerce require the Director of National Marine Fisheries Service to take the following four actions: Continue efforts to collect more types of data, such as data obtained from surveys in rocky, untrawlable waters, recruitment surveys, and ecosystem studies, for groundfish assessments where reliable data are now lacking.

Establish a standard approach that requires that non-NMFS data used in stock assessments be evaluated for its reliability, and continue efforts to implement the task force’s recommendations to improve data quality.

Require that stock assessment reports clearly present the uncertainties in the assessments, such as the margin of error associated with species biomass estimates.

Develop a comprehensive plan that integrates the NMFS stock assessment improvement plan with other NMFS plans to ensure that stock assessment improvement actions and budget requests are coordinated.

Agency Comments and Our Evaluation

We provided the Department of Commerce with a draft of this report for review and comment. We received a written response from the Under Secretary of Commerce for Oceans and Atmosphere that included comments from the National Oceanic and Atmospheric Administration (NOAA). NOAA generally agreed with the report’s accuracy and concurred with the reports recommendations. However, NOAA said it was concerned about the report’s conclusion—that the reliability of stock assessments is questionable for the five species reviewed—because it could be misconstrued to infer that the assessments are unreliable for use in managing the west coast groundfish fishery. In this regard, NOAA provided additional comments to show the usefulness of the assessments, even if some of the input data used in the assessments contained errors. We stand by our conclusions that the five stock assessments we reviewed were questionable because the input data were insufficient and/or potentially inaccurate and that four of the assessment reports did not present the uncertainties associated with the biomass estimates. Nonetheless, we added language to the report to address NOAA’s concern. Specifically, we expanded upon the fact that NMFS used the best information available at the time the stock assessments were conducted by adding information on the importance of the assessments to effectively manage the fisheries. Without these stock assessments, NMFS and fishery managers would have very limited information on which to base fishery management decisions.

NOAA agreed with the report recommendation to continue collecting more types of data for groundfish assessments where reliable data are now lacking. NOAA said that the reliability of stock assessments will be improved if NMFS survey efforts are expanded and additional NMFS fishery data are collected. NOAA said NMFS places a priority on these improvements and will continue efforts to address this and other recommendations to improve the collection of fishery data as funding becomes available.

NOAA also agreed with the report recommendation to establish a standard approach to evaluate the reliability of non-NMFS data used in stock assessments and continue efforts to improve data quality. NMFS said, through its west coast fishery science centers, it participates on interagency data committees, to develop quality assurance protocols and to assess the quality of non-NMFS data. NOAA agreed that it is important to ensure that these interagency data committees continue to highlight the need for standardized quality control procedures for the collection of data.

NOAA agreed with the report recommendation to clearly present the uncertainties in the stock assessments. NOAA said that quantifying uncertainty of stock assessments is important to sound decision-making by providing more information about the assessment, although this quantification does not reduce the uncertainty in the assessment itself. While the methods used and the completeness of the uncertainty characterization varied from assessment to assessment, NOAA said it is desirable to have both a quantitative analysis of model uncertainty and an evaluation of the consequences of alternative model scenario.

Finally, NOAA agreed with the report recommendation to develop a comprehensive plan that integrates the stock assessment plan with other NMFS plans to ensure that improvement actions and budget initiatives are coordinated. NOAA said that while much remains to be done, long-term planning efforts and coordination among field and headquarters are ongoing, and NOAA is committed to these actions.

NOAA’s comments and our detailed responses are presented in appendix II of this report. NOAA also provided technical comments that we incorporated in this report as appropriate.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 7 days from the report date. At that time, we will send copies of this report to the Secretary of Commerce and the Director of the National Marine Fisheries Service. We will also provide copies to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please call me at (202) 512-3841 or Keith Oleson at (415) 904-2218. Key contributions to this report are listed in appendix III.

Objectives, Scope, and Methodology

We reviewed National Marine Fisheries Service (NMFS) stock assessments for five species of Pacific groundfish: Pacific hake (Pacific whiting) as well as four types of rockfish—bocaccio, canary, darkblotched, and yelloweye. Specifically, for these five species you asked us to (1) assess the reliability of NMFS’ stock assessments, (2) identify which relevant recommendations from the stock assessment improvement plan have been implemented and which have not, and (3) identify the estimated costs associated with planned and ongoing improvements to groundfish stock assessments. We did not review the stock assessments of any of the other west coast Pacific groundfish species, thus the information contained in this report pertains to the five species we reviewed unless stated otherwise.

For all three objectives, we reviewed key laws and agency reports and interviewed officials from NMFS, including officials from the Northwest Fisheries Science Center, which has lead responsibility for conducting Pacific groundfish stock assessments with assistance from other west coast science centers.

To assess the reliability of the Pacific hake, bocaccio, canary, darkblotched, and yelloweye stock assessments, we examined methodological and administrative documents developed by NMFS and others to support the groundfish data collection, maintenance, and assessment process. We reviewed the controls over stock assessment data, the types of fish population surveys used, and recent Pacific groundfish stock assessment studies (2002 and 2004 studies for Pacific hake, 2002 and 2003 studies for bocaccio, 2002 study for canary, 2000 and 2003 studies for darkblotched, and 2001 and 2002 studies for yelloweye). We examined whether and to what extent NMFS has processes and procedures in place to ensure the reliability of data used in the Pacific groundfish stock assessments. We reviewed the stock assessment reports and determined whether they articulated the level of uncertainty in the assessment model estimates. We interviewed an array of government officials and fisheries experts, including the Oregon Department of Fish and Wildlife, the Washington Department of Fish and Wildlife, the California Department of Fish and Game, the Pacific Fishery Management Council, the Pacific States Marine Fisheries Commission, environmental groups, and industry associations, as well as fishermen and academics. We did not simulate NMFS’ stock assessment models nor evaluate the mathematical and statistical methodologies used in the models for Pacific hake, bocaccio, canary, darkblotched, and yelloweye.

To identify the relevant recommendations to improve stock assessments that NMFS has implemented and has not implemented, we reviewed agency reports on marine fisheries stock assessments, strategic planning, and data collection. We also interviewed officials from the Oregon Department of Fish and Wildlife, the Washington Department of Fish and Wildlife, the California Department of Fish and Game, the Pacific Fishery Management Council, and the Pacific States Marine Fisheries Commission as well as environmental groups, industry associations, fishermen, and academics.

To determine the estimated costs associated with NMFS’ planned and ongoing improvements to Pacific groundfish stock assessments, we reviewed relevant budget requests and funding documents for fiscal years 2001 through 2005 and interviewed National Oceanic and Atmospheric Administration officials. We did not evaluate the accuracy of NMFS’ budget requests for specific project items but rather used the amounts NMFS requested for these project items to estimate the total additional costs of implementing the planned and ongoing improvements to Pacific groundfish stock assessments.

We conducted our review from May 2003 through April 2004 in accordance with generally accepted government auditing standards.

Comments From the Department of Commerce

The following are GAO comments on NOAA’s letter dated May 13, 2004.

GAO Comments

1. We added clarifying language to the scope and methodology section of the report to clearly identify the species and activities covered by the review. 2. We revised the report to show the publication date. 3. We revised the report accordingly. 4. We revised the report to clearly show that NMFS has not collected enough ecosystem data and that the frequency and range of recruitment surveys are limited. The statement does not address untrawlable habitat. 5. We revised the report to clarify that NMFS “generally uses” NMFS’s staff or contracts with outside experts. 6. We revised the report to more clearly differentiate between NMFS as a whole and NMFS’ Northwest Center in particular. We made similar revisions, as appropriate, throughout the report. 7. We revised the report to specify “bottom” trawl survey. We made similar changes, as appropriate, throughout the report. 8. We revised the report to include the year and scope of the task force review. 9. We revised the report to indicate that the Northwest Center is responsible for coordinating groundfish stock assessments. 10. We revised the report to include the citation. 11. We revised the report to include the date and citation of the National Research Council report. 12. The NMFS data used in the bocaccio, canary, and darkblotched assessments were limited because NMFS conducted its surveys in trawlable waters only. NMFS data were not available for untrawlable waters, which these species also inhabit. For this reason, we did not revise the report. 13. We revised the report to clarify the shared responsibilities of the Northwest and Southwest Centers. 14. We revised the report to include NOAA’s recommended definition of stock abundance. 15. We revised the report to include larval fish. 16. We revised the report to clarify the role of the review panel. 17. We revised the report to more explicitly distinguish the five species related to our report from other overfished Pacific groundfish. 18. We revised the report to more clearly describe the distribution of Pacific hake. 19. Bocaccio survey data for untrawlable habitats, as stated in comment 12, was unavailable. For this reason, we did not change the report. 20. NOAA commented that highly standardized protocols are used for collecting non-NMFS data (fishery dependent data) for rockfish. We found that although NMFS does have collection and quality assurance procedures for state-collected non-NMFS data, NMFS does not check or have a standard process to verify that these data have been reviewed for reliability. As discussed in our report, some assessors chose to review the raw data, while others did not. Assessors who voluntarily reviewed raw non-NMFS data found mistakes that either made some of the data unusable or could have impaired the accuracy of the stock assessments. For these reasons, we did not change the report. 21. We revised the punctuation accordingly. 22. We revised the report to clarify that Pacific hake live in mid-water habitat. 23. The footnote placement and citation are in accordance with GAO guidelines. For this reason, we did not change the report. 24. We believe that our report has addressed this issue. By referring to the West Coast Groundfish Research Plan by its complete title, we adequately distinguish between the two reports. For this reason, we did not change the report. 25. We changed “health” to “stock abundance.” 26. The footnote placement is repositioned in report. 27. We revised the report to include assessment “improvements.” 28. As indicated in our report, we illustrate some of the actions that the Northwest Center took to improve data quantity and did not intend to provide a comprehensive list of all actions conducted to improve data quantity coast wide. However, we added footnote 16 to clarify the actions taken by the Southwest Center. 29. The example we provided is not intended to be a comprehensive list of all ecosystem research conducted on the west coast. Instead, it illustrates the type of work the Northwest Center is conducting and the opportunities for improving ecosystem research. For this reason, we did not change the report. 30. After reviewing the report we believe no change is required because of subject-verb agreement. 31. We added clarifying language. 32. We believe that table 2 notes “a” and “b” in our report already adequately address this issue. Annotations for projects that do not separate out groundfish funds occur only in items that are annotated as Southwest Center projects. For this reason, we did not change the report. 33. We changed “survivability” to “sustainability.” 34. We changed “survivability” to “sustainability” and added fishery “resources” for clarification. 35. NOAA commented that GAO does not adequately convey the different degrees of precision associated with the stock assessments and GAO’s conclusion that the reliability of the five assessments we reviewed is questionable and could easily be misconstrued to mean all these assessments are an unreliable basis for management of the west coast groundfish fishery. NOAA also commented that the five assessments GAO reviewed all passed scientific review and are serving as the basis for formal status determination and fishery management. Our report acknowledges that stock assessments are scientifically reviewed and are a key tool for managing fisheries. However, we found the reliability of the five assessments questionable for the three reasons we highlighted in our report, and we recommended actions on how to improve the reliability of the stock assessments. We added clarification to the report to show that stock assessments are a key tool for managing fisheries and are important in making decisions about setting harvest levels and developing plans to rebuild overfished stocks.

NOAA also commented that quality assurance for non-NMFS data is not absent. As stated in our response number 20, we found that although NMFS does have collection and quality assurance procedures for state- collected non-NMFS data, NMFS does not check or have a standard process to verify that these data have been reviewed for reliability. As discussed in our report, some assessors chose to review the raw data, while others did not. Assessors who voluntarily reviewed raw non- NMFS data found mistakes that either made some of the data unusable or could have impaired the accuracy of the stock assessments. For these reason, we did not change the report. 36. NOAA commented that it is more pertinent to focus on the degree of standardization of the survey data than the source. By categorizing data as NMFS data and non-NMFS data, we were not implying that non- NMFS organizations could not conduct useful fishery-independent surveys. We categorized the data in this manner because NMFS currently conducts nearly all of the fishery-independent surveys and non-NMFS organizations collect most of the fishery-dependent data. Footnote 7 in the report states that NMFS generally refers to its data as fishery-independent data and to non-NMFS data as fishery-dependent data. For these reasons, we did not change the report. 37. We believe the Pacific hake biomass estimates are questionable because the assessment used non-NMFS data that NMFS did not check or subject to standard data reliability testing. Assessors who reviewed raw non-NMFS data for other stock assessments found mistakes that either made some of the data unusable or could have impaired the accuracy of the stock assessments. For this reason, we did not change the report. 38. NOAA commented that bocaccio, canary, and darkblotched assessments all obtain adequate abundance trend information from the NMFS bottom trawl surveys. NOAA also commented that although bottom trawl survey cannot access the roughest habitat, it is useful as an index of relative changes in the overall abundance. As stated in our report, we found that the NMFS survey data used in these assessments were limited in scope because the surveys were conducted only in trawlable areas. Assessors estimated overall biomass using the NMFS data collected from the trawlable area, which has a different abundance rate than the untrawlable area. Stock assessors commented that relying on survey data from trawlable waters only increases the uncertainty of stock assessments. For these reasons, we did not change the report. 39. As noted in our report, the National Research Council found that the inclusion of NMFS survey data was the best option for a reliable estimate of abundance because such surveys use an unbiased statistical design, control sampling locations, and provide for quality assurance. Northwest Center officials said that to obtain reliable results, each stock assessment should include at least one source of NMFS-collected data of sufficient scope and accuracy because such surveys are unbiased and scientifically designed. NMFS data were unavailable for the yelloweye assessment. Northwest Center officials also raised concerns about basing assessments solely on non-NMFS data such as commercial and recreational catch data. Catch data do not provide the species’ relative or absolute biomass, according to NMFS officials. Catch data alone are insufficient because fishermen are not randomly sampling the ocean, but are fishing areas that they are allowed to fish and they believe have the most fish; fishing restrictions, such as a total allowable catch, can limit the amount of fish being caught; and catch data have often been inaccurate for a variety of reasons, such as imprecise accounting for dead fish tossed back into the ocean. For these reasons, we did not change the report. 40. NOAA commented that the doubling of estimated bocaccio biomass in 2003 was due to factors that would not be addressed in a standard statistical analysis. Although a standard statistical analysis may not fully address the doubling of an estimate, an assessment without an uncertainty range does not quantify and communicate any of the uncertainty. For this reason, we did not change the report.

GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the person named above, Leo G. Acosta, Kristine N. Braaten, Allen T. Chan, David Dornisch, Alan Kasdan, Robert Marek, Cynthia C. Norris, Carol Herrnstadt Shulman, and Tama R. Weinberg made key contributions to this report.

GAO’s Mission

The General Accounting Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to e-mail alerts” under the “Order GAO Products” heading.

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Public Affairs