Background

The Magnuson-Stevens Fisheries Conservation and Management Act  granted responsibility for managing marine resources to the Secretary of  Commerce. The Secretary delegated this responsibility to NMFS, which is  part of Commerce’s National Oceanic and Atmospheric Administration  (NOAA). The act also established eight regional fishery management  councils, each responsible for making recommendations to the Secretary  of Commerce about managing fisheries in federal waters. The eight fishery  management councils—consisting of fishing industry participants, state  and federal fishery managers, and other interested parties—and their areas  of responsibility include the following:   Caribbean Council, covering waters off the U.S. Virgin Islands and the  Commonwealth of Puerto Rico;   Gulf of Mexico Council, covering waters off Texas, Louisiana,  Mississippi, Alabama, and the west coast of Florida;   Mid-Atlantic Council, covering waters off New York, New Jersey,  Delaware, Maryland, Virginia, and North Carolina;   New England Council, covering waters off Maine, New Hampshire,  Massachusetts, Rhode Island, and Connecticut;   North Pacific Council, covering waters off Alaska;   Pacific Council, covering waters off California, Oregon, and Washington;  South Atlantic Council, covering waters off North Carolina, South  Carolina, Georgia, and the east coast of Florida; and  Western Pacific Council, covering waters off Hawaii, American Samoa,  Guam, the Commonwealth of the Northern Mariana Islands, and  uninhabited U.S. territories in the Western Pacific.
In addition to these eight councils, NMFS has six regional science centers,  which are responsible for generating the scientific information necessary  for the conservation, management, and use of each region's marine  resources. The six fishery science centers and their areas of responsibility  are as follows:   Alaska Center, covering the coastal oceans off Alaska and parts of the  west coast of the United States;   Northeast Center, covering waters along the Northeast Continental Shelf  from the Gulf of Maine to Cape Hatteras, North Carolina;  Northwest Center, covering the northeast Pacific Ocean, primarily  waters off the coasts of California, Oregon, Washington, and British  Columbia;   Pacific Islands Center, covering the central and western Pacific Ocean;   Southwest Center, primarily covering waters off the coast of California  and areas throughout the Pacific and Antarctic Oceans; and  Southeast Center, covering waters along the continental southeastern  United States as well as Puerto Rico and the U.S. Virgin Islands.
The Magnuson-Stevens Act, as amended by the Sustainable Fisheries Act,  also established national standards for fishery conservation and  management. These standards deal with preventing overfishing, using  scientific information, using fishery resources efficiently, minimizing  bycatch, and minimizing administrative costs. The regional councils use  these standards to develop appropriate plans for conserving and managing  fisheries under their jurisdiction, including measures to prevent overfishing  and to rebuild overfished stocks as well as measures to protect, restore,  and promote the long-term health and stability of the fishery.
In 1982, the Pacific Fishery Management Council (Pacific Council) released  its initial Fishery Management Plan for groundfish. The Pacific Council’s  goal is to have long-range plans for managing groundfish fisheries that will  promote a stable planning environment for the seafood industry, including  marine recreation interests, and will maintain the health of the resource  and environment. To help achieve these goals, stock assessments are  conducted on groundfish species to estimate fish populations. Since 1995,  the Northwest Center has had lead responsibility for conducting stock  assessments on Pacific groundfish. The Northwest Center receives  assistance from other NMFS science centers, such as the Southwest  Center, which conducted the bocaccio assessment.
Stock assessments are the biological evaluation of the status of fish stocks.  Stock assessments provide official estimates in key areas, such as the size  of the stock population, the size of the spawning population, the amount of  fish that have died (fish mortality), and the estimated number of fish at a  particular young age (recruitment). Stock assessments form the scientific  basis used by regional councils to determine biologically sustainable  harvests and guide the monitoring and rebuilding of overfished and  threatened stocks. For example, regional councils use stock assessments  and other indicators of biological productivity to recommend to NMFS a  maximum, or total allowable catch, in a particular fishery—typically for a  year. Stock assessments are a key tool for managing fisheries. Without  stock assessments, fishery managers would have limited information about  the status of fisheries in making decisions about setting harvest levels and  developing plans to rebuild overfished stocks.
For each species, the assessor reviews previous stock assessments, gathers  available data about the species being assessed, runs the data through  computer-generated models, and estimates the species’ total biomass.  Stock assessors use NMFS-collected data, such as stock surveys conducted  on NOAA vessels or contracted commercial fishing vessels, as well as data  collected by non-NMFS sources, such as commercial and recreational  catch data collected by state agencies. The following six key types of data  are collected:   Stock abundance—surveys of how many fish constitute a stock’s total  size or weight.
Commercial and recreational fisheries data—the amount and  composition of fish caught from a particular stock, whether caught  intentionally by commercial and recreational fishermen or  unintentionally caught and discarded. Data sources include fishing  logbooks, dockside samples, and onboard observations, among others.
Life history—biological data, such as the age and sex composition of  the stock, age at first maturity, fertility, average lifespan, and natural  mortality.
Ecosystem relationships—data on the relationship between a fish stock  and its physical environment, as well the relationship of a fish stock to  other species.
Recruitment research—data on the abundance of juvenile and larval  fish (fish at their earliest stage), which helps scientists forecast the size  of a particular stock in the future.
Synoptic oceanographic sampling—data on the ocean ecosystem, such  as water temperature or salinity, plankton composition, or ocean  currents.
For each stock assessment, a review panel, consisting of NMFS scientists  and outside experts, independently reviews the methodology of the  assessment and works with the assessor to ensure their comments are  adequately addressed.
Through 2003, 24 of the 82 species of Pacific groundfish have had a full  quantitative stock assessment. Relying on these assessments, NMFS has  declared that nine species of Pacific groundfish are overfished, including  the five species of Pacific groundfish we reviewed in this report—Pacific  hake as well as bocaccio rockfish, canary rockfish, darkblotched rockfish,  and yelloweye rockfish. Rockfish are long-lived, late-maturing and  slow-growing species, making them particularly susceptible to overfishing.  More specifically: Pacific hake, also called Pacific whiting, is generally found off the west  coast of North America. It is one of many species of hake distributed in the  Atlantic and Pacific oceans. Fishing of Pacific hake primarily takes place  off the coasts of northern California, Oregon, Washington, and British  Columbia. Fishermen use mid-water trawls and generally fish over the  ocean bottom at depths of 100 to 500 meters. Pacific hake was declared  overfished in 2002 because the 2002 stock assessment estimated Pacific  hake biomass at 700,000 metric tons. By 2004, the biomass was estimated at  between 2.7 and 4.2 million metric tons, and Pacific hake is no longer  considered overfished. Figure 1 shows a picture of a Pacific hake.
Bocaccio rockfish generally inhabit waters off the coast of northern Baja,  Mexico to Alaska. Bocaccio, commonly sold at market as “red snapper,” are  commercially fished using trawls, hook-and-line and gillnets. Adult  bocaccio commonly live over rocky areas or open areas of the ocean’s floor  to about 320 meters. Bocaccio were formally declared to be “overfished” in  1999. The 2003 stock assessment estimated the bocaccio biomass at 7,133  metric tons. Figure 2 shows a picture of a bocaccio rockfish.
Canary rockfish inhabit the northeastern Pacific Ocean, from northern  Baja, Mexico to the western Gulf of Alaska. Adult canary rockfish are  primarily found along the continental shelf, from 46 to 457 meters deep.  Canary rockfish are harvested commercially using trawl nets and  hook-and-line and are also considered an important species for  recreational fishermen. NMFS declared canary rockfish as overfished in  2000. The 2002 stock assessment estimated the canary rockfish biomass at  6,197 metric tons. Figure 3 shows a picture of a canary rockfish.
Darkblotched rockfish are found in the waters from Santa Catalina Island,  California to the Bering Sea on soft bottom areas at about 29 to 549 meters  deep. Commercial fishery concentrations are located off the coasts of  California and Oregon. Darkblotched rockfish are caught primarily by  commercial trawls and contribute to both commercial and recreational  fishing. NMFS determined that darkblotched rockfish was overfished in  2000, when the last full stock assessment was conducted; it was updated in  2003. This update estimated the darkblotched rockfish biomass at 7,266  metric tons in 2001. Figure 4 shows a picture of a darkblotched rockfish.
Yelloweye rockfish almost exclusively inhabit rocky areas from northern  Baja, Mexico to the Aleutian Islands, Alaska. Yelloweye rockfish, found in  depths ranging from 15 to 550 meters, are caught using both trawl nets and  line gear and are highly prized by both commercial and recreational  fishermen. Stock assessments for yelloweye rockfish were first conducted  in 2001, and NMFS determined that the species was overfished in 2002. The  2002 stock assessment estimated the yelloweye biomass at 2,325 metric  tons in 2001. Figure 5 shows a picture of a yelloweye rockfish.

	Reliability of NMFS’ Pacific Groundfish Stock Assessments Is Questionable

The reliability of NMFS’ stock assessments is questionable for the Pacific  hake and four rockfish species we reviewed, although they were based on  the best information available at the time the assessments were conducted.  The reliability of the stock assessments we reviewed is questionable  because (1) four of the assessments did not have at least one  NMFS-collected data source of sufficient scope and accuracy; (2) NMFS  lacked a standard process for assessing the reliability of non-NMFS data  used in all five assessments; and (3) for four of the assessments, the stock  assessment reports did not adequately identify the uncertainty of the  biomass estimates. (See table 1.) To address these limitations, the  Northwest Center plans to increase the scope and accuracy of its collected  data, as additional funds become available; is implementing changes that  will help ensure the reliability of non-NMFS data; and plans to update the  stock assessment model to provide uncertainty ranges for the 2005 stock  assessments.

		Absence of NMFS Survey Data Adversely Affects the Assessments’ Reliability

Stock assessors use a variety of data, including NMFS data and non-NMFS  data, in developing their assessments. Two key pieces of NMFS survey  data are the shelf and slope bottom trawl survey and the acoustic survey.  Other NMFS data that assessors sometimes use include larval surveys (data  for fish in their earliest stage) and recruitment surveys. The non-NMFS data  assessors use include commercial catch data and recreational catch data.
A 2002 National Research Council report found that the inclusion of NMFS  survey data was the best option for a reliable estimate of abundance  because such surveys use an unbiased statistical design, control sampling  locations, and provide for quality assurance. According to Northwest  Center officials, to obtain reliable results, each stock assessment should  include at least one source of NMFS-collected data of sufficient scope and  accuracy because such surveys are unbiased and scientifically designed.
Northwest Center officials raised concerns about basing assessments  solely on non-NMFS data such as commercial and recreational catch data.  Catch data do not provide the species’ relative or absolute biomass,  according to NMFS officials. For example, catch data alone is insufficient  because   fishermen are not randomly sampling the ocean but are fishing areas  that they are allowed to fish and they believe have the most fish;   fishing restrictions, such as a total allowable catch, can limit the  amount of fish being caught; and   catch data have often been inaccurate for a variety of reasons, such  as imprecise accounting for dead fish tossed back into the ocean.
Although the assessors used several different data sources, four of the five  assessments did not use NMFS survey data or the NMFS data used covered  only a portion of the species’ habitats. In the yelloweye assessment, no  NMFS survey data were available because yelloweye live almost  exclusively in the rocky habitat that NMFS trawl surveys cannot cover. As a  result, the yelloweye assessment was based solely on non-NMFS data.  Similarly, the NMFS survey data used in the bocaccio, canary, and  darkblotched assessments were limited in scope because the surveys were  conducted only in trawlable waters. Bocaccio, canary, and darkblotched  live in both the trawlable and untrawlable habitats. Using data from trawl  surveys conducted from 1977 through 1998, NMFS reported in 2003 that 77  percent of the survey area was trawlable and 23 percent was untrawlable.  Lacking data on species in the 23 percent of the ocean floor that is  untrawlable, the assessors estimated the overall biomass using the NMFS  data collected from the trawlable areas. However, the abundance in the  trawlable area is not representative of the abundance in the untrawlable  area. The 2003 NMFS report also found that darkblotched groundfish are  less abundant in untrawlable waters, while canary and bocaccio species  are more abundant in untrawlable waters. As a result, some rockfish  populations may be understated while others may be overstated. According  to stock assessors, relying solely on survey data from trawlable waters  increases the uncertainty of stock assessments.
In contrast, the fifth groundfish species, Pacific hake, lives primarily in  mid-water habitat; and so the concerns about the lack of NMFS data in  rocky, untrawlable habitats are not applicable.

		NMFS Lacks a Standard Process for Ensuring the Reliability of Non-NMFS Data Used in the Stock Assessments

NMFS does not have a standard process for evaluating whether the  non-NMFS data used in its stock assessments are reliable. We believe that  certain internal control activities, such as a standard process for ensuring  data reliability, can help ensure that information used to make management  decisions is complete and accurate. Lacking a standard process, some  assessors reviewed the quality of the raw non-NMFS data, while others did  not. Assessors who reviewed for data quality found mistakes that they  believed made some of the data unusable or that could have impaired the  accuracy of the stock assessments. For example, the assessor for the 2002  yelloweye stock assessment found numerous errors in the recreational  catch data, such as attributing the catch from an entire fishing vessel to a  single fisherman, and thus did not use the data because doing so could have  resulted in overestimating the biomass. According to another stock  assessor, commercial catch data frequently have inconsistencies.  Specifically, the assessor said California, Oregon, and Washington require  fishermen to enter catch and location information into logbooks, but  logbooks are often incomplete and inaccurate. While the stock assessment  review panels evaluated the assessments, the panels did not evaluate the  quality of the raw data used in the assessments.
According to a Northwest Center official, several assessors have raised  concerns about data quality and accessibility in feedback meetings. In  response to these concerns, the Northwest Center official has recently  begun assigning data stewards to each data set used in its assessments.  Data stewards are responsible for helping assessors compile relevant data  and for conducting quality assurance and quality control checks on the  data. The Northwest Center plans to conduct a data quality workshop in  July 2004 to formally establish the roles and responsibilities of the data  stewards, with the intent of standardizing the data evaluation process.

		Stock Assessments for Four Species Did Not Provide Uncertainty Estimates

In 1998, the National Research Council recommended that NMFS include  realistic measures of uncertainty in its stock assessments. NMFS’ 2001  stock assessment improvement plan also recognized the need to better  quantify and communicate the uncertainty in assessments. In a review of  the 2002 canary assessment, the stock assessment review panel  recommended that standard estimates of uncertainty be included in future  assessments because it is difficult to determine the reliability of the stock  assessment without them. Similarly, we believe that estimates based on  samples should have a range of uncertainty to show the amount of  variability in the estimates. However, the bocaccio, canary, darkblotched,  and yelloweye assessments did not present a measure of uncertainty  associated with the biomass estimates.
Without uncertainty ranges, it is difficult for regional councils and NMFS to  know how much confidence they can have in relying on the estimates for  determining stock abundance and hence for setting allowable harvests of  the fish. For example, lacking uncertainty ranges, the 2002 bocaccio stock  assessment estimated a bocaccio biomass of 2,914 metric tons in 2002. The  2003 assessment of bocaccio biomass, however, estimated 6,506 metric  tons in 2002—more than doubling the previous estimate because of  additional and updated data. With such wide variations, it is important to  provide uncertainty ranges, otherwise management may make  inappropriate decisions.
While assessors told us that their stock assessments included some  information about differences in estimated biomass when using different  data sources (sensitivity analyses), the mathematical model that NMFS  uses to estimate biomass (Stock Synthesis model) does not calculate  uncertainty ranges. NMFS officials told us that NMFS is updating the model  so that it can compute uncertainty ranges; NMFS expects to use the  updated model for all 2005 stock assessments. The Pacific hake assessor  used a mathematical model (AD Model Builder) that could compute  uncertainty ranges and included these ranges in the Pacific hake  assessment report.

	Some Recommended Stock Improvements Have Been Implemented, but Much Remains to be Done

NMFS has taken some steps recommended in the Marine Fisheries Stock  Assessment Improvement Plan to improve the quantity, quality, and type of  data used in Pacific groundfish stock assessments, but much remains to be  done to make the assessments more reliable. The Northwest Center has  concentrated most of its efforts on implementing recommendations aimed  at obtaining more data. Recommendations aimed at increasing the types of  data and improving their quality have not yet been fully implemented for a  variety of reasons, such as staffing and funding limitations. In addition,  other program priorities have precluded NMFS from implementing the  recommendation to create a comprehensive plan that incorporates the  improvement plan and related plans so that it can develop integrated  program initiatives to improve stock assessments.

		The Northwest Center Has Taken Some Steps Recommended for Improving Stock Assessments

The October 2001 stock assessment improvement plan identified three  scenarios (tiers) to consider when analyzing the resources needed to  improve stock assessments. The three tiers of assessment improvements  are as follows:   Tier 1—improve stock assessments using existing data without  initiating new data collection programs.
Tier 2—conduct baseline monitoring of species, which in most cases  requires sampling the species at least every 1 to 3 years, and preferably  at least every 1 to 2 years.
Tier 3—implement “next generation” stock assessments by explicitly  incorporating ecosystem considerations, such as multispecies  interactions and environmental effects in assessments.
The improvement plan also made a number of recommendations to  improve stock assessments. The recommendations fall into the following  four categories:  Data collection—pursue new initiatives to expand data collection  efforts that at a minimum bring stock assessment science to Tier 2. In  addition, continue to develop partnerships and cooperative research  programs with other entities, such as state agencies, commercial and  recreational fishing organizations, and individuals to improve the  quantity, quality, and type of data collected.
Communication—educate constituents about NMFS’ strategies for  improving stock assessments.
Training—implement comprehensive training and staff development  programs for NMFS’ analytical and quantitative staff, and augment  existing programs that support graduate students interested in stock  assessment science.
Planning—develop integrated program initiatives by preparing a  comprehensive plan that combines the improvement plan with its  complementary plans.
Improvement in the quantity of data collected for use in stocks  assessments is a key component to achieving Tier 2 status. The  improvement plan states that as the quantity of the data increases, the  assessments become more reliable because the data cover a longer period  of time, producing better population trend information. Northwest Center  officials said that the quality of the data improves with more frequent  surveys and more randomly selected survey locations that, over the long  term, provide a better understanding of the variability inherent in the  population distribution and abundance. A better sense of trends and  variability allow for improved short-term predictions of the status of the  species.
The Northwest Center has concentrated most of its efforts on  implementing improvements in data quantity, such as more frequent  acoustic, and shelf and slope bottom trawl surveys. The following illustrate  some of the actions the Northwest Center took in 2003 to improve data  quantity: Increased the frequency of the Pacific hake acoustic survey from  triennially to biennially. Beginning in 2003, the survey was restructured  into a single, integrated survey with Northwest Center and Canadian  officials jointly planning all survey elements. Officials from the  Northwest Center and Canada now jointly conduct all of the acoustic  surveys.
Increased the frequency of the groundfish shelf and slope bottom trawl  survey from triennially to annually, leveraging available resources by  cooperatively working with the fishing industry. Specifically, contracting  with private commercial fishing vessels to conduct the surveys.  According to Northwest Center officials, working collaboratively with  the fishing industry has afforded fishermen the opportunity to become  stakeholders in the data collection process.
Extended the geographic range of the groundfish shelf and slope bottom  trawl survey. The surveys are now coastwide from Cape Flattery,  Washington to the Mexican border, adding over 300 more miles along  the southern California coast. Previous surveys ended at Morro Bay,  California.
Efforts continue to communicate the strategies needed to improve stock  assessments and to augment existing programs aimed at developing future  stock assessment scientists. For example, through the Pacific Fishery  Management Council, Northwest Center staff meet with their constituents,  such as representatives from state agencies and the fishing industry, to  discuss groundfish management issues. In addition, the Northwest Center  organized a series of public meetings to discuss new initiatives that affect  the stock assessment program. For example, the Northwest Center held  public meetings in several communities along the Pacific coast to discuss  implementation of the Observer Program—a program designed to collect  information about discarded fish for the non-hake west coast groundfish  fleets.
Finally, the Northwest Center now participates in NMFS’ National Sea  Grant program to augment a Northwest Center-supported graduate study  program at the University of Washington that trains stock assessment  scientists. The Sea Grant program provides fellowships for students  interested in marine research, such as stock assessment methodology and  marine resource economics. The Northwest Center plans to employ two  Sea Grant students during the summer of 2004.

		Further Actions Are Needed to Improve Stock Assessments

While the Northwest Center has implemented some recommendations  aimed at improving stock assessments, it has not yet fully implemented  many others. These recommendations include collecting additional types  of data, such as ecosystem and recruitment information; improving data  quality, such as calibrating survey vessel equipment; and increasing training  opportunities for Northwest Center staff. Also, NMFS has not acted on the  task force recommendation to combine the improvement plan and its  complimentary plans into a comprehensive plan that provides integrated  program initiatives.
According to the improvement plan, additional types of data will allow  NMFS to further test and validate model assumptions, thereby increasing  the reliability of the stock assessments. The improvement plan further  states that information derived from ecosystem research and recruitment  surveys is essential if assessments are to meet the national standards of  “next generation” assessments or Tier 3 status. According to Northwest  Center officials, ecosystem information and coastwide recruitment surveys  are two of the most critical data sets needed to ensure continuous  improvement of groundfish stock assessments. The Northwest Center  conducts ecosystem research as part of its Science for Ecosystem-based  Management Initiative. Understanding the complex ecological  relationships between fish and the environment within which they exist  provides a better understanding of the effects of the ecosystem on the  groundfish fisheries and the scientific knowledge needed to make informed  ecosystem-based management decisions. Although research is ongoing to  develop ecosystem information, only a limited amount of the data is  collected and used in stock assessments. For example, ecosystem data are  collected during shelf and slope bottom trawl surveys as time and  resources allow. However, this information is not widely incorporated into  stock assessments. For the five species we reviewed, only the boccacio  assessment used ecosystem data—information on the temperature of the  ocean’s surface.
According to Northwest Center officials, the collection of ecosystem data  is limited because the relatively small size of the commercial vessels used  in the shelf and slope bottom trawl surveys cannot support the number of  researchers needed to effectively conduct comprehensive ecosystem  research and collection activities. Furthermore, the implementation of  comprehensive ecosystem research and data collection programs is  contingent upon the funding of a dedicated research vessel for west coast  surveys. Northwest Center officials said they are to receive a dedicated  research vessel sometime during calendar years 2008 or 2009, at the  earliest.
Better recruitment information for Pacific groundfish is also needed  because such information provides an early predictor of fish abundance,  especially for species such as hake, where there is a great variation in  recruitment. Northwest Center officials said that current recruitment  surveys are limited because existing funds support only yearly surveys in  selected areas. To achieve the best early predictions of stock status, these  officials said, recruitment surveys should be coastwide and conducted  twice a year. According to Northwest Center officials, 13 full-time staff are  needed to expand these and other high-priority data collection efforts, such  as surveys in untrawlable habitat and expanded acoustic surveys.
The lack of quality data was identified in the improvement plan as an  impediment to producing reliable stock assessments. For example, when  equipment on different survey vessels are not calibrated, the data are not  comparable, and trends may not be accurately determined. The Northwest  Center is continuing its efforts to calibrate survey vessel equipment.
The improvement plan also recommended that NMFS provide additional  training to ensure that qualified NMFS staff are available now and in the  future to conduct stock assessments and related activities. For example,  the plan recommended the development of a comprehensive training  program and more professional developmental opportunities for NMFS’  scientific staff. Northwest Center officials said they try to meet the training  and professional development needs of their scientific staff. However, to  date they have focused on developing external training programs, such as  the University of Washington graduate program, to develop stock  assessment scientists for the future and have yet to develop a  comprehensive training program for in-house stock assessment scientists.
Finally, the improvement plan recommended that NMFS prepare a  comprehensive plan that combines the improvement plan with other  complementary plans, such as the NOAA Fisheries Data Acquisition Plan  and the NMFS Social Sciences Plan. A comprehensive plan would allow  NMFS to better integrate and coordinate program initiatives for improving  stock assessments. For example, the acquisition plan—the key  complementary plan to the improvement plan—identifies the need for  fishery research vessels to satisfy NMFS’ data collection needs. Although  the improvement plan includes the number of staff that would participate  in data collection surveys, it does not contain the capital and operating  costs of the research vessels. Similarly, the staffing requirements for  augmenting the social sciences capabilities of NMFS to conduct economic  analyses is represented in the sciences plan and not in the stock  improvement plan. A NMFS official said that other program priorities, such  as conducting more stock assessments and improving data collection  activities, have precluded them from developing a comprehensive plan.

	Remaining Improvements Estimated to Cost at Least $8.9 Million

According to NMFS funding and budget requests, the Northwest Center  needs at least $8.9 million to complete ongoing and planned improvements  to the stock assessments for Pacific groundfish. However, the actual cost of  implementing remaining improvements to Pacific groundfish stock  assessments may be even higher because the Northwest Center’s budget  requests primarily reflect the amount of money the Center believed it could  realistically obtain, rather than the actual cost of the improvements.

		Remaining Improvements Are Estimated to Cost at Least $8.9 Million, but Estimate Is Likely Understated

According to NMFS, the Northwest Center needs at least $8.9 million to  complete ongoing and planned improvements for Pacific groundfish stock  assessments: $2.6 million that NMFS’ Northwest Center requested but did  not receive between fiscal years 2001 to 2003 and $6.3 million the Center  requested for fiscal years 2004 and 2005. Specifically, as shown in table 2,  the Northwest Center records have identified the following funding needs   $7.7 million to improve the types of data used, including $2.4 million for  surveys of untrawlable waters, $2.1 million to expand acoustic and  recruitment surveys, and $3.2 million to collect ecosystem data; and  $1.2 million to improve the quality of data used in stock assessments,  including $600,000 to enhance the calibration of vessel equipment;  $525,000 to develop and implement methods to collect information on  stock identification, structure, and movement; and $75,000 to  standardize trawl survey procedures.
The Northwest Center did not receive its full funding request, in part,  because NMFS did not receive all the funding it had requested. Between  fiscal years 2001 and 2003, NMFS received $20.6 million (80 percent of its  request) in additional funding to implement improvements for all marine  stock assessments. NMFS allocated $3.6 million (58 percent of funds the  Northwest Center requested) to the Northwest Center for improving  Pacific groundfish stock assessments, resulting in a $2.6 million shortfall in  the Center’s request. This shortfall occurred in part because of NMFS’ need  to balance the requests of its six science centers against its program  priorities and the available funds. According to NMFS officials, their goal  is to achieve parity among the science centers in terms of their capability to  conduct scientific work, such as stock assessments.

		Estimated Costs May Understate Actual Cost of Remaining Improvements

The $8.9 million needed to implement remaining recommended  improvements is probably understated because the Northwest Center’s  budget requests primarily reflect the amount of money the Center officials  believed they could realistically obtain, rather than the amount the  improvements would actually cost, according to NMFS officials. The  Northwest Center’s budget requests for fiscal years 2004 and 2005 are  preliminary requests submitted before the Northwest Center received its  fiscal year 2003 funding. Consequently, the Northwest Center will likely  submit revised budget requests for fiscal years 2004 and 2005 that account  for both its unfunded needs from fiscal years 2001 through 2003 and items  that were unexpectedly funded in fiscal year 2003. Moreover, the fiscal year  2004 and 2005 preliminary budget requests do not incorporate any  unanticipated problems or data gaps that have developed since the  Northwest Center submitted its preliminary requests.
According to NMFS officials, NMFS’ science centers, including the  Northwest Center, primarily make and justify their funding requests in  response to how much money Congress appropriates. After Congress  passes NMFS’ budget, NMFS asks its science centers to reassess and detail  how much new money each needs to implement science center programs,  such as marine stock assessment improvements. According to NMFS  officials, it is unrealistic for a science center to request more funds than are  available in its appropriation, even if it needs more. While NMFS’  Northwest Center requested $6.2 million to implement improvements to  Pacific groundfish stock assessments between fiscal years 2001 and 2003,  NMFS’ 2001 West Coast Groundfish Research Plan estimated that almost  twice as much money would be needed—approximately $11.7 million in  new funding—to implement top-priority improvements to Pacific  groundfish stock assessments.
NMFS is now updating its plan and cost estimates for improving Pacific  groundfish stock assessments. Using key findings from its December 2003  review of the groundfish program, the Northwest Center plans to update its  groundfish research plan, last published in 2001. According to NMFS, the  updated groundfish research plan should be completed in late 2004 and is  designed to (1) provide a comprehensive framework for Pacific groundfish,  (2) identify some of the greatest information gaps, and (3) provide  guidance for setting priorities on work to fill these gaps. In addition, the  updated plan will estimate how much such improvements will cost.

	Conclusions

Stock assessments are the key to effectively managing fisheries. They  provide estimates of the species population, which NMFS uses to set  harvest limits that allow for sustainability and/or recovery of the species.  While stock assessment results often change from assessment report to  assessment report, the more types of information used in the assessments,  such as recruitment surveys and ecosystem studies, and the greater the  accuracy and quality of the data, such as scientifically designed and  collected data, the more reliable the assessment results. However, the  Pacific groundfish assessments we reviewed did not (1) use scientifically  designed and collected NMFS data of sufficient scope and accuracy, such  as survey data on the abundance of groundfish residing in rocky,  untrawlable habitats; (2) subject the non-NMFS data used to a standard  process for assessing its reliability; and/or (3) identify the uncertainties of  the assessments total biomass estimates. As a result, the reliability of the  five assessments is questionable. Without reliable assessments, fishery  managers may reach erroneous conclusions and take actions that could  adversely affect the fishing industry economically or adversely affect the  recovery and sustainability of the fishery resources. Moreover, without a  comprehensive, integrated improvement plan, funding requests and  planned actions to improve the stock assessments may not be coordinated,  jeopardizing successful and timely implementation of assessment  improvements.

	Recommendations for Executive Action

To improve the reliability of Pacific groundfish stock assessments, we  recommend that the Secretary of Commerce require the Director of  National Marine Fisheries Service to take the following four actions:  Continue efforts to collect more types of data, such as data obtained  from surveys in rocky, untrawlable waters, recruitment surveys, and  ecosystem studies, for groundfish assessments where reliable data are  now lacking.
Establish a standard approach that requires that non-NMFS data used in  stock assessments be evaluated for its reliability, and continue efforts to  implement the task force’s recommendations to improve data quality.
Require that stock assessment reports clearly present the uncertainties  in the assessments, such as the margin of error associated with species  biomass estimates.
Develop a comprehensive plan that integrates the NMFS stock  assessment improvement plan with other NMFS plans to ensure that  stock assessment improvement actions and budget requests are  coordinated.

	Agency Comments and Our Evaluation

We provided the Department of Commerce with a draft of this report for  review and comment. We received a written response from the Under  Secretary of Commerce for Oceans and Atmosphere that included  comments from the National Oceanic and Atmospheric Administration  (NOAA). NOAA generally agreed with the report’s accuracy and concurred  with the reports recommendations. However, NOAA said it was concerned  about the report’s conclusion—that the reliability of stock assessments is  questionable for the five species reviewed—because it could be  misconstrued to infer that the assessments are unreliable for use in  managing the west coast groundfish fishery. In this regard, NOAA provided  additional comments to show the usefulness of the assessments, even if  some of the input data used in the assessments contained errors. We stand  by our conclusions that the five stock assessments we reviewed were  questionable because the input data were insufficient and/or potentially  inaccurate and that four of the assessment reports did not present the  uncertainties associated with the biomass estimates. Nonetheless, we  added language to the report to address NOAA’s concern. Specifically, we  expanded upon the fact that NMFS used the best information available at  the time the stock assessments were conducted by adding information on  the importance of the assessments to effectively manage the fisheries.  Without these stock assessments, NMFS and fishery managers would have  very limited information on which to base fishery management decisions.
NOAA agreed with the report recommendation to continue collecting more  types of data for groundfish assessments where reliable data are now  lacking. NOAA said that the reliability of stock assessments will be  improved if NMFS survey efforts are expanded and additional NMFS  fishery data are collected. NOAA said NMFS places a priority on these  improvements and will continue efforts to address this and other  recommendations to improve the collection of fishery data as funding  becomes available.
NOAA also agreed with the report recommendation to establish a standard  approach to evaluate the reliability of non-NMFS data used in stock  assessments and continue efforts to improve data quality. NMFS said,  through its west coast fishery science centers, it participates on  interagency data committees, to develop quality assurance protocols and to  assess the quality of non-NMFS data. NOAA agreed that it is important to  ensure that these interagency data committees continue to highlight the  need for standardized quality control procedures for the collection of data.
NOAA agreed with the report recommendation to clearly present the  uncertainties in the stock assessments. NOAA said that quantifying  uncertainty of stock assessments is important to sound decision-making by  providing more information about the assessment, although this  quantification does not reduce the uncertainty in the assessment itself.  While the methods used and the completeness of the uncertainty  characterization varied from assessment to assessment, NOAA said it is  desirable to have both a quantitative analysis of model uncertainty and an  evaluation of the consequences of alternative model scenario.
Finally, NOAA agreed with the report recommendation to develop a  comprehensive plan that integrates the stock assessment plan with other  NMFS plans to ensure that improvement actions and budget initiatives are  coordinated. NOAA said that while much remains to be done, long-term  planning efforts and coordination among field and headquarters are  ongoing, and NOAA is committed to these actions.
NOAA’s comments and our detailed responses are presented in appendix II  of this report. NOAA also provided technical comments that we  incorporated in this report as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 7 days from the  report date. At that time, we will send copies of this report to the Secretary  of Commerce and the Director of the National Marine Fisheries Service. We  will also provide copies to others upon request. In addition, the report will  be available at no charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please call me at  (202) 512-3841 or Keith Oleson at (415) 904-2218. Key contributions to this  report are listed in appendix III.

Objectives, Scope, and Methodology

We reviewed National Marine Fisheries Service (NMFS) stock assessments  for five species of Pacific groundfish: Pacific hake (Pacific whiting) as well  as four types of rockfish—bocaccio, canary, darkblotched, and yelloweye.  Specifically, for these five species you asked us to (1) assess the reliability  of NMFS’ stock assessments, (2) identify which relevant recommendations  from the stock assessment improvement plan have been implemented and  which have not, and (3) identify the estimated costs associated with  planned and ongoing improvements to groundfish stock assessments. We  did not review the stock assessments of any of the other west coast Pacific  groundfish species, thus the information contained in this report pertains  to the five species we reviewed unless stated otherwise.
For all three objectives, we reviewed key laws and agency reports and  interviewed officials from NMFS, including officials from the Northwest  Fisheries Science Center, which has lead responsibility for conducting  Pacific groundfish stock assessments with assistance from other west  coast science centers.
To assess the reliability of the Pacific hake, bocaccio, canary, darkblotched,  and yelloweye stock assessments, we examined methodological and  administrative documents developed by NMFS and others to support the  groundfish data collection, maintenance, and assessment process. We  reviewed the controls over stock assessment data, the types of fish  population surveys used, and recent Pacific groundfish stock assessment  studies (2002 and 2004 studies for Pacific hake, 2002 and 2003 studies for  bocaccio, 2002 study for canary, 2000 and 2003 studies for darkblotched,  and 2001 and 2002 studies for yelloweye). We examined whether and to  what extent NMFS has processes and procedures in place to ensure the  reliability of data used in the Pacific groundfish stock assessments. We  reviewed the stock assessment reports and determined whether they  articulated the level of uncertainty in the assessment model estimates. We  interviewed an array of government officials and fisheries experts,  including the Oregon Department of Fish and Wildlife, the Washington  Department of Fish and Wildlife, the California Department of Fish and  Game, the Pacific Fishery Management Council, the Pacific States Marine  Fisheries Commission, environmental groups, and industry associations, as  well as fishermen and academics. We did not simulate NMFS’ stock  assessment models nor evaluate the mathematical and statistical  methodologies used in the models for Pacific hake, bocaccio, canary,  darkblotched, and yelloweye.
To identify the relevant recommendations to improve stock assessments  that NMFS has implemented and has not implemented, we reviewed  agency reports on marine fisheries stock assessments, strategic planning,  and data collection. We also interviewed officials from the Oregon  Department of Fish and Wildlife, the Washington Department of Fish and  Wildlife, the California Department of Fish and Game, the Pacific Fishery  Management Council, and the Pacific States Marine Fisheries Commission  as well as environmental groups, industry associations, fishermen, and  academics.
To determine the estimated costs associated with NMFS’ planned and  ongoing improvements to Pacific groundfish stock assessments, we  reviewed relevant budget requests and funding documents for fiscal years  2001 through 2005 and interviewed National Oceanic and Atmospheric  Administration officials. We did not evaluate the accuracy of NMFS’ budget  requests for specific project items but rather used the amounts NMFS  requested for these project items to estimate the total additional costs of  implementing the planned and ongoing improvements to Pacific groundfish  stock assessments.
We conducted our review from May 2003 through April 2004 in accordance  with generally accepted government auditing standards.

Comments From the Department of Commerce

The following are GAO comments on NOAA’s letter dated May 13, 2004.

	GAO Comments

1. We added clarifying language to the scope and methodology section of  the report to clearly identify the species and activities covered by the  review. 2. We revised the report to show the publication date.  3. We revised the report accordingly.  4. We revised the report to clearly show that NMFS has not collected  enough ecosystem data and that the frequency and range of  recruitment surveys are limited. The statement does not address  untrawlable habitat. 5. We revised the report to clarify that NMFS “generally uses” NMFS’s  staff or contracts with outside experts. 6. We revised the report to more clearly differentiate between NMFS as a  whole and NMFS’ Northwest Center in particular. We made similar  revisions, as appropriate, throughout the report. 7. We revised the report to specify “bottom” trawl survey. We made similar  changes, as appropriate, throughout the report. 8. We revised the report to include the year and scope of the task force  review. 9. We revised the report to indicate that the Northwest Center is  responsible for coordinating groundfish stock assessments. 10. We revised the report to include the citation.  11. We revised the report to include the date and citation of the National  Research Council report. 12. The NMFS data used in the bocaccio, canary, and darkblotched  assessments were limited because NMFS conducted its surveys in  trawlable waters only. NMFS data were not available for untrawlable  waters, which these species also inhabit.  For this reason, we did not  revise the report. 13. We revised the report to clarify the shared responsibilities of the  Northwest and Southwest Centers.  14. We revised the report to include NOAA’s recommended definition of  stock abundance.  15. We revised the report to include larval fish.  16. We revised the report to clarify the role of the review panel.  17. We revised the report to more explicitly distinguish the five species  related to our report from other overfished Pacific groundfish.  18. We revised the report to more clearly describe the distribution of  Pacific hake.  19. Bocaccio survey data for untrawlable habitats, as stated in comment  12, was unavailable. For this reason, we did not change the report. 20. NOAA commented that highly standardized protocols are used for  collecting non-NMFS data (fishery dependent data) for rockfish. We  found that although NMFS does have collection and quality assurance  procedures for state-collected non-NMFS data, NMFS does not check  or have a standard process to verify that these data have been reviewed  for reliability. As discussed in our report, some assessors chose to  review the raw data, while others did not. Assessors who voluntarily  reviewed raw non-NMFS data found mistakes that either made some of  the data unusable or could have impaired the accuracy of the stock  assessments. For these reasons, we did not change the report. 21. We revised the punctuation accordingly. 22. We revised the report to clarify that Pacific hake live in mid-water  habitat. 23. The footnote placement and citation are in accordance with GAO  guidelines. For this reason, we did not change the report. 24. We believe that our report has addressed this issue. By referring to the  West Coast Groundfish Research Plan by its complete title, we  adequately distinguish between the two reports. For this reason, we did  not change the report. 25. We changed “health” to “stock abundance.” 26. The footnote placement is repositioned in report. 27. We revised the report to include assessment “improvements.” 28. As indicated in our report, we illustrate some of the actions that the  Northwest Center took to improve data quantity and did not intend to  provide a comprehensive list of all actions conducted to improve data  quantity coast wide. However, we added footnote 16 to clarify the  actions taken by the Southwest Center. 29. The example we provided is not intended to be a comprehensive list of  all ecosystem research conducted on the west coast. Instead, it  illustrates the type of work the Northwest Center is conducting and the  opportunities for improving ecosystem research. For this reason, we  did not change the report. 30. After reviewing the report we believe no change is required because of  subject-verb agreement. 31. We added clarifying language. 32. We believe that table 2 notes “a” and “b” in our report already  adequately address this issue. Annotations for projects that do not  separate out groundfish funds occur only in items that are annotated as  Southwest Center projects. For this reason, we did not change the  report. 33. We changed “survivability” to “sustainability.” 34. We changed “survivability” to “sustainability” and added fishery  “resources” for clarification. 35. NOAA commented that GAO does not adequately convey the different  degrees of precision associated with the stock assessments and GAO’s  conclusion that the reliability of the five assessments we reviewed is  questionable and could easily be misconstrued to mean all these  assessments are an unreliable basis for management of the west coast  groundfish fishery. NOAA also commented that the five assessments  GAO reviewed all passed scientific review and are serving as the basis  for formal status determination and fishery management. Our report  acknowledges that stock assessments are scientifically reviewed and  are a key tool for managing fisheries. However, we found the reliability  of the five assessments questionable for the three reasons we  highlighted in our report, and we recommended actions on how to  improve the reliability of the stock assessments. We added clarification  to the report to show that stock assessments are a key tool for  managing fisheries and are important in making decisions about setting  harvest levels and developing plans to rebuild overfished stocks.
NOAA also commented that quality assurance for non-NMFS data is not  absent. As stated in our response number 20, we found that although  NMFS does have collection and quality assurance procedures for state- collected non-NMFS data, NMFS does not check or have a standard  process to verify that these data have been reviewed for reliability. As  discussed in our report, some assessors chose to review the raw data,  while others did not. Assessors who voluntarily reviewed raw non- NMFS data found mistakes that either made some of the data unusable  or could have impaired the accuracy of the stock assessments. For  these reason, we did not change the report. 36. NOAA commented that it is more pertinent to focus on the degree of  standardization of the survey data than the source. By categorizing data  as NMFS data and non-NMFS data, we were not implying that non- NMFS organizations could not conduct useful fishery-independent  surveys. We categorized the data in this manner because NMFS  currently conducts nearly all of the fishery-independent surveys and  non-NMFS organizations collect most of the fishery-dependent data.  Footnote 7 in the report states that NMFS generally refers to its data as  fishery-independent data and to non-NMFS data as fishery-dependent  data. For these reasons, we did not change the report. 37. We believe the Pacific hake biomass estimates are questionable  because the assessment used non-NMFS data that NMFS did not check  or subject to standard data reliability testing. Assessors who reviewed  raw non-NMFS data for other stock assessments found mistakes that  either made some of the data unusable or could have impaired the  accuracy of the stock assessments. For this reason, we did not change  the report. 38. NOAA commented that bocaccio, canary, and darkblotched  assessments all obtain adequate abundance trend information from the  NMFS bottom trawl surveys. NOAA also commented that although  bottom trawl survey cannot access the roughest habitat, it is useful as  an index of relative changes in the overall abundance. As stated in our  report, we found that the NMFS survey data used in these assessments  were limited in scope because the surveys were conducted only in  trawlable areas. Assessors estimated overall biomass using the NMFS  data collected from the trawlable area, which has a different abundance  rate than the untrawlable area. Stock assessors commented that relying  on survey data from trawlable waters only increases the uncertainty of  stock assessments. For these reasons, we did not change the report. 39. As noted in our report, the National Research Council found that the  inclusion of NMFS survey data was the best option for a reliable  estimate of abundance because such surveys use an unbiased statistical  design, control sampling locations, and provide for quality assurance.  Northwest Center officials said that to obtain reliable results, each  stock assessment should include at least one source of NMFS-collected  data of sufficient scope and accuracy because such surveys are  unbiased and scientifically designed. NMFS data were unavailable for  the yelloweye assessment. Northwest Center officials also raised  concerns about basing assessments solely on non-NMFS data such as  commercial and recreational catch data. Catch data do not provide the  species’ relative or absolute biomass, according to NMFS officials.  Catch data alone are insufficient because fishermen are not randomly  sampling the ocean, but are fishing areas that they are allowed to fish  and they believe have the most fish; fishing restrictions, such as a total  allowable catch, can limit the amount of fish being caught; and catch  data have often been inaccurate for a variety of reasons, such as  imprecise accounting for dead fish tossed back into the ocean. For  these reasons, we did not change the report. 40. NOAA commented that the doubling of estimated bocaccio biomass in  2003 was due to factors that would not be addressed in a standard  statistical analysis. Although a standard statistical analysis may not  fully address the doubling of an estimate, an assessment without an  uncertainty range does not quantify and communicate any of the  uncertainty. For this reason, we did not change the report.

GAO Contact and Staff Acknowledgments

	GAO Contact

	Staff Acknowledgments

In addition to the person named above, Leo G. Acosta, Kristine N. Braaten,  Allen T. Chan, David Dornisch, Alan Kasdan, Robert Marek, Cynthia C.  Norris, Carol Herrnstadt Shulman, and Tama R. Weinberg made key  contributions to this report.

	GAO’s Mission

The General Accounting Office, the audit, evaluation and investigative arm of  Congress, exists to support Congress in meeting its constitutional responsibilities  and to help improve the performance and accountability of the federal government  for the American people. GAO examines the use of public funds; evaluates federal  programs and policies; and provides analyses, recommendations, and other  assistance to help Congress make informed oversight, policy, and funding  decisions. GAO’s commitment to good government is reflected in its core values of  accountability, integrity, and reliability.

	Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is  through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- text files of current reports and testimony and an expanding archive of older  products. The Web site features a search engine to help you locate documents  using key words and phrases. You can print these documents in their entirety,  including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and  correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site  daily. The list contains links to the full-text document files. To have GAO e-mail this  list to you every afternoon, go to www.gao.gov and select “Subscribe to   e-mail alerts” under the “Order GAO Products” heading.

		Order by Mail or Phone

	To Report Fraud, Waste, and Abuse in Federal Programs

	Public Affairs

Summary:
Why GAO Did This Study

Because of concerns raised about the accuracy of National Marine Fisheries Service (NMFS) stock assessments, GAO reviewed the assessments for five species of Pacific groundfish: Pacific hake and four types of rockfish--bocaccio, canary, darkblotched, and yelloweye. Specifically, for these five species GAO (1) assessed the reliability of NMFS' stock assessments, (2) identified which relevant recommendations from NMFS' stock assessment improvement plan have been implemented and which have not, and (3) identified the costs associated with planned and ongoing improvements to groundfish stock assessments.

What GAO Found

The reliability of the NMFS assessments is questionable for the five species GAO reviewed, although the assessments were based on the best information available at the time they were conducted. According to NMFS officials and a National Research Council report, to obtain reliable results each stock assessment should include at least one NMFS data source of sufficient scope and accuracy because such data are derived from unbiased, statistical designs. However, in the yelloweye assessment, no NMFS data were used, and in the darkblotched, canary, and bocaccio assessments, the NMFS data were limited because the NMFS' surveys were conducted in trawlable waters only. A 2003 NMFS report concluded that darkblotched groundfish are less abundant and bocaccio and canary are more abundant in untrawlable waters. Also for all five species, NMFS lacks a standard approach for ensuring the reliability of non-NMFS data used in stock assessments. Some assessors reviewed the quality of non-NMFS data; others did not. The assessors who reviewed the quality of the non-NMFS data found errors that made some of the data unusable or that could have impaired the reliability of certain stock assessments. Finally, for four species, the stock assessment reports were questionable because they did not present the uncertainty associated with the population estimates. For example, the canary stock assessment review panel recommended that standard estimates of uncertainty be included in the assessment report because without them it is difficult to determine their reliability. NMFS has taken steps to implement some of the recommendations contained in the NMFS stock assessment improvement plan, but much remains to be done. NMFS has concentrated its efforts mostly on improving data quantity. For example, NMFS increased the frequency of groundfish stock assessments and extended the geographic ranges of the shelf and slope surveys to cover over 300 more miles along the southern California coast. However, because of staffing and funding limitations, NMFS has not yet implemented many of the recommendations aimed at obtaining more types of data and improving data quality. For example, NMFS has not collected enough ecosystem data, and the frequency and range of recruitment surveys (estimated production of new members of a fish population) are limited. Finally, because of other program priorities, NMFS has not implemented the recommendation to create a comprehensive plan that combines the improvement plan and its complementary plans. NMFS records indicate at least $8.9 million is needed to complete ongoing and planned stock assessment improvements--$2.6 million that NMFS' Northwest Fisheries Science Center requested but did not receive in fiscal years 2001 to 2003, and $6.3 million requested for fiscal years 2004 and 2005. It will cost about (1) $7.7 million to improve the types of data used, such as more untrawlable water and recruitment surveys and (2) $1.2 million to improve the quality of data used in stock assessments, such as enhanced calibration of vessel equipment and standardized trawl survey procedures. The actual cost of the remaining improvements may be even higher than the $8.9 million estimated because the estimates primarily reflect the amount of money that agency officials believed could be realistically obtained, rather than what the improvements might cost.