March 12, 2004

Mr. David E. Frulla

Mr. Shaun M. Gehan

Mr. Andrew D. Herman

Brand and Frulla

923 Fifteenth Street, N.W.

Washington, D.C. 20005

Dear Messrs. Frulla, Gehan, and Herman:

This letter responds to your appeal (\"Appeal\"), on behalf of The
Associated Fisheries of Maine, Inc. and the Trawlers Survival Fund, of
the December 15, 2003, Denial of the Request for Correction (\"Initial
Decision\") of the revised biological reference points for the northeast
multispecies stock complex (\"New England groundfish\") contained in the
draft document for Amendment 13 to the Northeast Multispecies Fishery
Management Plan (\"Draft Amendment 13\"). My office has completed its
review of your appeal and has the following response:

[Summary of Response]{.underline}

Your appeal raised six issues:

1.  Independence of one member of the peer review panel.

2.  \"Usefulness\" of the reference points.

3.  Whether the National Oceanic and Atmospheric Administration (NOAA)
    considered the ASPM approach suggested by Dr. Butterworth.

4.  Whether the \"best available science\" must be unique to the
    exclusion of all other approaches.

```{=html}
<!-- -->
```
5.  The meaning of \"best available science\" as it relates to the
    evaluation of Dr. Butterworth's method.

6.  Whether NOAA has been open to investigation of Dr. Butterworth\'s
    work.

Upon review, your arguments have been found without merit, for the
reasons stated below.

[Detailed Response: Issues Addressed in the Appeal]{.underline}

Your Appeal questions the independence of one member of the peer review
panel. This is an issue you did not raise in your initial request for
correction, even though the request in fact relied on selected
observations of the peer review panel and on the Peer Review Report to
challenge the reference points. As noted in the Initial Decision, the
peer review conducted in this instance was

conducted with independence and objectivity under proper and
clearly-articulated procedures. The decision to classify Dr. McAllister
as an \"independent\" reviewer was made by the Center for Independent
Experts (CIE). NOAA was not involved in the selection process, which is
normal practice when using the CIE. We note that the independence of the
other four reviewers was not questioned and their comments are also
supportive of the Northeast Fisheries Science Center (NEFSC) results.

On your second point, regarding the usefulness of the reference points,
we note that your argument is predicated on the condition that the
reference points provided by NEFSC are \"potentially badly wrong,\"
which pertains to accuracy, an element of the *objectivity* criterion,
rather than the *usefulness* criterion. Nevertheless, there is no
evidence that the reference points are wrong. The reference points given
in the report of the \"Working Group on Re-Evaluation of Biological
Reference Points for New England Groundfish\" (March 2002) simply differ
from those provided by Dr. Butterworth. Several members of the peer
review panel chaired by

Dr. Andrew Payne in February 2003 commented on the results of the
age-structured production model (ASPM) method presented by Dr.
Butterworth. For example Dr. Andrew Payne summarized comments from
several reviewers, including Dr. Robin Cook, as: *\"The analyses
(Butterworth et al.) were not intended to be definitive, and some sample
runs were made during the meeting for illustrative purposes.\"* and
*\"One main point of concern to the panel was that the ASPM seems to be
overly sensitive to the assumptions made.\"* The peer review panel
members distinguished between the ASPM method on the one hand and the
assumptions involved and the manner in which the model was formulated on
the other.

Peer review panel members concluded that there was insufficient
information in

Dr. Butterworth's presentation to adequately determine how the model was
formulated and, therefore, saw no basis for comparing the ASPM results
with those provided by the NEFSC. For example, Dr. Ewen Bell noted:
*\"Any stock assessment model is just that - a model of the system and
the results are conditioned on the assumptions made by the model.\"* and
*\"The large range of results for stock size and subsequent reference
points obtained from ASPM is of concern in that the model appears to be
very sensitive to the assumptions made.\"* Finally, we note that the
Petitioners acknowledge the preliminary nature of the ASPM results:
*\"Dr. Butterworth performed the analyses under the ASPM approach for
the two cod stocks as an illustration of that model's potential
utility.\"* (Petitioners' Request for Correction of Information, Page
5.)

Furthermore, regarding the accuracy of the reference points, the peer
review panel agreed with the adaptive approach for management originally
suggested by NEFSC, whereby any potential incorrect estimates of B~msy~
would be discovered and modified at several points during the stock
rebuilding period.

There are many \"viable\" approaches and methods for conducting stock
assessments and deriving biological reference points. The ASPM method
employed by Dr. Butterworth is just one of a family of forward
projection models available today. It is not a question of whether one
model is \"demonstrably more correct,\" but rather how a particular
model is configured for a given assessment that determines whether the
results are \"viable.\" The peer review looked at the underlying
assumptions, input data, and configuration of the NEFSC assessments in
order to determine whether the results are useful for providing
management advice. As yet, the results of the ASPM method prepared by
Dr. Butterworth have not been subjected to this type of assessment. To
support this request for correction and appeal, the burden of showing
that the ASPM method is superior to those promoted by NEFSC for
Amendment 13 lies with the requester.

As to your third point, that NOAA did not consider the ASPM approach
suggested by

Dr. Butterworth, the published reference points were developed by the
Working Group on Re-Evaluation of Biological Reference Points for New
England Groundfish which met in March 2002. The Working Group evaluated
a number of approaches and concluded that the models eventually employed
for each stock were most appropriate based on available data and
individual stock dynamics, and how well the data and stock dynamics met
the assumptions of the selected model.

Your fourth point argues that \"best available science\" must be unique
to the exclusion of all other approaches. This is a misperception of the
meaning of \"best available science,\" and, in fact, members of the peer
review panel did comment on whether the ASPM results (as opposed to the
ASPM method) were sufficiently documented to be considered as a basis
for providing management advice. Peer review panel members expressed
concern over the lack of sufficient documentation of the ASPM results to
view them as \"best available scientific information.\" While reviewers
found the Butterworth approach ostensibly sound, the *results* were
questionable due to the inputs and statistical assumptions.

Your fifth point also shows a misperception of the meaning of \"best
available science.\" The peer reviewers could not evaluate Dr.
Butterworth's method because he did not provide sufficient information
on his formulation of the ASPM, the data he used as input, and
assumptions he was forced to use in order to apply the ASPM method to
New England cod stocks. The argument was not whether the ASPM method was
equal or superior to the methods supported by NEFSC, the argument was in
how the ASPM method was applied for this particular situation.

Your sixth point disputes that NOAA has been open to investigation of
Dr. Butterworth\'s work, but in fact NEFSC has maintained an open dialog
with Dr. Butterworth and his assistants, including full disclosure of
all data requested by him and several offers for meetings in person. It
is more important that Dr. Butterworth submit his work to a peer review
body of independent experts, such as the same CIE panel that called for
a comparison of his method with the ones adopted for use in Amendment 13
or the Stock Assessment Review Committee (SARC). This will allow a
thorough examination of assumptions, input data, and model formulation,
rather than a limited discourse responding only to concerns raised by
NEFSC scientists. In fact, at its November 2003 meeting in which Mr.
Frulla was in attendance, the New England Fishery Management Council
passed a motion requesting that such a review take place in the near
future:

> \... an assessment update of the groundfish complex will be conducted
> in 2005 for potential management action in 2006, but will include a
> full and independent review of Dr. Butterworth's Age-Structured
> Production Model work. Benchmark assessments, including a review of
> the status determination criteria, will be conducted in 2008 for
> implementation in 2009. The assessment update in 2005 will not update
> the numerical estimates of status determination criteria unless the
> review of Dr. Butterworth's work justifies reconsideration. Both
> actions will be subject to peer review by independent scientists.

The issue of formal peer review of the ASPM method is brought up again
on page 4 of the 14 January 2004 letter. The fact that the
age-structured production model approach is used by assessment
scientists around the world, including those at the Northeast Fisheries
Science Center, is not the issue. We fully agree that the ASPM method
has been subjected to considerable scrutiny worldwide and has been
judged to be sound; in fact, the method is included in the latest
version of NMFS's Stock Assessment Toolbox, a website providing methods
that have undergone scrutiny by the agency and have been approved for
use in stock assessments. The real issue is how the method (or any
method proposed by anyone outside the established peer review process)
is used by Dr. Butterworth to derive results sufficient for providing
advice to fishery managers. Each analysis must be judged individually,
with particular attention given to model formulation, input data, and
ability to meet the model's underlying assumptions. This has not yet
occurred for the results presented by Dr. Butterworth for the New
England cod stocks.

[Conclusion]{.underline}

NOAA does not claim that its model is superior to the exclusion of all
other approaches; nor is such a claim necessary to a finding of \"best
available science.\" The reference points obtained as a result of the
NEFSC modeling efforts have been scrutinized by several panels of
internationally recognized experts, who carefully examined the data and
the methodology. In their collective judgment, the results reported out
of these reviews are of sufficient scientific merit to serve as a basis
for providing management advice, and none of the alternatives presented
to them provided a higher standard of science or could serve as a basis
to challenge the NEFSC results.

Therefore, the Appeal is denied.

Sincerely,

xx/Signed/xx

William Hogarth, Ph.D.

Assistant Administrator

for Fisheries