APPENDIX C
Examples of "Charges" to Peer Review Team

Central Arizona Project Charge

The Central Arizona Project archeological program
is currently the largest archeological rescue project
in the land. Its completion will take several more
years of field work, analysis, and report writing.
The object of the review of the program you are
about to undertake is to assess the project at the
level of detail you deem appropriate in order to
provide a written report to the Departmental
Consulting Archeologist and the Bureau of
Reclamation. We want to know the weaknesses as
well as the strengths of the program. We want you
to know that we will seriously consider all of the
recommendations the team may make to improve
the program. Consequently, no aspect of the
program is off-limits to your review. You have
already been provided with a weighty set of
documents related to the program and over the next
couple of days you will have the opportunity to
meet with Bureau of Reclamation personnel, several
contractors, representatives of the State Historic
Preservation Officers and the Advisory Council on
Historic Preservation. We have developed a list of
topics in which we think that you will be
interested. Some of these topics are covered in the
documents we have provided. Today and tomorrow
we, and others, will provide you with additional
information as your review progresses.
Additionally, we have prepared a suggested outline
for your report. Hopefully the efforts we have put
forth will make your job easier and your work
efficient. In closing we want to emphasize that your
review of the project is to be thorough and
comprehensive. Be assured that we are here to
provide you with the facts and the contexts in
which the program has been developed and
managed.

Carson Desert Archeological Peer Review Charge
The objective of the review of the program you are
about to undertake is to assess the project at the
level of detail you deem appropriate in order to
provide a written report to the Departmental
Consulting Archeologist. We want to know the
weaknesses as well as the strengths of the program.
‘We want you to know that we will seriously

consider all of the recommendations the team may
make to improve the program. Consequently, no
aspect of the program is off-limits to your review.
You have already been provided with a weighty set
of documents related to the program and over the
next couple of days you will have the opportunity
to meet with interested individuals and
organizational personnel from the Fish and Wildlife
Service, Bureau of Reclamation, Bureau of Land
Management, Nevada State Museum, Fallon
Paiute-Shoshone Tribe, Anthropology Department
University of Reno, Intermountain Research,
Churchill County Museum and Archives, Advisory
Council on Historic Preservation, and the State
Historic Preservation Officer.

We have developed a list of topics which we suggest
that you review. Some of these topics are covered in
the documents we have provided. Additionally, we
have prepared a suggested outline for the report.
Hopefully the efforts we have put forth will make
your job easier and your work efficient.

I would like to emphasize that your review of the
project is to be thorough and comprehensive and is
not restricted to the suggested topics if the team
determines that other matters are pertinent. Be
assured that we are here to provide you with the
facts and the context in which the program has
been developed and managed.

We expect a formal written report of the Peer
Review Team. This report will constitute the Peer
Review Team’s official findings and
recommendations. The following outline is offered
only as guidance to the Peer Review Team. The
Peer Review Team may develop its own report
format but we expect that it will provide objective
criteria and your evaluation of them by which the
program can be judged in respect to its aims, costs,
productivity, quality, etc.

Bennie C. Keel
Departmental Consulting Archeologist
September 16, 1988

oes


Table 1. Projects for Which Peer Reviews Were Done by the Departmental Consulting Archeologist, Archeological

Assistance Division, National Park Service

Project/Location

Dolores Project, McPhee Reservoir, CO

Central Arizona Project, AZ

Jackson Lake Dam, Grand Tetons National Park, WY
Stillwater Wildlife Management Area, Carson Desert, NV
Libby Dam Project, Lake Koocanusa, MT

Alkali Creek Project, ND

Year Agency

1981 Bureau of Reclamation
1986 Bureau of Reclamation
1987 National Park Service
1988 Fish and Wildlife Service
1989 Corps of Engineers

1990 Soil Conservation Service

Project Selection

Since 1981, the DCA has conducted six peer reviews
(Table 1). The experience from conducting these
reviews is the basis for this Technical Brief. These
projects were selected for a variety of reasons. The
Dolores Reservoir Project was reviewed because of
problems between the contractor and the agency
about the scope of the investigation and project
management. The Central Arizona Project was se-
lected because the Bureau of Reclamation wanted
to confirm that the scope and focus of the archeo-
logical project were appropriate. The Jackson Lake
Dam Project was evaluated to demonstrate to the
members of the Wyoming Congressional delega-
tion that the work was adequate and satisfied State
interests. The Fish and Wildlife Service sought a
peer review to address Native American concerns
and to assure that historic preservation require-
ments were being met in the Stillwater Wildlife
Management Area Project. The U.S. Army Corps
of Engineers wanted to confirm that its historic
preservation responsibilities were being fulfilled in
a manner sensitive to Native American religious
concerns in the Libby Dam Project. The Soil Con-
servation Service wanted an evaluation of the Alkali
Creek Project in light of SHPO and professional
community concerns.

Organizing and Conducting
Peer Reviews
A peer review of a project is an activity that must

be developed harmoniously as a partnership be-
tween the peer review manager and the agency. For

all but the Alkali Creek peer review, the DCA
served as peer review manager. In the case of Alkali
Creek, a senior NPS archeologist well acquainted
with the project area was designated by the DCA
as peer review manager. Normally the agency rep-
resentative has been the archeologist responsible for
the administration and conduct of the agency’s
archeological program, although another agency
official may serve in this role. The need for frequent
and clear communication between the peer review
manager and the agency archeologist or other offi-
cial in charge of the project is mandatory. Good
communication is an absolute necessity for a suc-
cessful peer review. No matter who serves in them,
these roles are essential components of the peer
review process.

All aspects of the peer review are developed
through close coordination between the peer re-
view manager and the agency. Both parties must
agree on the scope of the review. Frequent and open
communication is necessary in all aspects of the
project. The agency archeologist needs information
about the progress of the planning effort by the peer
review manager to develop agency plans. The peer
review manager also discusses the peer review with
the SHPO and ACHP to gather information that
should be supplied to the team. Frequent commu-
nication bolsters the commitment made by both
parties to assure that the necessary actions are com-
pleted in a timely manner. In undertaking a peer
review, one of the peer review manager's goals is to
create a partnership with the agency. The creation
of such a partnership is the principal ingredient that



review team. The purpose of the proposed report
outline is to provide the team with guidance. The
interviews provide specific and important informa-
tion and views from a variety of perspectives. The
peer review manager always assures the team in
each review that the content and recommendations
of the final report are entirely their responsibility.
The team is welcome to change or reorganize the
outline and format. Team members may add or
drop topics as they think necessary or appropriate,
though departures from the model usually concern
format or organization.

The importance of coordination and careful atten-
tion to details in production of the final draft report
cannot be overestimated by the peer review man-
ager. Some cautionary examples are cited from the
peer reviews conducted by the DCA to demon-
strate the importance of organized, considered ac-
tivities and then continuing to learn from
experience.

The peer review team for the Dolores Project at
McPhee Reservoir spent an intensive week inter-
viewing various parties concerned with the project.
After the on-site visit the team members returned
to their regular employment and prepared the draft
final report. As a long-distance effort, completion
of the draft was delayed considerably. This delay
was caused by factors such as lack of communica-
tion between the members of the team, the time
taken to circulate drafts and comments sequentially
through the mail, and other commitments or pri-
orities of the panel members. The Dolores Project
peer review team was not provided with a proposed
report outline, so team members had to develop an
outline and report format, which proved to be very
time-consuming. Based on this experience the DCA
determined it would be helpful in the future to
provide guidance and suggested topics to cover.
This practice has expedited the preparation of draft
peer review reports.

The Central Arizona Project peer review team
decided that it would complete a draft report by the
end of the on-site visit. Despite certain misgivings,
this proposal was encouraged and the draft report
was delivered as promised. The DCA edited the
draft report for grammar and worked with the for-

mat, but made no changes in content. A revised
draft was mailed simultaneously to the authors for
revision. Marked and revised copy was returned to
the DCA and changes made. The final report was
produced and distributed to the Bureau of Reclama-
tion in a short period along with the DCA’s own
recommendations. Other interested parties were
sent copies as part of this distribution. The Central
Arizona Project peer review demonstrated the clear
value of requiring the team to complete a draft
report by the end of the on-site visit.

Appendix D provides examples of report outlines
presented to the peer review teams for the Central
Arizona and Carson Desert projects.

8. The Final Report

The production of the final report of the peer
review can bea rather simple, but intense task. The
draft report is edited for style and format and
unclear language is revised. All changes in the
manuscript are made using the “strikeout,” “bold”
or other options of the chosen word processing
program. This allows the author to identify easily
the changes that have been made in the revised text.
This draft is mailed simultaneously to the authors
for their corrections and approval. The peer review
manager makes the changes required by the authors
and produces the final report. Depending on the
quality of the draft report, production of the final
version requires some 16 to 24 person hours of
effort.

The peer review manager transmits the report and
recommendations for action to the agency for con-
sideration. The peer review manager may wish to
include his or her own recommendations, which
address the findings and recommendations of the
peer review team and may be mandated according
to the authorities by which the peer review man-
ager undertook this task. While the peer review
manager normally endorses most of the recommen-
dations made by the peer review team, occasionally
it will be determined that the recommendations do
not really improve the project when compared to
costs by the agency to implement them. Irrespec-
tive of whether the peer review manager supports
or rejects the recommendations of the team, a clear
explanation is always provided.

10

er


gists from academic positions. Two were experts in
the archeology of the project region. One was an
expert from outside the area. This composition was
modified for later peer reviews. It was learned that
academic archeologists were unfamiliar with the
NHPA Section 106 process and unaware of many
of the legal requirements of Federal historic preser-
vation. They had little or no knowledge of Federal
procurement and contracting regulations and,
therefore, they found it difficult to readily identify
efficient, effective solutions to problems related to
these topics.

For later reviews, an archeologist from a public
agency, but one different from the one under re-
view, was added to the team. This served the addi-
tional goal of increasing the interaction among
government archeologists and their academic col-
leagues. This was a way to improve the credibility
of the government archeologists with their aca-
demic counterparts. We have found that a balanced
peer review teams consists of one public archeolo-
gist from an agency other than the one responsible
for the project, an academic expert in the archeol-
ogy of the project area or region, and an academic
expert from outside the research area.

To identify potential review team members the
peer review manager and the agency archeologist
independently compile three lists of candidates for:
(1) the regional academic expert, (2) the outside
academic expert, and (3) the public agency arche-
ologist. They consolidate their lists and decide
which candidates and alternates to invite.

Care is to be given in selecting members of the
panel. Beyond a candidate’s scientific credentials
and availability, other elements are considered, for
example:

(1) Does the candidate have a reputation for
completing work in a reasonable time?

@

Does the candidate have a previous relationship
or any other consideration with the agency or the
project that could be construed as a conflict of
interest?

(3) Was the candidate an unsuccessful bidder for the
contract?

(4) Does the candidate work well with others?

While these elements may seem straightforward,
ignoring them could make completing an accept-
able peer review difficult or affect its timeliness.

Once the peer review manager and agency arche-
ologist agree on the candidates and the approximate
dates for the review, the peer review manager tele-
phones the candidates to determine their willing-
ness and availability, Acceptance of the invitation
is followed by a formal letter from the peer review
manager providing information about the project,
the schedule, and other details.

4, Compiling Documentation for the

Peer Review Team

By the time of the on-site visit the peer review team
should have a thorough knowledge of the project.
The agency responsible for the project compiles and
furnishes the panel with the documents that de-
scribe the undertaking, the archeological resources,
their significance, the research questions, and back-
ground information about the local archeology.
Acquisition of this necessary knowledge by the peer
review team is facilitated by providing all the docu-
mentation to team members at least three weeks
before the on-site visit. If it is provided with less
time before the on-site visit the peer review team
will not have time to review this basic information.

The package of material compiled for the orienta-
tion of the peer review team will vary from project
to project. Typically it will consist of survey and
evaluation reports and NHPA Section 106 docu-
mentation, such as the Memorandum of Agreement
or Determination of No Adverse Impact, scope of
work or request for proposal, and research design.
Interim or annual progress project reports, if avail-
able, also are provided. Additionally, correspon-
dence and internal documents may be included. (A
list of the kinds of materials supplied for the Central
Arizona Project and Stillwater National Wildlife
Management Area peer reviews is provided in Ap-
pendix B.)

5. Scheduling the Peer Review

The ideal time for a peer review is at the midpoint
of a project. Typically, field and laboratory work
has proceeded to the stage where the review team
can make a reasonable evaluation of the results. A
peer review at the beginning of a project is prema-
ture because there will be few results to assess. If the



has made the process, as conducted by the DCA,
successful.

Once a project is selected, the following tasks, at a
minimum, must be undertaken to conduct the peer
review:

(1) Defining the review’s scope and schedule.

(2) Determining the review’s cost and securing

funds.

(3) Planning the team’s composition and choosing
members.

(4) Compiling documentation and mailing to the
team.

(5) Scheduling the on-site visit.
(a) Arranging travel and lodging.
(b) Reserving meeting space.
(c) Making appointments for interviews.
(d) Preparing the agenda and itinerary.
(e) Identifying review topics.
(f) Developing support for the team.
(g) Making arrangements for office space.
(h) Securing equipment for team use.

(6) Conducting the on-site visit.

(7) Preparing the draft final report (by peer review

manager).

(a) Obtaining revisions to the final draft
report from the review team.

(b) Revising manuscript.

(c) Obtaining review team approval of final
report.

(d) Producing the final report.

(€) Compiling recommendations for the
agency.

() Completing remaining administrative
tasks.

(8) Providing final report to agency officials.

The following sections of this Technical Brief will
provide information about each of these tasks. The
examples cited are drawn from peer reviews con-
ducted in the past by the DCA.

1. Defining the Scope of the Peer Review

The reasons for selecting a project for peer review
normally defines the scope of the review. If charges
of inadequacy have been made, the peer review will
focus on the conduct of the archeological investiga-
tions under the agreements and documents cover-

ing the project. If the peer review is a response to
problems between the agency and the contractor,
the peer review will focus on these matters. The
topics identified for review for the Stillwater Wild-
life Management Area are provided as examples in
Appendix A; those for the Jackson Lake Dam pro-
ject were similar. In any case, the scope of the peer
review should focus on identifying solutions to
problems and not foster inappropriate or adversar-
ial investigative roles.

2. Cost of the Peer Review and Funding
Arrangements must be made to fund the various
costs for peer reviews and establish the budget
parameters within which the peer review manager
must operate. The costs of peer reviews conducted
by the DCA, for instance, have been covered by
various means, including the DCA’s annual operat-
ing budget, shared expenses of the DCA and the
agency, and full funding by the agency. When an
agency shares or covers the costs, the funds typi-
cally come from the project budget. Cost sharing
between the DCA and the agency has been the most
common arrangement.

Honoraria or consulting fees have not been paid to
members of the DCA peer review panels. Payment
of honoraria to non-Federal members of the team
is legal, but it is illegal to pay a Federal team
member. It seemed inappropriate to compensate
some team members but not others. Furthermore,
there was concern that if honoraria were paid the
peer review team might not feel that it could be
critical and candid in its assessment. There was also
a potential for the review to appear prejudiced.

The costs of transportation and per diem of the peer
review members must be considered. The purchase
of airline tickets may take a large portion of the
funds available. Other costs include: reproduction
and postage to provide project information to the
peer review team; on-site equipment, materials, and
work space for the team; administrative support;
and final report preparation and distribution. Many
of these costs can be covered in-kind by the agency
or the peer review manager.

3. Composition and Selection of the

Peer Review Team

The first peer review team assembled by the DCA
consisted of three nationally recognized archeolo-

»



related to the archeological work. The plan for the
rest of the on-site visit will depend on the schedules
of individuals, the weather, field conditions, and
other variables. During the week-long peer reviews
conducted by the DCA, personnel interviews and
field visits normally were concluded by the end of
the third day so the team could review its findings
and draft its report. Typically the draft report was
completed by the evening of the fourth day. Exit
interviews with the DCA, agency personnel, and
other interested parties were conducted on the fifth
day.

The peer review team normally interviews agency
personnel such as the project engineer, environ-
mental staff, and the archeologist. Other historic
preservation specialists, the contracting officer, or
contracting officer’s technical representative may
be interviewed. The team meets with the contractor
or principal investigator and the principal investi-
gator’s senior staff, such as field directors, crew
chiefs, data control manager, and laboratory direc-
tor. The team also meets with the interdisciplinary
consultants, such as geologists, geomorphologists,
pedologists, paleobotanists, paleozoologists, and
physical anthropologists. The team usually will
visit the site or sites currently under investigation
and the laboratory facilities of the contractor to get
a feel for the day-to-day operation and management
of the project. In some instances the peer reviewers
will meet with some members of the professional
community and Indian Tribes or their repre-
sentatives. The team also may interview interested
members of the public and Congressional staffers.

Each peer review will vary regarding the parties
interviewed because the focus of the peer review
will be different in each project. In the peer reviews
conducted by the DCA, it was standard procedure
to provide the SHPO and ACHP with a chance to
meet with the peer review team. Sometimes this
was done by telephone interviews. This considera-
tion is based on the regulatory roles of these agen-
cies in historic preservation. Furthermore, the
professional staff of the SHPO are especially knowl-
edgeable about their State’s archeological resources
and can provide important perspectives to the team.

The scope of the peer review and the specific cir-
cumstances will determine whether or not the
agency archeologist or official attends all of the

=
interviews conducted by the team. In cases where
problems exist between the agency and the contrac-
tor, SHPO, ACHP, or other parties, it has been |
found prudent for the team to meet both with and
without the agency personnel. The absence of
agency personnel may allow for open and candid
discussion that otherwise would be difficult.

The on-site visit is a period of intense work by the
peer review team. Every effort is made to keep the

Re-excavation conference with contractor and peer re-
view team, Central Arizona project.
Photo courtesy Bureau of Reclamation/Dan McKeever

team on schedule and focused on its task. The peer
review manager and agency personnel provide all
reasonable support requested to assure that the
team’s tasks are completed by the scheduled time.

7. Peer Review Draft Report

The peer review manager and the agency archeolo-
gist prepare a proposed outline for the report, mak-
ing it clear to the peer review team that the topics
and proposed outline are only suggestions. The
content of the final report is determined by the peer



U.S. Department of the Interior
National Park Service
Cultural Resources

Technical Brief No. 14
September 1993
ISSN 1057-1574

This Technical Brief describes objectives, organization,
and methods that could be used for conducting peer
reviews in public archeology projects. Each of these
elements is illustrated with examples drawn from peer
reviews conducted by the Department of the Interior’s
Departmental Consulting Archeologist between 1981
and 1990, The purpose of this Technical Brief is to
provide guidance to government agencies and other
archeological resources management programs on use
of the peer review process as one tool to improve the
effectiveness of their activities. It may be an especially
important tool to help resolve conflicts which may arise
due to the need for interagency cooperation, the com-
plexity of the archeological work, or professional dis-
agreements. This Technical Brief demonstrates how
peer reviews can address these typical problems and
explains ways the process can be implemented.

Introduction

The review of research by peers has been one hall-
mark of science throughout its history. Peer re-
views are the conscience of science; they provide a
necessary check on its practitioners and an explicit
evaluation of their efforts. Other professions have
recognized the importance of peer reviews for ap-
plied sciences as well. For example, the American
Society of Civil Engineers and American Consult-
ing Engineers Council (ACEC) have developed a
program that offers peer review services for engi-
neering projects and programs (ACEC 1990
a,b,c,d).

The Departmental Consulting Archeologist (DCA)
and his staff in the Archeological Assistance Pro-
gram are authorized by the Secretary of the Interior
to provide leadership and coordination for Federal

Departmental Consulting Archeologist Archeological Assistance

The Peer Review of Public Archeology
Projects: A Procedure Developed by the
Departmental Consulting Archeologist

Bennie C. Keel, Southeast Archeological Center, National Park Service

and other types of public archeology. The Secretary
is assigned these roles by several statutes including
the Antiquities Act, Historic Sites Act of 1935,
National Historic Preservation Act, and Archae-
ological Resources Protection Act. The peer review
of selected Federal archeological projects is one
among a number of DCA programs that provide
leadership and coordination (Canouts 1992;
Canouts and McManamon 1991; Knoll 1991; Knud-
son and McManamon 1992; McManamon, Smith,
and Waldbauer 1990).

This Technical Brief describes the purposes and the
goals of the Federal archeology peer review process
as conducted by the DCA between 1981 and 1990.
The government agencies and the professionals
who have been in these reviews recognize that peer
reviews can provide an important contribution to
project design and management, thereby advancing
American archeology. This brief provides an exam-
ple of the peer review process in the Federal archeo-
logical arena, as opposed to the academic domain.
It also supplies information about the planning,
design, and implementation of peer reviews. Four
appendices with examples of documents illustrating
efforts at planning and implementing peer reviews
are provided.

Purposes and Goals of the Federal
Peer Review Process

In an academic research framework, as opposed to
typical public archeology projects, the principal
investigator develops a research program to inves-
tigate problems that are of theoretical, methodo-
logical, and/or substantive interest. The researcher
develops a research design, identifies and obtains



References Cited

American Consulting Engineers Council
1990a Project Peer Review Guidelines. American Consulting Engineers Council, Washington, DC.

1990b Peer Review: Preparing for Your Peer Review. Peer Review Program, American Consulting Engineers
Council, Washington, DC.

1990c Peer Review: Reviewer's Manual. Peer Review Program, American Consulting Engineers Council,
Washington, DC.

1990d Peer Review: Expanded Discussion of Management Issues. Peer Review Program, American Consulting
Engineers Council, Washington, DC.

Canouts, Veletta
1992 Computerized Information Exchange on the Local and National Levels in USA. In Sites &
Monuments National Archaeological Records, edited by Carsten U. Larsen, pp. 231-245. National
Museum of Denmark, Copenhagen.
| Canouts, Veletta, and Francis P. McManamon
1991 The Development and Use of the Reports Portion of the National Archeological Database. In
Automation Tools for Cultural Resource Management, edited by Fredrick Briuer. Technical Report,
WES Miscellaneous Papers. Environmental Laboratory, U.S. Army Corps of Engineers Waterways
Experimental Station, Vicksburg, MS.

Knoll, Patricia C.

1991 Clearinghouses for Archaeological Education and Looting: LEAP and LOOT. In Protecting the Past,
edited by George S. Smith and John E. Ehrenhard, pp. 193-199. CRC Press, Boca Raton, FL.

Knudson, Ruthann, and Francis P. MCManamon

1992 The Secretary’s Report to Congress on the Federal Archeology Program. Federal Archeology Report,
5(2):1,4-10.

McManamon, Francis P., George S. Smith, and Richard C. Waldbauer

1990 The Present and Future Archeological Assistance Program. American Society for Conservation
Archeology 1989 Proceedings. Portales, NM.

12



public archeology research are clarified for the aca-

Conclusion demic disciplines and therefore can be incorporated
Peer reviews for public archeology projects have appropriately into the most current theoretical and
been and will continue to be important tools for interpretive developments for explaining the ar-
agencies to use to improve both research and pres- cheological record.

ervation activities. They assist agencies in meeting
the objectives for which public archeology projects Implicit also in the purpose of this Technical Brief

are undertaken. In addition, they can help improve to provide guidance to governmental agencies and
public awareness of the value of contributions to other archeological resources management pro-
knowledge about the nation’s cultural past that are grams on the peer review process is encouragement
central to such projects. When peer reviews are for those agencies to undertake their own. Such
conducted in this comprehensive way they have peer reviews may be full-scale versions, similar to
had two important long-term results. First, the ones completed by the DCA, or they may be
agency gains a better perspective on the impacts of smaller, to fit the needs of more localized projects.
sound archeological resources management and In either case, the results can lead to cost savings and
therefore can develop effective means to improve increased support for archeology among members
its programmatic efforts. Second, the benefits of of the public and within agencies.

DCA Peer Review Reports to 1993

Aiken, C. Melvin, Jerald T. Milanich, and Douglas D. Scott. 1988, Carson Desert Archaeological Peer
Review. Report submitted to the Departmental Consulting Archeologist, National Park Service, Washing-
ton, DC.

Crespi, Muriel, and Hester A. Davis. 1989. Libby Dam Historic Preservation: A Peer Review. Report
submitted to the Departmental Consulting Archeologist, National Park Service, Washington, DC,

Dean, Jeffrey, Jefferson Chapman, and Douglas W. Schwartz. 1981. Report to Bureau of Reclamation on
the Dolores Archaeological Project. Report submitted to the Departmental Consulting Archeologist,
National Park Service, Washington, DC.

Hannus, Adrien, Terry Del Bene, and Jack Hofman. 1990. Peer Review, Alkali Creek Project, Soil
Conservation Service, Dunn County, North Dakota, Report submitted to the Departmental Consulting
Archeologist, National Park Service, Washington, DC.

Knudson, Ruthann, Frank C. Leonhardy, John S. Sigstad, and Stephen Williams. 1987. Peer Review
Committee Report for the Jackson Lake Archeological Project. Report submitted to the Departmental
Consulting Archeologist, National Park Service, Washington, DC.

Wendorf, Fred D., George J. Gumerman, and Larry Banks. 1986. Peer Review Committee Report on the
Central Arizona Archeological Project. Report submitted to the Departmental Consulting Archeologist,
National Park Service, Washington, DC.

11



Poe ees ce
Archeologists sharing information at site of Hohokam
village.

Photo courtesy Bureau of Reclamation/Tom Lincoln

project is near completion the important recom-
mendations of the peer review team will be difficult
to implement; usually time and money have be-
come short at this stage.

A second consideration in scheduling the peer re-
view is for atime that is acceptable to potential team
members. It is useful to identify a 3- to 4-week
period for the peer review approximately 3 months
in advance of the on-site visit. Within such a time
frame it is usually possible to fit the schedules of all
personnel involved. Situations that can affect the
scheduling of the review must be considered. These
include college and university calendars or special
events like conventions or athletic contests in the
project area. These situations can affect travel ar-
rangements, as lodging may be difficult to find in
such circumstances. The schedules and commit-
ments of the individuals representing the various
organizations who will be interviewed by the peer

review team must be considered. The on-site visit
must be scheduled so that those who will travel
from outside the project area, the SHPO or appro-
priate staff, ACHP personnel, and other interested
parties, can participate.

The peer review manager, in consultation with the
agency archeologist, develops an agenda for the peer
review team’s on-site visit. A detailed agenda assures
that the on-site visit is carefully structured. Each
interview focuses on a specific task or topic. Each
interview session is allotted a specific amount of
time. Sessions usually do not exceed two hours |
without a break. Maintaining strict adherence to |
the agenda assures that the peer review team stays
focused on the task and adheres to the schedule.

During the first session of the on-site visit the peer
review manager, the agency archeologist, and the
peer review team discuss the agenda in detail and
modify it as necessary. Implementing modifications
in the agenda may require complex changes in the
interview schedule. It is important to determine the
final agenda as early as possible.

6. On-Site Visit

The peer review manager and the agency archeolo-
gist plan the on-site visit to use everyone’s time
efficiently. Usually the peer review manager,
agency personnel, and peer review team arrive the
day before the on-site visit begins.

The peer review manager and the agency archeolo-
gist provide the necessary administrative support to
the peer review team. This assures that an efficient
review occurs. Meeting facilities, telephones, trans-
portation, and lodging and dining facilities must be
available. Secretarial assistance, equipment, and
supplies may be needed. The object of taking care
of these necessities is to allow the team to devote its
energies to the peer review. Personal computers
have been very useful. In providing these kinds of
administrative support, the peer review manager
and agency function as aides to the team.

The first part of the agenda is devoted to briefings
by the peer review manager and agency personnel.
The peer review manager describes and then em-
phasizes the specific scope of work of the peer
review (see Appendix C for examples). The peer
review team can examine any aspects of the project

L

ey


tribute to the research goals. They evaluate the
methodology used to get needed data and the ana-
lytical methodology. They determine if the re-
| search was conducted in an appropriate manner. In
| Federal archeology, the peer review examines addi-
tional topics as well as these. Is the agency following
the stipulations of the compliance documents
agreed to under the Section 106 process of NHPA?
Have data fulfilled the needs of the research design?
Have unexpected data or research domains come to
light that need additional investigation? Does the
project need to be changed or redesigned to take
these findings into consideration?

During the 1980s, Federal archeology project peer
reviews were undertaken when controversy stirred
about an undertaking or the archeology associated
with it. Some reviews were conducted because of
major disagreement between the agency and the
contractor about the work. Allegations or ques-
tions may have come from the archeological profes-
sion, the media, special interest groups, or members
of Congress. More recently agencies have requested
reviews by the DCA to investigate whether projects
were adequate. One review was conducted for an
agency that wished to improve a project.

In public archeology the use of the peer review
process also serves other purposes:

(1) Demonstrating to the professional community,
Congress, the Administration, and others that the
Federal archeology program produces excellent
results;

Ss

Creating networks for communication between
government and academic archeologists that
improve archeology;

(3) Helping resolve disputes between Federal
agencies, contractors, special interest groups, and
the media;

4)

Providing Federal archeologists with
professional credibility among their managers and
the academic community.

Philosophy of Peer Reviews for
Public Projects
In designing the peer review process the scientific

value of an archeological project, as determined by
the NHPA Section 106 consultation, is accepted

and used. Furthermore, the research goals and pri-
orities and the stated applications of archeological
methodology also are accepted and used, The peer
review evaluates the project using the following
criteria:

(1) Is the archeological research consistent with the
needs of the research design?

(2) Is it consistent with the contract, memorandum
of agreement, and other covering directives?

(3) Is the project on schedule?
(4) Is the project within budget?

(5) Are the agency and the contractor fulfilling the
requirements of the contract?

(6) Are there unresolved disputes or disagreements

between the contractor and the agency related to |

the archeological work?

(7) Are the materials and data recovered consistent
with the needs of the research design?

(8) Are there gaps in the data? If so, is it possible to
Pi
fill them?

(9) Is it desirable to suspend a particular line of
research?

(10) Based upon new findings, should other lines of
research be pursued in place of or in addition to
those identified in the research design?

(11) Are the public education and outreach aspects
of the project being done adequately and
effectively?

(12) Do the steps being taken to clean, catalog,
describe, and analyze excavated material conform
to the plans for ultimate, long-term curation of
the material and data?

This approach accepts previous decisions made by
the various consulting parties to mitigate the loss of
important archeological information caused by a
Federal undertaking. It also provides an occasion to
assess the project to determine if any changes are
desirable. This approach steers clear of second-
guessing earlier decisions or trying to identify all of
the approaches for conducting archeological inves-
tigations. The manager of the peer review and the
agency archeologist establish a project review
agenda and develop a list of suggested review topics.
These aid the peer review panel in examining arche-
ology defined by the NHPA Section 106 process.

|
|
|

oD


Suggested Outline for Peer Review Report,
Central Arizona Project

I. Introduction
Tl. General Evaluation
A. Research Design(s)
B. General Statements, which tie research
design and its implementation together
C. Evaluation of Performance
1, Bureau of Reclamation Management and
Administration
2. Contractor Performance
a. Field work
b. Laboratory processing
c. Analysis
d. Report writing production
3. Quality versus Cost versus Results
4, Dissemination of Program Results
a. Public
b. Scientific community
c, States
d, Other Federal land manager
5. Curation and Conservation
6. Resource Preservation
IIL. Findings
IV. Recommendations

APPENDIX D
Examples of Peer Review Report Outlines

Proposed Report Outline for
Carson Desert Report

I. Introduction
IL. General Evaluation
A. Research Design—Appropriateness
B. Translation of Research Design into Work
Elements
C. Evaluation of Program
- Bureau of Reclamation Management and
Administration
Fish and Wildlife Service Management,
Administration, and Performance
a. Field work
b. Laboratory processing
c. Analysis
d, Report writing production
. Participation by Other Organizations
a. Field work
b. Laboratory processing
c, Analysis
d. Report writing production
Quality versus Cost versus Results
. Dissemination of Program Results
a. Public
b. Scientific community
c. States
d. Other Federal land managers
6, Curation and Conservation
7. Resource Preservation
IL. Findings, a general summary of review
IV. Recommendations

bad

we

yk



Archeologists at site of Hohokam pioneer period pithouse.
Photo courtesy Bureau of Reclamation/Tom Lincoln

institution to carry out whatever archeology was
needed within the time and funding available.

Today Federal agency archeologists or contractors
from the public and private sectors conduct Federal
archeological projects. Competition among archeo-
logical contractors is the norm unless the work is
done by agency personnel. It does not matter who
does the project for the Federal peer review process
to work,

The perception has existed that Federal archeologi-
cal project research design is less open to evaluation
and criticism than an academic research project
funded by a granting agency. This is hardly the case.
As mentioned, evaluations occur as part of memo-
randa of agreement, National Register of Historic
Places nominations, determinations of effect, re-
views of scopes of work for data recovery plans, and
requests for proposals that form the bases for con-
tracts. Depending on the specifics of the project,
these evaluations are performed by archeologists,

other cultural resources specialists, and perhaps
officials of various agencies. These reviews repre-
sent a broad range of scientific and legal compliance
concerns. Frequently the evaluations improve the
agency proposal to mitigate the loss of important
information, and stipulations for formal peer re-
views are often specifically included in memoranda
of agreement or agency contracts. This is all done
before a project is announced for bidding. As part
of the proposal, the winning bidder also prepares a
research design or work plan. This provides refine-
ments for conducting the research. It also defines
the protocols that will serve as project guides,

The primary goal of the peer review process devel-
oped by the DCA is to evaluate the conduct of
Federal archeological projects and assess the com-
petence and efficiency of projects relative to archeo-
logical practice and legal compliance. An academic
peer review rates a project on its scientific merits
alone; reviewers examine the theoretical basis for
the research, the data needed, and how they con-

LL


permission to investigate the sites that fit the pro-
ject’s scientific needs, and raises the necessary funds
by obtaining a grant. The researcher plans and
conducts the work and publishes the results. Re-
view of the scientific merits of the proposed work
by his or her peers occurs during the grant applica-
tion process, in the review of the results by the
granting organization, as part of the publication
process, and in the reception of the product by the
scientific community.

On the other hand, archeology conducted to com-
ply with Federal historic and archeological preser-
vation laws must follow sound archeological
practice as well as the regulations, procedures, and
guidelines mandated by and developed from those
laws. Archeology conducted by these authorities is
to benefit the American people generally through
the preservation of important archeological re-
sources and information. The first objective of re-
search in public archeology is to identify, evaluate,
and record significant archeological resources. The
scientific information is used for a variety of prac-
tical purposes, including education and creating an
appreciation of the nation’s heritage among its citi-
zens. The concomitant benefits that individual
archeologists may derive from these endeavors with
regard to professional standing and career advance-
ment are secondary.

Early in the historic preservation process, the Fed-
eral agency identifies archeological properties
within the boundaries of an undertaking. This is
accomplished through literature reviews, records
checks, and field surveys. The significance of the
sites, especially for the important archeological and
historical data they contain, is determined based
upon eligibility for the National Register of His-
toric Places. Significance commonly is assessed in
reference to State comprehensive historic preserva-
tion plans or historic contexts used by the State
Historic Preservation Officer (SHPO).

If important data will be lost because of the under-
taking, the Federal agency must consider this po-
tential and consult with the SHPO to develop ways
to mitigate the loss. The agency also may have to
get the comments of the Advisory Council on
Historic Preservation (ACHP) before moving for-
ward with the undertaking.

Several parameters have been defined for Federal
archeological projects that constrain the scientific
freedom to develop research topics. The principal
difference between an exclusively academic re-
search design and one developed for a public project
is that the former develops from scientific curiosity
while in the latter scientific research needs are con-
ditioned by the potential data content of the archeo-
logical site being impacted by the public
undertaking.

These constraints on the opportunity for an indi-
vidual researcher to develop theoretical, methodo-
logical, or substantive aspects of the research design
do not mean that Federal projects are devoid of such
concerns. On the contrary, the process of develop-
ing a research design and a strategy to implement it,
completing the work, and reporting are the same
general steps of any scientific research project. Also,
the development of the archeological project within
Federal historic preservation is a consultative proc-
ess in the hands of Federal archeologists and their
counterparts in the SHPO office. Project plans and
archeological results also may be reviewed by
archeologists and historic preservation profession-
als of the ACHP. Frequently suggestions made
during these consultations result in design improve-
ments. As a result of this coordination, the project
has a firm scientific foundation.

Before the passage of the National Historic Preser-
vation Act of 1966 (NHPA), rescue or salvage ar-
cheology was handled without input from the
States or the ACHP. Archeological rescue work
was usually controlled by the time and money
available rather than by the development of
thoughtful and explicit research designs. The Na-
tional Park Service (NPS), operating with its own
professional staff through the River Basin Survey
program of the Smithsonian Institution or with
academic contractors, conducted the archeological
salvage program of the nation. Private energy de-
velopers and State highway departments contracted
directly with academic institutions to complete sal-
vage archeology in their construction projects.
Competition among academic institutions for gov-
ernment, private utility company, or highway de-
partment contracts for archeological work did not
exist during this period. Normally the NPS, the
utility company, or highway department ap-
proached an academic institution and asked the



|
|

APPENDIX A
Suggested Peer Review Topics from Carson Desert Archeological Project Peer Review
(Stillwater Wildlife Management Area)

© History of archeological project development

Management and administration of project

® Project planning and sequencing

@ Interagency/group cooperation and coordination

Compliance with Section 106, National Historic
Preservation Act—how handled

Maintenance of quality control

Budget and financing of archeological program,
estimate, cumulative cost, projected finishing cost

APPENDIX B
Examples of Lists of Background Material Supplied to Peer Review Teams

List of Central Arizona Project
Background Documents

1. Section 106, National Historic Preservation Act
case report
2. Central Arizona Project Overview (in press)
background and general management plan
Programmatic Memorandum of Agreement
Plan 6
5. Selected feature descriptions of archeological
projects:
a. Cave Creek feature (an early archeological
project—Survey 7)
i. Case report
ii. Memorandum of Agreement
iii. Scope of work
iv. Contract
v. Product (report)
b. Granite Reef of Salt River Aqueduct feature
i, Case report
ii. Memorandum of Agreement
ili. Scope of work
iv. Contract
v. Product (report)
c. Current feature (Waddell Dam)
i. Survey report
ii. Mitigation plan
iii. Memorandum of Agreement or PMOA
iv. Scope of work
v. Contract
vi. Progress reports
6. Research design for historic archeology

Ae

@ Status and summary of past archeological work

Public benefits

Volunteer program

Curation

Dissemination of reports to professional
community

@ Value of peer review

7. Copies of report reviews published in professional
journals, if any

8. Curation agreement—Western Archeological and
Conservation Center and Arizona State Museum
agreements

9. Copies of letters from State Historic Preservation
Officers and Advisory Council on Historic
Preservation in reference to their reviews of
mitigation activities in various features

10. Maps of Central Arizona Project

List of Carson Desert Background Documents

1. Peer Review Team
2. Preliminary Agenda |
3. Involved individuals/organizations
4. Plan of action for cultural resource management
at Stillwater Wildlife Management Area
5. Selected chapters from Preliminary Investigations
in Stillwater Marsh: Human Prehistory and
Geoarcheology, Volume 1
6. Osteological analysis of human remains removed
from Stillwater Marsh
7. Newlands Project information
a, Secretary of the Interior’s decision document
b. Operating criteria and procedures
c. Map of Newlands Project
8. Final Report on 1987 Fieldwork Conducted on
Stillwater Marsh Sites

13