The Task of the Referee
Alan Jay Smith
University of California at Berkeley

T
Computerresearchers havea professional
obligation
to referee the workof others. This
article tells you
how to evaluate apaper and write areport using
commonstandards andprocedures.

©1990 IEEE

here is an endless stream ofresearch papers submitted to conferences, journals, newsletters,
anthologies, annuals, trade journals, newspapers, and other periodicals. Many such
publications use impartial, external
experts to evaluate papers. This approach
is often called peer review, and thereviewers are called referees. Refereeing
is a public service, one of the professional
obligations of a computer science andengineering professional. Unfortunately,referees typically learn to produce refereereports without any formal instruction;
they learn by practice, by feedback fromeditors, by seeing referee reports for theirown papers, and by reading referee reports
written by others.
This article tells you how to evaluate apaper, write a referee report, and applycommon standards and procedures. It is
intended to replace Forschers rules, 1
which are distributed by some editors but
do not reflect the procedures used in computer science and engineering. This article
focuses on research papers in applied
areas of computer science and engineering, such as systems, architecture, hardware, communications, and performance
evaluation, but most of the discussion is

generally applicable; separate sections
consider research proposals and survey
and tutorial papers. Authors might findthis material useful for preparing papers
for publication. Another recent paper discusses refereeing in theoretical computersciences; there are some differences
between theory and the applied areas considered here.

The referees task
Your role as referee is to decide
whether a paper makes a sufficient contribution to the field. The contribution
can be new and interesting research
results, a new and insightful synthesis ofexisting results, a useful survey of or
tutorial on a field, or a combination of
those types. To quote a referee for thisarticle:Small results which are surprising and mightspark new research should be published;papers which are mostly repetitions of otherpapers should not; papers which have goodideas badly expressed should not be published but the authors should be encouraged
to rewrite them in a better, more comprehensible fashion.

-1-

Reading a paper as a referee is closer towhat a teacher or professor does when
grading a paper than what a scientist or
engineer does when reading a publishedwork. In the latter case, the reader presumes that the paper has been checked(refereed) and is thus correct, novel, andworthwhile. As a referee, on the other
hand, you must read the paper carefully
and with an open mind, checking andevaluating the material with no presumption as to its quality or accuracy. The
result of your reading should be a refereereport that recommends for or against
accepting the paper and lists necessary
and suggested changes.
It is important that you walk the uncertain line between being too permissive(publish everything) and being too
restrictive (nothing is good enough topublish). If you are not critical enough,you encourage poor research, recognize
and honor those who dont deserve it, mislead naive and inexperienced readers, mislead the author as to what is publishable,encourage disrespect for the field, distortcommercial development, hiring, promotion, and tenure decisions, and perhaps
actually subtract from the general store ofknowledge; consider the Piltdown man
fraud, which misled anthropologists foryears. As has been noted by Thompson3
and others, unrestrained publication buries
the professional under mounds of paper,
only a very small fraction of which can be
examined, let alone read.
If you are too critical, you block ordelay good research from publication,
waste the time of authors, damage careers,and perhaps leave journals with nothing to
publish and conferences with nothing topresent. It is particularly important not toreject new and significant work that runs
counter to the prevailing wisdom or current fashion.
If you want to be taken seriously as areferee, you must have a middle-of-theroad viewyou must be able to distinguish good from bad work, major fromminor research, and positive from negative contributions to the literature. A referee who always says yes or always says
no is not helpful.

The referee report
A good referee report should have several parts. First, you should briefly state©1990 IEEE

your recommendation and the reasons for
it. Second, you should summarize thepoint of the paper in one to five sentences,both for the editors use and to ensure thatyou actually understand the paper. Third,
you should evaluate the validity and significance of the research goal. Fourth, you

How to becomea referee
Editors are always looking for
qualified and responsiblereferees. The easiest way to
become a referee is to write apaper, thus bringing your nameand expertise to the attention of
the community. You can also
become active in professional
activities, such as local IEEE or
ACM groups, IEEE TechnicalCommittees, ACM Special
Interest groups, and conferenceorganizing committees.
Participating in these activities
will help you meet editors andprogram chairs. Also, editors
sometimes actively solicit
referees.

should evaluate the quality of the work
(methodology, techniques, accuracy, andpresentation). Finally, you must provide
an overall recommendation for or against
publication. If you recommend against
publication, you should clearly state why.
You should also make the strength of youropinions clear; an equivocal (maybe)
recommendation is acceptable if your reasons for it are clearly documented. In any
case, your report must contain enough discussion and information to justify yourrecommendation.
If your recommendation is favorable,
you must list both necessary and suggested changes. If your recommendation isnegative, but you think the paper can besalvaged and either submitted elsewhereor resubmitted, then you should provide asimilar (but perhaps less detailed) list.
Suggestions for alternate places to publish
are always welcome.
Refereeing a paper can require considerable time and effort; dont waste thateffort on a detailed critique of a badly
flawed paper that can never be made publishable. Finding one or more fatal anduncorrectable flaws excuses the referee
from checking all subsequent details.
Typically, the author receives the text of
the referee report stripped of all material
identifying the referee. Thus, while it isimportant to be clear and explicit, yourreport should not be insulting. Dont referto the author as a fool or an idiot nor
to the paper as trash. Your review
should be directed at the paper, not theauthor. The review of a proposal, though,is also a review of the investigator. In thiscase, it is appropriate to evaluate theauthors abilities as well as the research
proposed. In all cases, however, the evaluation should be objective and fair. The
more psychologically acceptable thereview, the more useful it will be.

Evaluating a researchpaper
As a referee, you must evaluate apapers novelty, significance, correctness,and readability. This general set of goals
can be broken down into a much more
specific series of questions.
What is the purpose of the paper?
What is the problem? Is it clearly stated?
Does the author make the important
issues clear? Does the author tell you
early in the paper what he or she has
accomplished? For example, if the paper
is a system description, has the systembeen implemented or is it just a design?
Is the paper appropriate? Does thispaper have anything to do with computerscience or engineering? If so, is theresearch appropriate for this forum?
(Authors should not submit papers onqueueing theory to Datamation or market
analyses of the latest MVS release to the
Journal of the ACM or the Proceedingsof the IEEE.)-2-

Is the goal significant? For that matter, is the problem real? Does it contradict any physical laws (as do perpetualmotion machines) or widely reported measurements?
Is there any reason to care about the
papers results, assuming they are correct?
In other words, is the problem or goal
major, minor, trivial, or nonexistent? Keepin mind what the Walrus said4:
The time has come,
the Walnus said,To talk of many things;
Of shoesand shipsand sealing waxOf cabbagesand kings
And why the sea is boiling hotAnd whether pigs have wings.

Is this a careful analysis of how the sea
became boiling hot or an elegant study ofthe flight characteristics of pigs?
Sophisticated mathematical analyses canbe applied to models so unrealistic that theeffort is useless and the results of no interest.
Is the problem obsolete, such as a reliability study for vacuum tube mainframe
computers? Is the problem so specific or
so applied that it has no general applicability and does not merit wide publication?
Is the problem, goal, or intended result
new? Has the design been built before?
Has the problem been solved before? Is
this a trivial variation on or extension ofprevious results? Is the author aware of
related and previous work, both recent andold? Does he or she cite that work andgive specific distinctions between it andthe current work? If the paper describes animplementation, are there any new ideas?
Is the method of approach valid? Is
there something about the approach that
invalidates the results? Can you tell what
the method is, or do you have to ferret itout from mathematical formulas? What
are the assumptions? How realistic are
they? If they arent realistic, does it matter? How sensitive are the results to theassumptions?
Is the approach sufficient for the purpose? For example, does it matter if theauthor overlooked available data and useda random-number-driven simulation withunrealistic parameters? Back of the envelope calculations are often sufficient.
If this is a new idea, does the author
present enough discussion or analysis?
©1990 IEEE

There should be neither too much nor too
little. Published archival papers are traditionally terse and complete but not cryptic; extensive and detailed discussions,along with voluminous supporting data,
are better published as technical reports.

Unrestrained publication
buries the professional
under mounds of paper,
only a very small fractionof which can be
examined.
Is the actual execution of the researchcorrect? Are the mathematics correct?
One or more referees should check themath in detail; you should always tell theeditor if you didnt read or check somepart of the paper. Are the proofs convincing? Are the statistics correct? Is the simulation methodology described in sufficientdetail to convince you that the results arevalid? For stochastic simulations, does theauthor give confidence intervals for the
results? Are the results consistent with theassumptions or with observed facts or
measurements? Have boundary conditions
been checked? Are the results plausible oreven possible? Did the author do what heor she claims? For example, did the author
simulate the original system, a reasonablemodel of it, or just the approximate mathematical model?

Does the introduction adequately explain
the problem and the research framework?
Are the remaining sections clear, and dothey follow in a logical order? Is there toomuch or too little detail? Are the grammar
and syntax correct? Are the figures andtables well labeled, legible, and meaningful? Are there too many or too few tables
and figures? Are explanations poor oreven nonsensical? Is the author too verbose or too terse and cryptic? Is the paper
sufficiently self-contained that someone
knowledgeable in the field can understandit, or does the reader need detailed knowledge of results published elsewhere? If theauthor refers the reader to other papers for
crucial details, do you believe him or her?
If sections of the paper are missing or
incomplete due to a deadline, do you
believe they will be filled in as promised?
Is the paper too colloquial or too formal instyle? Is the formalism useful or necessary? Are there many typographicalerrors? Is the paper too long? If so, does it
contain too much material, or has theauthor been too wordy? Could the paper
be split into two or more papers without
losing coherence? The paper should belong enough to present the necessary
material and no longer. Within reason, let
the editor or program committee chairworry about specific page limits.
Does the paper contain typographicalerrors or problems in grammar, punctuation, and wording? You should identify all
such problems you find. Such errors canbe a serious problem when an authorsnative language is not English. It is notyour job, however, to rewrite the paper.

Are the correct conclusions drawn
from the results? What are the applications or implications of the results? Does
the author adequately discuss why he or
she obtained these results?

What did you learn? What did you, orwhat should the reader, learn from thepaper? If you didnt learn anything, or if
the intended reader wont learn anything,
the paper is not publishable.

Is the presentation satisfactory? Is thepaper written well enough for you to evaluate the technical content? A paper that is
incomprehensible is not publishable. A
paper that requires extensive revision is
not publishable in its present form andmight never be. If the paper is readable atall, you must evaluate the presentation as
well as the technical content. (Refer toarticles such as that by Day5 to learn how
to write a paper.)
Does the abstract describe the paper?

Makingrecommendations
After comparing the paper to an appropriate standard (not your own standards,
which may be high or low), to the averageof the papers that you write, or to the average of the papers that you find worth reading, you should be able to put it into oneof these categories:-3-(1) Major results; very significant(fewer than 1 percent of all papers).
(2) Good, solid, interesting work; a definite contribution (fewer than 10 percent).
(3) Minor, but positive, contribution toknowledge (perhaps 10-30 percent).
(4) Elegant and technically correct but
useless. This category includes
sophisticated analyses of flying pigs.
(5) Neither elegant nor useful, but not
actually wrong.
(6) Wrong and misleading.
(7) So badly written that technical evaluation is impossible.
The next question is: What are the standards of this journal or conference? Is thisthe Proceedings of the IEEE, ACM
Transactions on Computer Systems, or the
ACM Symposium on Operating Systems
Principles (all quite selective), or is it the
Tahiti Conference on Beach Ball andComputer Systems (fictional, but a presumed boondoggle)? You should compare
the paper with the average paper in thatspecific journal or conference, not withthe best or worst. Of course, in some cases
the average is too low and needs to beraised by critical refereeing. Note that you
cannot tell how selective a conference orjournal is by the percentage of papers it
accepts; far fewer bad papers are submitted to the best conferences and journals.
You should then make a recommendation, whether favorable (publish) or
unfavorable (reject). Your recommendation is your opinion as to whether thepaper makes a sufficient contribution.
Generally, this will include all papers in
Categories 1 and 2 above and some in
Category 3. The strength of your recommendation should be clear to the editor(wonderful paper, definitely accept;useful paper, probably accept; marginal
paper, see how many better ones havebeen submitted; or wrong and misleading, definitely reject). If you feel that the
paper has something worthwhile to say,
but youre not sure it is good enough forthis journal or conference, you can saymaybe. You can also recommend that apaper be rejected as inappropriate for thejournal or conference. If the paper is inappropriate or marginal in quality for theforum but is suitable elsewhere, you cansuggest other places to submit the paper.
In any case, you must discuss and justify
your recommendation. A recommendation
without sufficient justification will carryvery little weight.
©1990 IEEE

If the author is asked to prepare arevised version, the revision will usually
be sent to the same referees for further
review. It is important to ensure that therevisions are satisfactory, but you should
avoid comments inconsistent with yourfirst review. You should also avoid harassing the author by unnecessarily recommending revision after revision. It is possible, however, that a revised manuscript
still contains serious problems due tothings overlooked in the first review, problems that have only become apparent afterrevision, or new errors introduced in therevision. Such problems must be
addressed. The presence of serious problems after a second revision suggests thatthe author is incapable of fixing the problems, in which case it is often appropriate
to recommend final rejection.
For a conference, a paper that requires
substantial revision generally cannot be
accepted due to the short time available
for revisions and the difficulty of arranging for additional rounds of revisions. For
journal publication, however, the extent ofnecessary revisions is a separate issue
from the recommendation for (eventual)publication.

Surveys and tutorials
Surveys and tutorials differ fromresearch papers in that most or all of thework reported is not new. Such a paper,
however, might include a variety of minor
research results that would not stand on
their own in separate papers.
Surveys and tutorials are similar but not
identical. A pure tutorial explains somebody of material to nonexperts, usuallynovices. It might not cover the entire field,
and it might have a specific point of view.
A survey provides broad and thoroughcoverage of some field or body of knowledge. It can be aimed at readers ranging
from the novice to the near-expert.
In reviewing a tutorial, there are specific issues to address: Does the paper cover
the material promised by the title or
abstract? Is this a reasonable body ofknowledge to cover in a tutorial? Is thescope too wide, too narrow, or too bizarre
to be useful? Does the paper have a consistent theme? Is the material correct? Is
the level of coverage too simple-minded

or sophisticated, given the likely audience? Is the paper well written and clear?
This last is crucial for tutorials, but journals that publish tutorials, such as
Computer and ACM Computing Surveys,
often have editors and a professional staff
to help with revisions.
For a survey, many of the same questions apply. Does the paper cover thematerial promised by the title or abstract,and is this a reasonable body of knowledge to survey at one time? Is the material
correct, and is the author sufficiently
expert on the subject to interpret resultscorrectly and provide perspective on thefield? Has the author integrated the material in a consistent manner, or is this just an
annotated bibliography? Is the authorscoverage balanced and thorough? Does heor she cite all important relevant literature,or is the presentation biased, slanted, or
unevenly selective? Controversial opinions and evaluations should be identified
as such. If the survey includes new
research results, do they meet the validity
and correctness criteria given above forresearch papers? (A survey does not haveto stand on its own as a research paper, so
the research does not have to be so significant as to justify publication as a research
paper.) Finally, is the paper well written
and clear?

Proposals
A proposal is a request to a funding
agency, company, or foundation for financial support, supposedly to do the research
described in the proposal. Reviewing proposals is quite different from reviewing
papers, and some special considerations
apply. Reviews of papers address only thescience; reviews of proposals must also
consider the investigator.
The primary difficulty with reviewing a
proposal is that the investigator is supposed to tell you what he or she plans todo, in addition to what has been done
already. The questions you must ask, then,are: Is the research topic significant? Is themethod of approach described, and is it
reasonable? Do the investigators andassistants appear to have sufficient expertise to produce useful results? Is the budget reasonable given the proposed
research, the qualifications of the investi--4-

gators, and the typical level of funding
provided by the agency in question? Are
the necessary facilities available?
The easiest way to write a detailed andspecific proposal is to propose researchthat is already complete or at least substantially underway; this approach is quite
common among established researchers.
Unfortunately, that isnt the purpose of a
research proposal, and requiring a highlevel of detail and specificity in the proposal discriminates against newcomers
and those who propose new work. Also, a
proposal might include a larger scope ofwork than can be reasonably accomplished with the time and effort specified.
This is not a negative factor if the investigators clearly recognize it and indicatethat they will choose subtopics, dependingon their interest and the availability ofassistants to work on them.
A major difference between a research
proposal and a paper is that a proposal isspeculative, so you must evaluate what is
likely to result. Therefore, when you evaluate a proposal by a well-known investigator, a substantial fraction of that evaluation should depend on the investigators
reputation. People with a consistent history of good research will probably do goodwork, no matter how sloppy or brief their
proposal. People with a consistent historyof low-quality research will probably continue in the same manner, no matter how
exciting the proposal, how voluminous
their research, or how hot the topic.
However, you must also consider the possibility that a well-regarded researcher
may propose poor research or that a
researcher noted for poor-quality work has
decided to do better work.
It is important that you do not discriminate against newcomers who have no reputation, either good or bad. In this case,
you must rely much more heavily on the
text of the proposal and such information
as the investigators PhD institution anddissertation, academic record, host institution, and comments by his or her advisor
or others.
Reviewers are asked to comment on the
proposed budget. Keep in mind that manyfactors affect the size of the budget otherthan the proposed scope of research, suchas the agency providing the funding andthe availability of facilities and staff. Also
note that for a new investigator, there is amajor difference between no funding andminimal funding (two months summer
salary plus amounts for travel, supplies,©1990 IEEEand computer time). Funding a new investigator at a low level is often a good gamble; two or three years later the investigator will have a track record and, if it is agood record, higher levels of funding canbe justified. Such small grants are often
called initiation grants and should bemuch easier to get than regular grants.

The primary difficultywith reviewing a proposal is that the investigatoris supposed to tell you
what he or she plans todo.

Other issues
Simultaneous submission, prior publication, and unrevised retries. If an
author submits a paper simultaneously to
two or more places, he or she must havethe approval of all editors or program
chairs.
All referees should also be notified.
Submitting a paper simultaneously without notification is unethical and a sufficient basis for rejection. There is a goodchance that a simultaneous submission
will be detected through the review
process.
If a paper has already been published(in conference proceedings, for example)
and is submitted for republication (perhaps in an archival journal), the editor andreferees must be notified. Some associations such as the IEEE and ACM permit
republication in their journals, but the
paper generally must meet a higher standard than if it had never been published.
Significant extensions or major revisions
are often a sufficient reason for republication. As a reviewer, you should be alert tothe author who tries to publish the same
work in all its various combinations, permutations, and subsets, and to the author
who adds the least publishable unit ofnew material to each paper. Also note that
if the first version of the paper was published elsewhere, copyright restrictions
might require the first publishers explicit
permission to republish the paper.

You will sometimes receive a paper toreferee that you have previously recommended for rejection by some other publication. If the paper has not been rewritten
to comply with your previous review, it isappropriate to return a copy of that review
along with a blunt note suggesting that theauthor comply with referee reports.
Acknowledgments and plagiarism.
Authors must not plagiarize, and theymust fully acknowledge joint work andthe contributions of others. You should
explicitly point out any such problems.
Timely response and returning apaper. You should return your report
promptly. For a conference, referee reports
must reach the program chair well before
the program committee meeting so thatthe material can be assembled and prepared for discussion. Reports received
after the program committee has met areuseless.
Journals do not generally have firm
deadlines, but preventing consideration ofa paper by taking a long time to review it
is unethical. Computer science journals
are notorious for long delays between submission and publication; the two majorbottlenecks are the referees and the publication queue for the journal itself. Imagine
if it were your paper. If you cant read thepaper in a reasonable amount of time, typically four to eight weeks, send it back to
the editor or at least get the editors agreement to the delay. Dante probably had aplace for referees who promise to doreports and then dont do so.6
Keep in mind that if you expect to haveyour own papers published, you have a
responsibility to referee a reasonable number of papers. It is part of your job as a
researcher. The option of sending a paperback to the editor should not be abused.
Editors can choose not to handle papers
by authors who dont fulfill their refereeing responsibilities.
If you are sent a paper that you are notqualified to referee, you can send it backto the editor. However, if the editor has
selected you to provide an outside viewof the field, you should provide a limited
opinion (see the section on the editorsrole).
If you are going to send a paper backwithout refereeing it, do so immediately.
Be sure to return the manuscript.

-5-

The authors reputation. Should theauthors reputations influence the evaluation of a paper, as opposed to a proposal?
There is no consensus. In my opinion, you
should consider the authors name andreputation only with regard to ambiguities,
unclear points, and references to work that
isnt presented. If the author is justifiably
well regarded, you can probably assume
that any problems will (and must) be corrected on revision. If the author has awell-earned bad reputation, you can reasonably assume that omissions and ambiguities probably represent concealed(deliberately or otherwise) errors. Because
they usually have insufficient time forrereview, conference program committees
must make assumptions about whether
problems can and will be corrected; forjournals, assumptions are generally notnecessary.
Confidentiality. In computer scienceand engineering, editors generally send
the verbatim referee reports to the author,usually as photocopies without the referees names, institutional letterheads, etc. If
you dont want to be identified, dont put
identifying information in the text of yourreport. There is no easy solution to thedelicate problem of asking the author tocite your own work without giving him or
her a hint of who you are.
Papers submitted for publication are notnecessarily public. You should neither use
material from a paper you have refereed
nor distribute copies of the paper unless
you know it has been made public, forexample, as a technical report.
Conflicts of interest. You should tell
the editor of any conflicts of interest. If
the conflict is severe, you should not referee the paper and should retum it to theeditor. For example, if you have a professional feud or a significant personal disagreement with an author, you should
send the paper back. If you are reviewing
a proposal, and you are competing withthe author for funding, you should tell the
program officer.
The opposite type of conflict also
occurs when you are asked to referee apaper written by a friend, colleague, former or current student, supervisor, subordinate, or former advisor. If you feel you
cannot provide an objective review, you
should return the paper.
©1990 IEEE

The editors role. The editor has several tasks.7 (Editor in this section refers toboth the editor-in-chief, who typicallydecides whether to accept a paper, and theassociate editors, who solicit the referee

An author who feels
insulted and ignoresreferee reports wastes aninvaluable resource andthe referees time.
reports and recommend to the editor-inchief whether to publish.) The editor
receives the paper from and maintains correspondence with the author, selects thereferees, sends them copies of the paperwith suitable instructions, and awaits theirresults. He or she should check up on
tardy referees and should find new referees if no response has been received after
a certain period.
The editor should select referees who
are knowledgeable in the subject and canbe relied on to provide a fair and objective
evaluation. Unfortunately, this is notalways possible; there are too many
papers to be reviewed and too few people
who are sufficiently expert and responsible.
There is also another problem: people
who work in area X tend to believe thatarea X is inherently worthwhile. Referees
who work in area X will usually evaluate
papers in area X by the standards of area
X; they will seldom, if ever, say that workin area X is pointless and should be discontinued. This is why an editor who
wants to debunk a paper on alchemy sends
the paper to a chemist, not an alchemist.
Someone has to say the emperor has noclothes.
After receiving a sufficient number ofreferee reports, typically three, the editor
must read them and decide whether toaccept the paper and what revisions are
required. The editor does not simply countthe referees recommendations as votes. In
theory, he or she can overrule the unanimous recommendation of the referees; inpractice, the editor can and sometimes
does side with a minority of the referees.
It is important that the referees justify
their recommendations; their reasonscount as heavily or more heavily than therecommendations themselves.
The editor must also resolve conflicting
reports and tell the authors to what extent
they must comply with the referees comments when making changes. A wise editor will also transmit copies of all refereereports to all referees, both to educate thereferees and to be fair to the author in caseof conflicting reviews.
The program chairs role. For a conference, the program chair selects referees
and collects and tallies their reports.
Typically, the program committee will
decide which papers to accept by majority
vote. The chair may or may not have alarger vote than the others, but he or she
seldom has the authority to overrule amajority of the committee. Because theyhandle a large number of papers in a very
short time, program committees usually
do not give referees and authors the personal attention provided by editors who
handle only a few papers per month. The
committees often use numerical scores toprepare ranked lists of papers; such scoresshould be assigned carefully and viewed
skeptically by the committee.

When you are the author
This article has been directed at potential referees, but instructions to referees
are also instructions to authors. Whenstarting research, writing a paper, finishing the paper, and deciding where to submit it, ask yourself: How will this paper
stand up when refereed according to thecriteria given here?
You should also consider if youre submitting your paper to the right place.
Some journals and conferences will notconsider material outside a specific scope;
why waste three months to a year to findout your paper wasnt appropriate?
Likewise, if you know your paper isminor, why send it to a highly selective
forum? Send it where it has a reasonablechance of being accepted. If you suspectthat further work will be needed before
publication, do the work before submitting
the paper; it may turn an unpublishable
paper into a publishable one, without thedelay. You can answer many of thesequestions by looking at an issue of the
publication. You should also look at the
information the journal sends to prospective authors.8, 9-6-

Keep in mind that a good referee report
is immensely valuable, even if it tears
your paper apart. Remember, each report
was prepared without charge by someone
whose time you could not buy. All theerrors found are things you can correct
before publication. All the mistaken interpretations could have been made by the
final readers. Appreciate referee reports,
and make use of them. An author who
feels insulted and ignores referee reports
wastes an invaluable resource and the referees time.
Some authors suspect that a negativereferee report indicates that the editor,
program committee, program chair, andreferees are incompetent, biased, or otherwise unfair. While this is sometimes thecase, it is the exception. There is seldom asingle correct evaluation of a paper, andequally skilled and unbiased readers will
differ. However, a set of negative refereereports is an accurate indication that yourpaper should be rewritten or reworked
before resubmission or discarded as
unpublishable or embarrassing. A reader
forms an opinion of you based on yourpaper; if your paper s quality would
reflect badly on you, you should not evensubmit it for publication.
Authors should note that Day,5 Levinand Redell, 10 Manola, 11 and Wegman 12
discuss how to write technical papers.
Refereeing is also a good way to learn towrite better papers; evaluating the work ofothers will give you insight into your own.
Scientific progress relies heavily on thepeer review processthe evaluation ofresearch for publication and funding byresearchers qualified to evaluate the work.
Referee reports are essential to thatprocess. The referees task is necessarily amatter of opinion; as a referee gains experience, the quality of the evaluation should
improve. The guidelines and instructions
presented here should be particularly useful in training and instructing novice referees.

AcknowledgmentsId like to thank Peter Denning, Domenico
Ferrari, Susan Graham, Anita Jones, Edward
Lazowska, and Ken Sevcik for their commentson drafts of this article. The opinions expressed
here are, however, my own. A number of thereferees also made useful suggestions, many of©1990 IEEE

which have been incorporated. My research(regarding which I have received many refereereports) is supported in part by the National
Science Foundation under grant MIP-8713274,
by NASA under consortium agreement
NCA2128, by the State of California under theMicro program and by IBM, Digital
Equipment Corporation, Apple Computer, andSignetics/Philips Research Laboratories.

References
1. B. Forscher, Rules for Referees, Science,
Oct. 15, 1965, pp. 319-321.
2. I. Parberry, A Guide for New Referees in
Theoretical Computer Science, SIGACT
News, Vol. 20, No. 4, Apr. 1989, pp. 92-109.
3. K.S. Thompson, Marginalia-The Literatureof Science, American Scientist, Vol. 72,
Mar.-Apr. 1984, pp. 185-187.
4. L Carroll, Alice Through the Looking Glass,1865.
5. R. Day, How to Write a Scientific Paper,
IEEE Trans. Professional Communication,
Vol. PC-20, June 1977, pp. 32-37.

Sciences at the University of California at
Berkeley, where he has been on the faculty
since 1974 and was vice chair of the department from July 1982 to June 1984. His
research interests include the performance
analysis of computer systems and devices,computer architecture, and operating systems.
He received the IEEE Best Paper Award for thebest paper in the IEEE Transactions on
Computers in 1979.
Smith is an associate editor of the ACM
Transactions on Computer Systems, is a subject-area editor of the Journal of Parallel andDistributed Computing, and is on the editorial
board of the Journal of Microprocessors andMicrosystems. He was program chair for the
Sigmetrics 89/Performance 89 Conference andis program cochair for the 1990 Hot Chips
Symposium.
Smith received the BS degree in electricalengineering from the Massachusetts Institute ofTechnology, and the MS and PhD degrees incomputer science from Stanford University. He
is an IEEE fellow and a member of the ACM,
SIAM, the Computer Measurement Group, Eta
Kappa Nu, Tau Beta Pi, and Sigma Xi.
∗Readers may write to Smith at the Computer
Science Division, EECS Department,
University of California, Berkeley, CA 94720.

6. D. Alighieri, The Divine Comedy, Cantica
1: Llnferno, 1314, translated by D. Sayers,
Penguin Books, Baltimore, Md., 1949.
7. C.T. Bishop, How to Edit a Scientific
Journal, ISI Press, Philadelphia, Pa., 1984.
8. Information for Authors, Comm. ACM,
Vol. 32, No. 3, Mar. 1989, pp. 411-414.
9. Guidelines for Authors, IEEE Software,Vol. 1, No. 1, Jan. 1984, pp. 7-8.
10. R. Levin and D. Redell, An Evaluation of
the Ninth SOSP Submissions, ACM
Operating Systems Review, Vol. 17, No. 3,
July 1983, pp. 35-40.
11. F. Manola, How to Get Even with
Database
Conference
ProgramCommittees, IEEE TC Database
Engineering newsletter, Vol. 4, No. 1, Sept.
1981, pp. 30-36.
12. M.N. Wegman, What Its Like to Be a
POPL Referee, or How to Write an
Extended Abstract so that It Is More Likely
to Be Accepted, SIGAct News, Vol. 17,
No. 4. Spring 1986, pp. 50-51.
Alan Jay Smith is a professor in theComputer Science Division of the Department
of Electrical Engineering and Computer-7-