Citation Indexing and Evaluation of Scientific Papers
Authors: Margolis J. 
Published: October 2006

Evaluation by means of citation patterns can be successful only insofar as published papers and their bibliographies reflect scientific activity and nothing else. Such an innocent description is becoming less and less tenable. The present scientific explosion gave rise to more than a proportional publication explosion, which not only reflects the scientific explosion but has its own dynamics and vicious circles. Publication of results is probably the main means of accomplishing the almost impossible task of accounting for time and money spent on research. Inevitably, this puts a premium on quantity at the expense of quality, and, as with any other type of inflation, the problem worsens: the more papers are written, the less they count for and the greater is the pressure to publish more. What makes matters worse is the fact that the sheer volume of the "literature" makes it increasingly difficult to separate what is worthwhile from the rest. Critical reviews have become somewhat of a rarity, and editorial judgment is usually relegated to referees, who are contemporaries and, perhaps, competitors of the authors—a situation which has its own undesirable implications. It requires little imagination to discover other vicious circles, all arising from distortion of the primary reasons for publishing the results of scientific inquiry.

There are, it is true, signs of adjustment to this crisis, partly due to some easing of the pressure to publish at all costs, and partly due to the readers' changing attitudes toward the flood of publications. An increasing amount of research is now being carried out in the form of collective projects in large institutions where publication is no longer the standard method of accounting for individual work. At the same time there is apparent an increasing tendency for scientific journals to polarize into the relatively few leading ones which carry important information and the many subsidiary journals which serve as vehicles for interim local accounting and, in a way, substitute for detailed intradepartmental reports. This division is a result not of some arbitrary decree but of normal competition between journals, as a result of which, however, the strong usually get stronger and the weak get weaker. Were it not for these changes and also for a striking improvement in abstracting, indexing, and alerting services, most research workers would have found long ago that, even in their own specialized fields, new information is accumulating faster than it can be sorted out. These developments can provide only a temporary reprieve, so long as there remains a strong incentive to publish the greatest possible number of papers. A new scale of values based on citations is by no means infallible or, in many cases, even fair, but at least it provides an alternative to the existing one, which is at the root of the crisis.

It might, of course, be asked whether wide acceptance of such new standards would not lead to deliberate abuses. A little reflection shows that the system is less open to manipulation than might appear. First, the referees are expected to see to it that the submitted papers cite work which is pertinent to the subject. An increased awareness of the usefulness of citation indexing as a tool for retrieval and evaluation will make this aspect of refereeing more important, and what now passes for minor carelessness or discourtesy could easily come to be regarded as serious malpractice. Second, as noted above, careful selection of references is in the author's own interest, because it helps him to reach his readers. There is, therefore, some room for hope that healthy feedback in the system will tend to keep it viable. At the basis of this hope lies the supposition that, in the long run, only good work can ensure recognition.

As Martyn has pointed out, as an information-retrieval method, citation indexing is rather "noisy." The word
noisy
may apply even more to the problem of evaluation. Whereas in information retrieval much of the unwanted information can be filtered out by suitable search strategy, this is not so easy to do for the purpose of evaluation, because a simple descendence relationship between papers is still an ideal far removed from actuality. The situation would be much better if we could at will exclude all citations which do not indicate real indebtedness. A scheme of citation relationship indicators, first mentioned by Garfield and elaborated by Lipetz, would be a help, but, even if it were technically feasible, to provide such indicators would greatly add to the production costs of the
Index
.

Another possible way to minimize the effects of "noise" is to increase the size of the samples on which the reckoning is based. Now that research has become a rather popular occupation, it seems that a kind of public vote may have to be accepted as a factor in evaluation. Since this is the case, there is something to be said for extending the "franchise" to minimize accidental effects. An index which attempted to process all scientific publications would be several times the size of the present
Index
, and, what is more, it would not necessarily be an improvement as a tool for information retrieval because most of the significant work is already concentrated in the present
Index
. Whether this attempt will ever be considered worthwhile remains primarily a matter of policy and economics. In the meantime there is an urgent need for more experience with the existing services.

It is not the purpose of this article to advocate evaluation of scientific work by some kind of public opinion poll; its purpose is to recognize a possible trend in this direction. Any judgment by public acclaim is subject to obvious fallacies, but we must not be carried away by the analogy to the Stock Exchange or to electoral practices. The fact that, in this case, the "public" consists of authors whose contributions are generally linked creates quite a new pattern of organization. In this discussion some of the aspects of this pattern have been explored through analogy to idealized genetic or mechanical network models, but the very uniqueness of the system, with its many self-organizing ramifications, makes it a new field which deserves close study, since these developments may have profound effects on the future of scientific communication.