Rating, not ranking.

Doctorate programs at academic institutions are not susceptible to any simple, one-dimensional description. The recent report of the National Research Council (1) sought to recognize-even emphasize-this in its presentation of data, by presenting ratings and data on a large number of aspects of the programs, including reputational ratings, citations, and program size, each measured in several different ways. These measures are spread over the 700-plus pages of the report, but nine of the more interesting are presented by field in Appendix Tables P-1-P-41 (pp. 469-624). Two strong messages emerge from these tables: first, that the different measures lead to different linear orders and second, that the different linear orders are not greatly different. These messages should be reassuring; they lend support to the view that there is no such thing as a single linear ordering or a single best program in a field, while suggesting that the measures are nonetheless capturing related aspects of a real attribute of these programs. Should we be surprised at this? And what is that "real attribute"? Some critics of studies such as the NRC report maintain the measures should not be given weight because they are "merely reputational surveys." Leaving aside the fact that the NRC study included several nonreputational measures, a reply to this charge is readily at hand: academic reputations are not to be lightly dismissed. Whether it be a single scholar or a program faculty, reputation for academic scholarly work is subject to constant (and often highly critical) evaluation in all active research centers. Scholarly reputation is not mere gossip. It is subject to substantial limitations, to be discussed later, but it is the product of a great deal of deliberation and constant reassessment, and it should come as no surprise that different ways of using measures related to reputation should be in substantial agreement. Reputation is a real attribute with real consequences for a program. Reputation, no matter how assessed, does not-cannottell the whole story. It does not capture the internal value of a program to a university's curriculum, and it only refers to sets of faculty, not to individuals. Any program will consist of several faculty, and raters will, no matter how they are questioned, bring different perspectives to bear on the question of how to weigh the whole as a sum of the parts. One of the Committee's hopes in the planning for the 1995 NRC report was that the results of the survey would shed light on this question. For example, is academic reputation of programs more influenced by that of the best known members of the program, or by the presence of a larger number of less well-known faculty? How does a program that is superb in one narrow specialty compare to a program with a more diverse but less accomplished faculty? The survey was planned with the possibility of addressing such questions in mind, but it was insufficiently funded to permit the needed analysis. Furthermore, the NRC report considered only domestic United States reputational ratings. Early on a plan was drawn up to incorporate a survey of international raters, but that too was eliminated because of a lack of funds. Saunders Mac Lane (2) is worried that the raters' judgments were expressed on a discrete scale ("Distinguished," "Strong," etc.) and that combining these in terms of numerical scores may mislead or at least violate the raters' original intent. An examination of the study is reassuring on this point. The report's Appendix Tables P1-P41 include among their measures the percent of raters who judged a program as "Distinguished" and the percent of raters who judged a program as "Distinguished" or "Strong." The agreement in the ordering based on these measures and that based on a numerical average is remarkable, except for the lower rated programs. Mac Lane also is concerned that different raters will be unequally knowledgeable, and that the results may thus be too susceptible to the luck of the draw-to sampling variation. Again the tables of the report are reassuring. Tables P1-P41 give an average quality rating based on all raters, and a second rating where the average is computed using only the raters from the top 50% of the programs as rated by all raters. Now, the selection of raters was not a full random sample of program faculty; about 53% were selected at random from faculty lists and the remainder were chosen from lists of names submitted by graduate deans or provosts of people most likely to be familiar with other institutions' programs. The raters are thus likely to be more knowledgeable than a random sample of faculty. Even so, one would expect that raters at the top 50% of rated programs would be more active in research, and hence likely to be more knowledgeable of their colleagues' work at other institutions. The results of these two measures do differ, but only slightly (usually by no more than four positions in rank order). The study planners had hoped to be able to investigate the differences found; for example, they might reflect a greater sensitivity to recent changes in faculty at top schools. Again, a lack of funding curtailed the analysis. Despite these reassurances, the 1995 report is not without problems. Some of these problems were a consequence of the necessity of dealing with an extraordinarily broad constituency. For example, the raters depended on the lists of program faculty provided by the institutions' provosts or graduate deans. In some cases, a part of a program's faculty was inadvertently omitted and, in a very few instances, notable departments were omitted entirely due to blunders on the part of a graduate dean. Also, there were errors of commission as well; in one case, an energetic dean included two programs in the same field with substantially the same faculty list to capitalize upon the strong reputation of one of them. Other problems, problems of interpretation, are inherent in any such enterprise that deals with units that are themselves aggregates of often diverse sets of specialties. Only one set of ratings is provided for each program-what if a program achieves a level of excellence in one subfield but is mediocre in others, how will their ratings be affected by the fact that all parts are rated together? There is no general answer. If the subfield is one that is known widely in the field, its excellence may lift the entire program to the level of the elite. If the subfield is followed only by its own specialists, the program will not fare well in the ratings. This does not mean that it is reasonable for academic administrators to ignore the ratings, but it does highlight the need to interpret the ratings with care and in the light of local knowledge. No matter how superior a university is in subfield X, if that excellence is only apparent to the few who work in X and not to the broader field, that limited reputation may carry no collateral benefit to the