Rain, Snow, and Sleet are Just Different Types of Precipitation

n "The Rain Man Cometh?Again" Robert Zemsky takes shots at the rankings of "America's Best Colleges" published for the last 25 years by U.S. News and World Report. The article is well written and quite clever, and takes us on a history of how those rankings evolved over time. We are told how the rankings have been attacked over the years and how they have led to a process described by the author as an "admissions arms race" (p. 6), and we are eventually provided some data that was collected to try to determine "ex actly what the rankings measure" (p. 9). Zemsky concludes that, in fact, the rankings do not assess program quality but rather something he defines as market position. While the U.S. News rankings miss the mark, says Zemsky, other rankings, such as those from BusinessWeek, come closer to what is needed because they try to assess customer satis faction, which Zemsky suggests is a better measure of the learning environment, which is what we needed in the first place. I was interested to see that the article focused primarily on the ranking of undergraduate pro grams. Many members of the Academy, and surely all the deans of business schools, are more inter ested in rankings (from various sources) of MBA programs. I believe that the issue with these latter sets of rankings is much the same as those identi fied by the author, and so I will focus my attention on rankings of business schools, and especially of MBA programs, and try to expand on some of the points raised in the article. As the dean of a school of business, I protest scores we receive that I disagree with; I cheer accomplishments (i.e., mov ing up), and cry foul when we drop down in someone's rankings. As a dean, the rankings "game" angers me at some level, but as a scholar interested in measuring and managing perfor manee, I have an entirely different view of the rankings. In 1949 Robert Thorndike introduced the notion of "the criterion problem" (Thorndike, 1949). I believe that understanding the criterion problem will help us understand what we really don't like about business school rankings. The problem is actually simple. As stated in Zemsky's paper, "the rankings can not possibly measure what they propose to mea sure?that immeasurable quantity: academic qual ity" (p. 5). In the terminology of criterion theory, then, academic quality would be considered the "ultimate criterion." Thorndike suggested that, in all cases, the ultimate criterion is a hypothetical construct that should guide our efforts to find measurable criteria, but cannot itself be measured. Thorndike went on to state that instead, we must rely on what he termed the "actual criterion"? the thing we can and do measure since we can't measure the ultimate criterion. But the actual criterion is always less than perfect, and the rela tionship between the ultimate criterion and the actual criterion can be viewed in a Venn diagram, as shown in Figure 1. Always, there will be aspects of the ultimate criterion not captured by the ac tual criterion; this is referred to as criterion defi ciency. There are also some things that are cap tured by the actual criterion that are not really part of the ultimate criterion; this is referred to as criterion contamination. Thus, there will always be criterion contamination and criterion defi ciency present with any criterion measure (objec tive or subjective); the key is to minimize both by maximizing the portion of the ultimate criterion that is captured in the actual criterion?criterion overlap. These concepts are best captured in the illustration (from Blum & Naylor, 1976) in Figure 1.