Are surveys of experts unbiased? Evidence from college football rankings.

I. INTRODUCTION There is a voluminous literature dedicated to determining the aptness of expert performance (Shanteau 1992), as well as uncovering the premier methods for eliciting their collective opinions (Otway and von Winterfeldt 1992). Additional research is present on when betting markets might outperform experts or vice versa (Sunstein 2006). Of principle importance in all these literature streams is the concern over various forms and sources of bias that could undermine the validity of these judgments. College football polls, which are weekly rankings of the "best" college football teams provided through tallying votes from coaches and specialized journalists, has served as an experiment in much of this literature on bias (Avery and Chevalier 1999; Fair and Oster 2007; Paul, Weinbach, and Coate 2007). These rankings represent the aggregate subjective judgment of groups of "experts," and their judgments are tested on a weekly basis as these teams compete against one another. The purpose of this article is to examine those judgments for evidence of any systematic biases. Evaluating college football polls for bias, as well as their overall accuracy, might provide some insight into how similar polling could be implemented for policymakers interested in ranking alternative proposals. There has also been much interest among academics in the possibility of using prediction markets to inform public policy decisions. For instance, if policymakers were interested in increasing economic growth or reducing income inequality, they could potentially establish a betting market where the outcome was determined by some observed change in measured results, conditional on alternative policy proposals. Theoretically, the betting market would generate a price or line that was reflective of the aggregate beliefs of the population regarding the ability of the proposed policy to actually accomplish the objective(s). For this reason, betting on sports has been studied extensively as an experiment to inform what might become of policy betting markets. For instance, there seems to be a "long-shot" bias in betting markets (Wolfers and Zitzewitz 2007). However, if the polling of experts is considered an alternative approach, then it is important that it is similarly studied for bias. This article analyzes both the American Football Coaches Association (AFCA) and Associated Press (AP) college football polls for several possible forms of bias. The AP poll surveys college football journalists selected by a committee in each state, and the AFCA poll represents the survey of head football coaches representing each conference. (1) Although the focus of this article is merely on uncovering the existence of a bias rather than the underlying causal cognitive or survey process, the variables tested for bias are motivated by the existing literature on expert and cognitive bias. For example, a school with a highly regarded journalism program may receive some sort of favorable treatment from polled journalists via the affiliation bias. Similarly, schools from a particular conference may benefit or suffer as a result of pollsters relying on simplified informational heuristics. To test for bias, regular season games from 2003 to 2008 in which at least one team had been ranked in the top 25 are compiled into a data set. With more than 1,300 observations in both polls, the higher ranked team won the game 79.2% of the time in the AP poll, as compared to 78.4% in the AFCA poll. When both teams were ranked, the higher ranked team still won the game 66.67% of the time in both polls. A probit regression is employed to determine which characteristics and attributes determine the probability that the higher ranked team wins the game. Ideally, the aggregate results of the poll should reflect all relevant information regarding the quality of teams, while the error should be random and not correlated with any other attribute like conference affiliation, or having a quality journalism school. …