Intuitions About Combining Opinions: Misappreciation of the Averaging Principle

Averaging estimates is an effective way of improving accuracy when combining expert judgments, integrating group members’ judgments, or using advice to modify personal judgments. If the estimates of two imperfect judges ever fall on either side of the truth, which we term bracketing, averaging must outperform the average judge. We hypothesized that people often hold an incorrect theory about averaging, falsely believing that the average of two judges’ estimates would be no more accurate than the average judge. Experiment 1 confirmed that this misconception was common across a range of bracketing rates. Experiment 2 demonstrated that the effectiveness of averaging can be recognized when bracketing was made transparent. We conclude by describing how every day life provides few opportunities to learn the benefits of averaging and how misappreciating averaging contributes to poor intuitive strategies for combining estimates. Intuitions about Combining Opinions 3 Intuitions about Combining Opinions: Misappreciation of the Averaging Principle An old saying has it that two heads are better than one. This aphorism first gained scientific support in the 1920s and 1930s when psychologists discovered that averaging individual quantity judgments led to more accurate estimates than those of the average individual judge (Bruce, 1935; Gordon, 1924, 1935; Knight, 1921). However, a debate ensued. Why did so-called “statisticized” groups outperform the average individual? Is there truly something special about groups? As one writer put it, “In every coming together of minds... [t]here is the Creative Plus, which no one mind by itself could achieve” (Overstreet, 1925, cited in Watson, 1928). The ultimate conclusion, however, was that statisticized groups revealed nothing special about groups of people, but confirmed instead the statistical principle that aggregation of noisy estimates reduces error (Eysenck, 1939; Kelley, 1925; Preston, 1938; Stroop, 1932). In hindsight, it is surprising that it took two decades of theoretical arguments and empirical demonstrations to accept this conclusion (Lorge, Fox, Davitz, & Brenner, 1958; Steiner, 1972). Why did it take so long to grasp such a simple statistical principle? The answer, we believe, is that the principle is not so simple to recognize. People lack the intuition for it, and rarely have an opportunity to learn it. As a consequence, they often reason poorly about how best to use information from others to form or revise judgments. This paper focuses on how well people recognize what we call the averaging principle: When combining uncertain quantity estimates, the discrepancy between the average estimate and the truth can be no greater than the average discrepancy of the component judgments (this is a special case of Jensen’s inequality; cf. Hogarth, 1978). To illustrate, imagine two people forecasting the high temperature in Honolulu tomorrow, which turns out to be 73o. If they guess 60o and 70o, they miss by 13o and 3o degrees respectively, or 8o on average. The Intuitions about Combining Opinions 4 average guess, 65o, also misses by 8o degrees. Thus, the average estimate performs no worse than the average judge. Now imagine they guess 60o and 80o, in which case the two estimates “bracket” the truth. In this instance, their guesses miss by 13o and 7o, or 10o on average. But the average guess of 70o misses by only 3o. Averaging outperforms the average individual (and, in this case, happens to outperform both individuals). When accuracy is measured by absolute deviation from the truth (or any weakly convex function), the averaging principle always holds. Averaging is even more advantageous if mean squared error is used, because in that case averaging outperforms the average individual even if both estimates err in the same direction. An important implication of the averaging principle is that, over multiple judgments, averaging will outperform the average judge given at least one instance of bracketing. The actual rate of bracketing is likely to be much higher, and as it increases, so does the power of averaging. Two judges who have normally distributed errors that are mean-zero and uncorrelated will bracket the truth 50% of the time. If the judges are approximately similar in skill, averaging will improve accuracy by about 29%. The bracketing rate will be lower than 50% if judges share a bias (e.g., always guessing above the truth, such as 105o and 110o in the Honolulu example), or if they have positively correlated errors (e.g., underestimating in winter and overestimating in summer). Analogously, opposing biases and negatively correlated error will result in bracketing rates greater than 50%. Extensive research in the forecasting (Armstrong, 2001; Clemen, 1989), decision making (Wallsten, Budescu, Erev, & Diederich, 1997), and groups (Einhorn, Hogarth, & Klempner, 1977; Gigone & Hastie, 1997) literatures has confirmed that averaging is a powerful and robust way of reducing error in judgment. Although applied statisticians originally treated averaging as a baseline against which to compare more sophisticated combination methods, the baseline proved surprisingly difficult to beat (Clemen & Winkler, 1986; Fildes & Makridakis, 1995). In an extensive review, Armstrong (2001) reanalyzed Intuitions about Combining Opinions 5 thirty studies conducted between 1960 and 2000. Across the studies, averaging improved forecast accuracy from 3.4% to 23.5% relative to the mean performance of the forecasts being averaged, with a mean improvement of 12.5%. Averaging itself is not critical to improving judgments. Weights in rough proximity to whatever weights are analytically optimal yield similar levels of improvement (von Winterfeldt & Edwards, 1973, as cited in Dawes, 1979). For example, if the appropriate weighting scheme is 50/50, using a 70/30 split will be nearly as accurate. However, our own empirical studies indicate that laypeople frequently do not combine at all, but instead choose between estimates (Soll & Larrick, 2003). In one study, participants first estimated salaries for graduates of 25 business programs. They were paid based on their revised estimates after viewing the responses of another participant. Although participants reduced error by 10% with their intuitive revision strategies, they would have improved by 16% had they consistently averaged. Because participants used extreme weighting schemes of 100/0 or 0/100 about half the time, they often missed out on the benefits of combination. It could be that people avoid averaging because they are confident in their ability to identify the better judge. Such confidence may be misplaced but it upholds the romantic notion that one should strive for the very best, and not “settle” for averaging (cf. Kleinmuntz, 1990). But a second reason people might not average is that they simply do not recognize its effect. Specifically, we propose that many people mistakenly believe that averaging leads to average performance. Partly this belief reflects the application of an incorrect mathematical rule. However, individualistic norms in the West that equate compromise with mediocrity may also be a contributing factor. This paper presents two experiments that examine people’s beliefs about the effect of averaging judgments on reducing error. The first experiment tested people’s understanding of averaging as an abstract principle. The second experiment investigated whether people could induce the benefits of averaging by observing sets of data. Of critical interest, we manipulated Intuitions about Combining Opinions 6 the bracketing rate, or the relative frequency with which the estimates of two judges fall on opposite sides of the truth. Averaging is more effective to the extent that the bracketing rate is high. In general, we expected that the averaging principle would be difficult to grasp (the dozen empirical papers on averaging between the 1920s and 1940s are a testament to this difficulty!), but that the benefits of averaging would be more easily recognized when the bracketing rate is both high and salient. Experiment 1 Assuming at least some bracketing, do people hold the abstract principle that averaging must outperform the average judge? We manipulated the bracketing rate for two hypothetical judges by varying correlation in error across three levels: Positive correlation (24% bracketing), weak negative correlation (58%), and strong negative correlation (90%). A supplemental condition included judges with opposing biases (90% bracketing). We predicted that many participants would equate the accuracy of averaging judgments with the accuracy of the average individual judge. We expected, however, that people would increasingly recognize the effectiveness of averaging as the bracketing rate increased. Method Participants. Participants were 145 masters of business administration (MBA) students enrolled in a statistics course at INSEAD. The population is mathematically sophisticated; the median score on the quantitative section of the GMAT was in the 94 percentile. Materials. Participants read the following scenario: Ms. A and Ms. B are currency analysts at two banks. On the first of every month they have the task of forecasting the yen to dollar exchange rate for the following month. The banks use these forecasts in deciding their currency positions. The two banks have decided to merge and now must decide how to make best use of Ms. A and Ms. B. To help with this decision, the new combined bank has analyzed the past forecasts of Ms. Intuitions about Combining Opinions 7 A and Ms. B for the fifty months prior to the merger. There is no evidence that the accuracy level for the forecasters has been changing over this period. The concept of mean absolute deviation (MAD) was explained, using specific illustrations, after which the MADs of Ms. A (4.7) and Ms. B (5.3) were presented. Participants were told that the banks also tracked the frequency