THE HEBREW UNIVERSITY OF JERUSALEM DOES DECISION QUALITY (ALWAYS) INCREASE WITH THE SIZE OF INFORMATION SAMPLES? SOME VICISSITUDES IN APPLYING THE LAW OF LARGE NUMBERS by KLAUS FIEDLER and YAAKOV KAREEV

Adaptive decision-making requires that environmental contingencies between decision options and their relative advantages and disadvantages be assessed accurately and quickly. The research presented in this article addresses the challenging notion that contingencies may be more visible from small than large samples of observations. An algorithmic account for such a "less-is-more" effect is offered within a threshold-based decision framework. Accordingly, a choice between a pair of options is only made when the contingency in the sample that describes the relative utility of the two options exceeds a critical threshold. Small samples – due to their instability and the high dispersion of their sampling distribution – facilitate the generation of above-threshold contingencies. Across a broad range of parameter values, the resulting small-sample advantage in terms of hits is stronger than their disadvantage in terms of false alarms. Computer simulations and experimental findings support the predictions derived from the threshold model. In general, the relative advantage of small samples is most apparent when information loss is low, when decision thresholds are high, and when ecological contingencies are weak to moderate. 3 Sample size and decision quality Does Decision Quality (Always) Increase with the Size of Information Samples? Some Vicissitudes in Applying the Law of Large Numbers In cognitive psychology and decision-making research, amount of information is considered to be a function that cannot take negative values. Apparently, adding information can only increase its overall value, implying a monotonically increasing function. Psychologists implicitly agree with economists that the utility of information cannot decrease, just as the value of other resources (e.g., money, property, power, legal rights) can never decrease when its amount increases. To be sure, the function that relates usefulness or total value of information to the amount of evidence may be negatively accelerated, and level off asymptotically; however, the slope of the function is assumed to remain non-negative. For example, consider the typical form of a learning curve (see Figure 1). Performance increases monotonically with the number of learning trials, though the increment gets smaller as learning approaches an asymptote. The reason for concavity is easily understood. The increment that each trial adds is maximal at the beginning when there is little prior learning, when each trial is likely to contribute something new; however, as learning proceeds, the likelihood increases that further trials only reiterate what is already known. The concavity and the constantly positive slope would thus appear to be quite natural. Note that such a function is characteristic of information acquisition in general, and is not peculiar to human or animal learning. A similar function results when the overall reliability of a test is depicted as a function of test length, or number of items included (Kuder & Richardson, 1937). The same rationale applies to the accuracy of judgments or decisions as a function of the number of independent judges (Rosenthal, 1987). Increasing the number of test items or judges leads to the canceling out of errors, and a reduction in error with a corresponding increase in the proportion of systematic variance in the aggregate test score or judgment. In general, the function relating accuracy of predictions to sample size often resembles that of Figure 1. 4 Sample size and decision quality Indeed, the logic underlying Bernoulli's law of large numbers appears so natural that we can hardly imagine that the monotonicity of information value might be questioned. Thus, if some researcher came up with data showing better learning following k trials in one experimental condition than following 2k trials in another, one would hardly postulate a new learning curve but try instead to find out what was wrong with these data: The two groups may differ in talent or motivation, or the researcher may have fallen prey to some artifact or uncontrolled factor such as fatigue arising from extended learning. Similar suspicion would arise if a psychometrician were to claim that a score based on a random subset of test items was more valid than a score based on all items, or if teachers were reported to discriminate more accurately between smart and poor students when they based their appraisal on a small rather than a large number of observations. Can Less be More? However, is the possibility that less information can be worth more or that decisions based on small samples can be superior to those based on larger ones really that far-fetched, or incompatible with logical and axiomatic underpinnings? Indeed, there is compelling anecdotal and experimental evidence that knowing or thinking too much can be of disadvantage. Just as the penalist in soccer should not dwell too long on where to kick the ball, consumers are typically most satisfied with their product choices when they make quick gut decisions based on minimal information. Additional product information can turn decisions into a torture and increase the likelihood of post-decision dissonance (Festinger, 1957). Or, to give another example, for an electronic search, an output of, say, 50 references is more informative and useful than one of 5000. Relevant research to back-up such anecdotal experience includes Wilson and Schooler's (1991) findings of impaired decisions with extended thinking, Forest and Feldman (2000) demonstration that lie detection performance decreases with amount of reflection, or increasing conflicts experienced when judgments are based on a large rather than small amount of evidence (Fiedler, Semin, Finkenauer & Berkel, 5 Sample size and decision quality 1995; Sande, Goethals & Radloff, 1988). Ambady and Rosenthal's (1992) work on the amazing diagnosticity of "thin slices" of behavioral evidence is also relevant in this respect. Being confronted with such examples, one may still be inclined to dismiss the evidence regarding the benefits of scarce information as hardly bearing on the axiomatic view depicted at the outset. One may dismiss the performance impairment sometimes observed with high amounts of information as "just" due to cognitive overload, the inadequate administration and organization of resource-limited cognitive procedures applied to large arrays of information. Or one might reason that the examples reflect cases in which implicit, unconscious decisions are more appropriate than explicit, reflected decisions (Wilson, Dunn, Kraft, & Lisle, 1989). However, again, the spontaneous tendency to discount such counter-intuitive findings or reconcile them with the axiomatic view may not be warranted. Recent research on adaptive cognition has revealed a number of less-is-more effects (Borges, Goldstein, Ortman, & Gigerenzer, 1999; Gigerenzer & Goldstein, 1996; Hertwig & Todd, 2003; Krauss & Wang, 2003; Martignon & Hoffrage, 1999) that cannot be reduced to inefficient cognitive procedures, as they also occur in computer simulations based on a fully rational, unbiased program. Another noteworthy phenomenon is evident in Elman's (1993) demonstration of the "importance of starting small". Using computer simulations of artificial language learning, Elman observed superior learning of a complex language system when the window size of the input utterances, or the short-term memory window of the learning individual, was limited rather than expanded in size in the early stage of learning. Allowing for more information in early learning impaired subsequent learning. Similarly intriguing evidence for less-is-more effects in language acquisition has been reported by Newport (1988, 1990). Sample Size and Contingency Assessment Our prime example, which is in the focus of the present research, refers to contingency assessment. Detecting environmental contingencies between signals and significant events, between causes and effects, or between situations and behaviors is a central module of 6 Sample size and decision quality adaptive intelligence. As Kareev (1995, 2000; Kareev, Lieberman, & Lev, 1997) has shown, because of the skewness of the sampling distribution of correlations, and the fact that the skew is more pronounced the smaller the sample, the likelihood of detecting a contingency may, under certain conditions, be higher when the sample of observations is small rather than large – a phenomenon that we will elaborate on shortly. Again, one would assign almost axiomatic status to the assumption that accurate assessment of correlations increases with sample size. After all, sample size corresponds to number of learning trials, and additional trials can hardly reduce learning (cf. Figure 1). Imagine an animal exposed to the contingency between territories and danger. Choosing the less dangerous territory is crucial for survival in the long run. Is it not obvious that the probability of correct choices increases with the number of opportunities to learn about the environment? How could the animal ever be expected to reach a better decision with a smaller, rather than a larger sample of experience? An intriguing answer to this paradox can be derived from a statistical principle that governs all probabilistic ecologies. When repeated samples of a certain size are drawn from a universe in which two variables (e.g., territories and danger) are actually correlated, the resulting distribution of sample correlations r around the "true" value ρ is not symmetrical but skewed. Most sample correlations are higher than the correlation in the universe. Thus, one of nature's nice features is that empirical snapshots often amplify actual effects. Even more amazing, this tendency is more pronounced for small samples. Small samples are thus particularly likely to accentuate and to expose the contingencies that actually exist in the world (Kareev, 1995). Furthermore, Kareev (2000) has shown that