Small N's and Big Conclusions: An Examination of the Reasoning in Comparative Studies Based on a Small Number of Cases

An increasing number of studies, particularly in the area of comparative and historical research, are using the method of agreement and method of difference proposed by Mill (1872) to infer causality based on a small number of cases. This article examines the logic of the assumptions implicit in such studies. For example, the research must assume: (1) a deterninistic approach rather than a probabilistic one, (2) no errors in measurement, (3) the existence of only one cause, and (4) the absence of interaction effects. These assumptions are normally inappropriate, since they contradict a realistic appraisal of most social processes, but are mandatory if wefollow Mill's causal analyses based on small N's. Research should not attempt employment of such methods in smallN cases without a more rigorous justification of heroic assumptions and a guard against possible distortions. This article evaluates a sociological approach that is gaining in usage, especially in researching historical and comparative problems. Namely, we will consider the causal inferences drawn when little more than a handful of nations or organizations sometimes even less are compared with respect to the forces driving a societal outcome, such as a political development or an organizational characteristic. Application of this method to a small number of cases is not new to sociology, being in one form or another a variant of the method of analytic induction, described by Znaniecki (1934) and analyzed succinctly by Robinson (1951) and Turner (1953). 2 The conclusions rely on a formalized internal logic derived from Mill's (1872) method of agreement and his method of difference (see the discussion of Mill in Nichols [1986]). The formal rigor of this type of analysis sets it off from other small-case procedures that also imply causality, i The University of North Carolina Press Social Forces, December 1991, 70(2):307-320 * I am indebted to Williatn Alonso, Rogers Brutbaker, John Campbell, William Kruskal, and Peter V. Marsdenfor stimulating discuissions or comments on this topic. A modified version of this article will be reprinted in What Is a Case, edited by Charles C. Ragin and Howard S. Becker, Cambridge University Press, 1992. Direct correspondence to Stanley Lieberson, Departmnent of Sociology, Harvard University, Cambridge, MA 02138. 308 / Social Forces 70:2, December 1991 such as in Street Corner Society (Whyte 1943) or in the development of the urban structure and growth model of Burgess (1925). It is also different from case studies that seek to point out merely that a given phenomenon exists in some setting, as opposed to analyzing its causes. The comments are, however, to some degree relevant in evaluating the Boolean method proposed by Ragin (1987) for dealing with somewhat larger samples used in comparative and historical research. Moreover, although the analysis is stimulated by recent developments in macrohistorical research, it is pertinent to a wide variety of other studies that use Mill's logic with a small number of cases. There are numerous instances of these small-N studies, which apply Mill's methods to infer causality. For example, Moore (1966) uses them in his analysis of the role of various classes and peasantry leading to three different outcomes in seven different nations. Skocpol's (1979) influential study of social revolutions in a small number of countries also uses these methods and attributes their application to de Tocqueville and Bloch. Katznelson (1985) explains "the puzzle of variation between English and American patterns of working-class formation" (260) by eliminating any common features shared by the two nations, searching for a distinctive difference that must account for these developments. Likewise, Stepan (1985) employs similarities and differences between Argentina, Brazil, Chile, and Uruguay to infer relations between 'the power of the state and the power of civil society" (318). Orloff and Skocpol (1984) compare welfare developments in the U.S. and Britain around the turn of the century, eliminating major conventional theories of welfare-state development because the independent variables are considered the same in both nations. Only one variable is found to differ between the two nations and causal significance is therewith assigned. More generally, we are dealing with any study that attempts to infer the existence or absence of causality through the correlation between the dependent variable and an independent variable across a small number of cases. When the cases differ in outcomes, then it is assumed that the causal variable will have a correlation of ? 1.0 with the dependent variable, with all other correlations having other values. When the cases have the same outcome, then it is assumed that the causal variable will be the same for all cases, with each of the other variables having some variability across cases. Even the classic study by Weber (1930) of the role of the Protestant ethic in the development of capitalism can be used as an illustration of the method of difference. A capitalistic system is observed to develop in only one part of the world. Weber then asks what variable is found in the Protestant West but nowhere else. All other variables, which the countries share, are eliminated as the necessary causal variable. One can readily understand why the logic is appealing despite Mill's (1872) warning that it is inappropriate for social science problems. He specifically illustrates the inapplicability of the method of difference by describing differences between nations, and he observes the weakness of the method of agreement for the typical social science problem as well. As Tilly (1984) observes, the presence of more than one common causal variable makes it impossible for the latter method to sort out the actual cause. We can easily appreciate the goal of making causal inferences based on a small number of cases. If data were available with the appropriate depth and Small N's and Big Conclusions / 309 detail for a large number of cases, obviously the researcher would not be working with these few cases (assuming a minimal time-energy cost). Since the data are not available, or the time-energy cost is too great, one can only approach these efforts with considerable sympathy for their objective. We address three questions: (1) What are the assumptions underlying these studies? (2) Are these assumptions reasonable? (3) What can we do to improve such studies in those instances when they might be appropriate forms of inquiry? However, there is a special difficulty in comprehending the weakness of these applications and the gross assumptions that are necessarily made; namely, the methods are typically applied to topics for which the reader has no experiential grasp. Readers cannot easily see the unreasonableness of a conclusion about macrosocietal events in a distant time and place. To display the nature of this logic, this article draws illustrations from hypothetical applications to automobile accidents and the evaluation of airlines. Probabilistic and Deterministic Perspectives Let us start by distinguishing between causal propositions that are deterministic as contrasted with those that are probabilistic. The former posits that a given factor, when present, will lead to a specified outcome. The latter is more modest in its causal claim, positing that a given factor, when present, will increase the likelihood of a specified outcome. When we say, "If X1, then Y," we are making a deterministic statement. When we say, "The presence of X1 increases the likelihood or frequency of Y," we are making a probabilistic statement. Obviously, if given the choice, deterministic statements are more appealing. They are cleaner, simpler, and more easily disproved than probabilistic ones. One negative case (Y's absence in the presence of X1) would quickly eliminate a deterministic statement. Alas, a probabilistic approach is often necessary to evaluate the evidence for a given theoretical perspective, even if we think in deterministic terms. This occurs for a variety of reasons, not the least being measurement errors a serious problem in the social sciences. The existence of a measurement error means that a given data set may deviate somewhat from a hypothesized pattern without invalidating the hypothesis. In addition to this technical matter, there is an additional problem: complex multivariate causal patterns operate in the social world, such that a given outcome can occur because of the presence of more than one independent variable and, moreover, may not occur at times because the influence of one independent variable is outweighed by other influences working in the opposite direction. Under such circumstances, the influence of X1 is only approximate (even without measurement errors), unless one can consider all the other independent variables, through controls or otherwise. Furthermore, we often do not know or cannot measure all the factors that we think will influence Y. As a consequence, we are again obliged to relinquish a deterministic measurement of the influence of X1 on Y, even if we are prepared to make a deterministic statement about its influence. There are yet other reasons for reverting to a probabilistic rather than a deterministic approach, 310 / Social Forces 70:2, December 1991 namely, the role of chance in affecting outcomes. Beyond consideration here is the question of whether chance per se exists or is simply a residual label referring to our ignorance about additional influences and/or inadequate measures for the variables under scrutiny. In either case, some form or another of indeterminacy is clearly useful to employ in the physical sciences, let alone in the social sciences (see examples in Lieberson [1985:94-97]). Any of these factors would lead to probabilistic statements of outcomes rather than deterministic ones. This distinction is more than merely an academic one. Rather,