Where the Model Frequently Meets the Road: Combining Statistical, Formal, and Case Study Methods

This paper analyzes the working or default assumptions researchers in the formal, statistical, and case study traditions typically hold regarding the sources of unexplained variance, the meaning of outliers, parameter values, human motivation, functional forms, time, and external validity. We argue that these working assumptions are often not essential to each method, and that these assumptions can be relaxed in ways that allow multimethod work to proceed. We then analyze the comparative advantages of different combinations of formal, statistical, and case study methods for various theory-building and theory-testing research objectives. We illustrate these advantages and offer methodological advice on how to combine different methods, through analysis and critique of prominent examples of multimethod research. 1 Draft; comments welcome. Corresponding author: bennetta@georgetown.edu The great potential for synergistic research combining the three leading methodological traditions in political science—formal modeling, statistics, and case studies—is not being fully realized. There is much to be gained by combining alternative research methods, as each method can complement the others. Formal models can provide deductively valid and counterintuitive insights, but they are not an empirical method and must be combined with either statistical or case studies to become so. Case study methods can use process tracing to look for detailed evidence on causal mechanisms and causal processes within cases, but they face limitations in generalizing these results to cases not studied. Statistical studies can identify patterns and correlations in specified populations, but they lack the guarantee of logical theoretical consistency that formal models can provide, and they can benefit from the rich appreciation of processes and contexts that case studies can provide. Combining methods provides opportunities for the development and testing of theories that no single method can match.2 Yet the promise of multimethod work has been only partly fulfilled because of methodological challenges, institutional barriers, and conceptual limitations. These include the technical difficulties of each method alone and the even greater difficulties of combining them, the disincentives to co-authorship in the discipline, and the relative neglect of some possible combinations of methods and kinds of theory-building. There is also the danger that in published reviews (and even more importantly in job talks) researchers carrying out multi-method work 2 For analyses of the contributions different methods have made to specific research programs, see Sambanis (2004) on civil wars, Back and Dumont (2007) on political coalitions, Hafner-Burton and Ron (2009) on human rights, and George and Bennett (2005) on the inter-democratic peace. will be unfairly held to impossible standards and viewed as falling between three stools: not sufficiently up on the latest models for the formal modelers’ tastes, not on the cutting edge of the very latest statistical methods, and not sufficiently deep into the archives and interviews and historical context to satisfy the qualitative researchers. Perhaps the most pervasive inhibition on multimethod work is conceptual: researchers in each methodological tradition have tended to share methodological assumptions that clash with the assumptions adopted by practitioners of alternative methods. Barriers such as these have kept multimethod work from growing even as each methodological tradition has become more refined in the last two decades. One survey found that from 1975 to 2000, the proportion of articles using more than one method remained steady, ranging between 15% and 19% of the articles surveyed. Moreover, these multimethod articles rarely used some combinations of methods. Most (between 50% and 90% in different time periods) combined formal models and statistical research, but only 33 to 50 percent combined statistics and case studies, and only 5 to 16 percent combined formal models and case studies (Bennett, Barth, and Rutherford, 2003). Similarly, despite the great volume of literature explicating each methodological tradition, scholars have written very little on how to combine methods. We explore one explanation for this disjuncture that arose in our analysis of examples of multi-method work: substantial and often implicit differences in the methodological assumptions that researchers from each tradition typically hold regarding the sources of unexplained variation, the meaning of outliers, parameter values, human motivation, functional forms, time, and external validity.3 We argue that these working assumptions are often not essential to each 3 There is of course a diversity of opinion on these issues among methodologists and practitioners of each methodological tradition. We hope only to paint an accurate portrait of views commonly held within each tradition, method, and that there is much to be gained by relaxing them in ways that allow multimethod work to proceed. We then analyze the comparative advantages of different combinations of methods for various theory-building and theorytesting research objectives. We illustrate these advantages, and offer methodological advice on how to combine different methods, through analysis and critique of prominent examples of multimethod research. The Clash of Assumptions One of the major obstacles to integration across methods is the seeming incompatibility among the assumptions widely held by practitioners of different methodological traditions. While these assumptions are thoroughly embedded in each methodological tradition and are usually treated as unproblematic assertions about the ontology of the social world, they are in fact often merely simplifying working assumptions that, albeit convenient, are not usually necessary for research to proceed. We cannot ultimately know, for example, whether a given phenomenon is produced by a deterministic process or an inherently stochastic process, as in either case our observations will include some unavoidable amount of measurement error and model mis-specification. At times we proceed by making working assumptions, often implicit ones, on such issues as we proceed in our research. The danger lies not in making such assumptions but in reifying them, thereby rendering them both invisible and inviolable. It therefore behooves us to examine the key working assumptions held by researchers in each not to imply that these views are universal within each research community. Also, many methods contribute to “multimethod” research, including experiments, surveys, ethnographic studies, simulations, and other approaches. We focus on the three broad methods of case studies, formal models, and regression analysis for reasons of space. methodological tradition and to explore how they can be relaxed or adapted in ways conducive to multimethod research.4 The Sources of Unexplained Variance Whether couched as abstract arguments over the fundamental nature of causation or concrete arguments about the sources of error in the dependent variable, disagreements over assumptions about the sources and magnitude of error in the dependent variable are a persistent feature of methodological discourse.5 The most extreme divergence here is between qualitative and quantitative methodologists. For quantitative methodologists, dependent variables in the social world are typically thought to be inherently and unproblematically stochastic: like electrons fired at gates, their behavior can be predicted in the aggregate but not in any particular case. For this reason, the criticism that this or that case does not fit the story captured by the statistical model is not seen as persuasive: it is to be expected that some cases will not fit the model, simply because their outcomes cannot be perfectly predicted. Even “failure'' in a case in which the predicted probability of a “success'' is 0.9 says little about the quality of the model, as such events should happen one time out of ten. Only the aggregate results are seen to be convincing. In relying on the underlying assumptions of the central limit theorem, statistical researchers assume that a large number of small and relatively non-influential variables will produce a normal error term with occasional large outliers. At the other end of the spectrum, case study researchers focus to a much greater degree on the systematic components of the theory and are inclined to demand that any apparent 4 For a similar discussion of the differing assumptions of statistical and case study researchers, see Goertz and Mahoney, 2006. 5 As Hall (2003) demonstrates, accounts of the empirical implications of causal relations dovetail all too well with descriptions of the empirical patterns that different methods are designed to unearth. randomness in outcomes be explained in detail. Instead of taking a fundamentally stochastic world for granted, as do many researchers in the statistical tradition, case study researchers problematize this assumption. Absent some convincing rationale, quantum fluctuation is thought to be a property reserved for electrons rather than people, and case study researchers seek strong causal explanations of cases on the assumption that theoretically intelligible social behavior rather than quantum variation accounts for outcomes. Interestingly, formal modelers are of two minds on this issue. Most equilibrium solutions in formal models are deterministic, but observed social realities clearly do not neatly conform to this expectation. Thus, modelers have found it necessary to devise equilibrium concepts that can accommodate out-of-equilibrium observations. Consequently, one category of formal models, largely outside of the rational choice tradition, describes outcomes in stochastic terms. Agentbased evolutionary modeling is perhaps the best example of this approach (Cederman, 2001, Weibull, 2002). In this kind of