Translating Quantitative Methods for Qualitative Researchers: The Case of Selection Bias

y pingg, Keohane, and Verba's Designing Social Inquiry: Scientific Inference in Qualitative Research ..L _(KKV) is an ambitious attempt to translate for the qualitative researcher a series of insights derived from quantitative methodology. The authors build on the basic framework of regression analysis-which has been enriched over the past two decades with innovations introduced by econometricians and statisticians-to make recommendations about how qualitative analysts should confront a variety of methodological problems. KKV are strongly committed to the premise that the underlying logic of quantitative and qualitative research is basically the same (p. ix). At the same time, they are attentive to the specific dilemmas that arise in actually carrying out qualitative research, and they provide many useful examples and employ clear, nontechnical language. This is a book that moves the discussion forward, and therefore merits close attention. Selection bias is one of the important topics into which quantitative methodologists have recently offered significant new insights. Hence, the assessment of KKV's treatment of this topic' provides a useful window for evaluating their effort to transpose complex issues of quantitative method to the sphere of qualitative research. This is also an interesting topic to address because KKV are centrally concerned with selection bias resulting from deliberate selection by the investigator. Their recommendations are consequently of special importance: if their diagnosis is correct, a small improvement in methodological selfawareness can yield a large improvement in scholarship. Finally, KKV's recommendations merit examination precisely because they are quite emphatic. Given their emphatic character, readers may desire assurance that they are, in fact, receiving sound advice. The question of how to situate the problem of selection bias in relation to a spectrum of other methodological and theoretical issues is not an easy one. At one pole, in discussions of selection bias in quantitative sociology, one finds an influential article suggesting that the impact of selection bias is not as serious as has been believed, that efforts to introduce statistical corrections for selection bias may create more problems than they solve, and that among the many problems of quantitative analysis, this one does not merit special attention (Stolzenberg and Relles 1990). In the present context, the appropriate point of entry into the problem is different. First, in the examination of selection bias in qualitative political research, it is much more difficult to assess its precise impact, so that conclusions about its importance are inevitably more tentative. Second, KKV's recommendations about selection bias are centrally concerned with deliberate selection by the investigator. Hence, relevant corrections do not involve statistical procedures, but rather basic choices about case selection that are relatively easy to achieve. Third, one of KKV's central goals is to explore the interrelations among a series of different methodological problems. They are not singling out selection bias as a paramount problem: in their view, it is one of many. I conclude that KKV offer useful recommendations regarding selection bias. Yet they subsume under this term various issues with which qualitative researchers may already be familiar, but under different labels. Obviously, in a complex field it is common to find that a given phenomenon is named in different ways. For example, what was probably the first paper ever published on selection bias referred to it as a problem of spurious correlation (Berkson 1946, 51). Nevertheless, the overlap of labels raises the question of whether KKV's methodological insights really offer something new to the qualitative analyst. In fact, some of the important recommendations offered by KKV can just as well be viewed not as insights derived from advanced quantitative methods, but rather as part of a long-standing effort to encourage qualitative scholars to be more methodologically and theoretically aware of which cases they are analyzing. This self-consciousness can also be encouraged by insistently posing a question that, according to the traditional lore of the comparative politics subfield, should often be asked at doctoral dissertation defenses: "What is this a case of?" Many issues that underlie this question and that are highly relevant to KKV's discussion of selection bias have previously been raised in-discussions of the comparative method, that is, the branch of methodology concerned with the systematic, qualitative analysis of relatively small numbers of cases (a "small N"l).2 In assessing methodological claims about selection bias, it is useful to take these earlier discussions as a base line. With regard to issues of case selection, they include the ongoing evaluation of J. S. Mill's methods of experimental inquiry, the related distinction between "most similar" and "most different" systems designs, and a new perspective on case selection in small-N studies arising from counterfactual analysis. Regarding the problem of applying concepts and indicators across diverse contexts, an issue that arises in KKV's discussion, relevant insights from work on comparative method include the traditional concern with "conceptual stretching" and the use of "system-specific," as opposed to "common," indicators. Regarding the issue of generalization, relevant insights include the argument that it may at times be appropriate for scholars to limit severely the scope of generalizations from a given set