Where Do We Get Our Ideas? A Reply to Nelson Polsby

In 1991 the Kettering Foundation published a paper on Americans' attitudes toward politics and the political system in the United States today. The report, Citizens and Politics: A View from Main Street America, was written by Richard Harwood and was based on focus group research Harwood conducted with ten groups composed of roughly twelve persons each. The report described Americans as frustrated with, angry about, and alienated from the political system and those who run it. In "Where Do You Get Your Ideas?" (PS, March 1993), Nelson Polsby holds up Citizens and Politics as an example of how not to do social science and as a cautionary tale about the importance to a democratic society of vigilant political scientists ("alert watchers") who, constant in their rigorous fidelity to the standards and methods of scientific inquiry, stand between us and epistemological-perhaps even political-chaos. That a democratic society needs such persons I do not doubt, and I respect Polsby's commitment to the cause. It's not his intentions that I question, but rather his faith in the unique superiority of the particular standards and methods he recommends. Polsby is surely correct about one thing: because qualitative research (of which focus groups are one kind) is open-ended and will be influenced by the experience and judgment of the researcher, there is always a risk that the results will appear to be those that the researcher wishes, albeit perhaps unconsciously, to obtain. Polsby is right to urge those who encounter such results to treat them cautiously, taking care not to accept them at face value. This is good advice when approaching any qualitative research, whether based on participant-observation, case studies, one-on-one in-depth interviews, or focus groups. The danger that one who employs such methods might introduce bias into the questions he or she poses to participants and into the interpretation he or she places on the answers participants give should not be minimized, and it must be guarded against. Any researcher using focus groups ought to take a long, hard look at findings that confirm his or her expectations. Polsby also acknowledges the methodological modesty of Citizens and Politics, quoting Harwood's explicit recognition that "the observations detailed in the report are 'hypotheses or insights that would need to be validated by reliable quantitative methods before being considered definitive.' " He complains, however, that the authors made no attempt to confirm or disconfirm their "hypotheses." It is not clear, however, why it should be incumbent on Harwood (whose investigation was, by his own admission, meant to yield only tentative and suggestive conclusions) to undertake the attempt at verification that Polsby urges. Wouldn't it be better to have the conclusions of Citizens and Politics confirmed or disconfirmed independently? Polsby cites the debacle not long ago at the University of Utah involving two scientists' claim that they had achieved so-called cold fusion. Now, imagine the Utah scholars' counterparts at UC-Berkeley leaving it up to the original investigators to replicate their own findings and remaining content to complain about the lack of verification in their report. Precisely because of the danger that the investigators themselves may not treat their results as critically as they should, skeptics have a responsibility to test conclusions they find hard to credit. I wonder, too, why Polsby quotes at length, not from the findings and discussion prepared by Harwood, nor even from Kettering president David Mathews's foreword to the report, but from a subsequent editorial by Mathews in an issue of The Kettering Review. This is like basing one's response to a newly published book on a review of the book rather than on the book itself. Mathews's editorial interpretation of Citizens and Politics was precisely that. It wasn't the report itself, and as such is at best a secondary source for the conclusions Polsby draws. But these are small points. More serious is Polsby's insinuation that the authors of Citizens and Politics may have acted in bad faith, or, more precisely (and charitably), that they could have done so. "A really skilled moderator," Polsby says, "ought to be able in two short hours to get a focus group of 'approximately 12 people' to say nearly anything" (p. 85). Polsby actually raises two questions here. One has to do with the size of the sample (ten groups of about twelve persons each, roughly 120 people) on which the conclusions of Citizens and Politics are based. The second concerns the scientific value of focus group research in general. Let me address the former question briefly before delving into the latter. Although focus groups should be as representative as possible of the target population (those used in preparing Citizens and Politics constituted a stratified random sample that took account of differences among Americans related to race, age, income, education, and region), it is unnecessary and inappropriate to insist that they involve as many subjects as surveys require if the sample is to be considered adequate. Focus group research aims to get at the concerns, needs, and feelings that underlie people's opinions and preferences. Time and again, such research has revealed that, despite the diversity of circumstance, experience, constitution, perception, etc. that people bring to consideration of a public problem, a relatively few,