Improving the Quality of Survey Research in Democratizing Countries

The dramatic expansion of democratic regimes throughout the world has produced a boom in the field of survey research. There are at least six reasons for this boom. First, democracy brings with it elections, and with elections, parties and candidates who want to know where and how to campaign and contributors who want to know on which “horse” to place their bets. Second, democratic governments care very much about public opinion since not only does their reelection depend upon the public will, but their ability to govern depends to a great degree on how well they are able to gauge public reaction to their policies. Third, democratic governments want objective information to help them plan their programs and to be able to gauge their impacts once implemented. Fourth, international donors increasingly carry out surveys of “users” or potential users of public services to help them plan their investment strategies. Fifth, international donors regularly examine program impacts as a means of evaluating project success and as a means of targeting future grants and loans. Finally, within the field of political science at least, there is a growing consensus that political culture matters for sustaining democracy and that the entire “democracy game” goes beyond finding the right institutions to having citizens believe in democratic principles. There is, of course, no unanimity on this point, raised so forcefully years ago in Dahl’s “Preface to Democratic Theory,” and there will be those among the “new institutionalists” who entirely dismiss the role of what Dahl called the “consensus on the polyarchal norms” (1956, 135). Yet, broadly speaking, it is fair to say that most democracy experts would agree that in democracies, the public matters; publics vote, protest, and even rebel, and to exclude them in the calculus of the study of democratic consolidation is to risk missing an important part of the story. Social scientists, it turns out, are major beneficiaries of this boom in polling in the developing world. However, cost is a major factor constraining our abilities to advance the frontiers of science. Survey methodology is very costly to implement in advanced industrial countries. Indeed, the cost per interview has grown so high that few surveys are conducted face-to-face. Instead, social scientists in the United States and Western Europe rely heavily on telephone interviews, self-administered mail questionnaires, or one of several variations of Internet polling. Yet, we are all aware of the problems in alternative modes of survey research; in some phone surveys a very high percentage of all calls are diverted by answering machines, callblocking devices, or flat refusals by respondents, not to mention the growing challenge of cell phone-only households. Since cell phones are by definition mobile, designing a sample that includes cell phones is extremely difficult since one does not know precisely into which sampling unit each cell phone belongs. Without that knowledge, it is very difficult to define sample parameters and determine sampling errors. A further problem in all phone surveys is that both non-contact and refusal rates are increasing. Moreover, the duration of interviews in telephone (and Internet) surveys is constrained since it is very difficult to prevent respondents from hanging up when the interviews go beyond 15 or 20 minutes. This forces researchers to limit the data that they can obtain from each respondent. Survey research faces a special constraint in much of the developing world where large portions of the population, especially the poor, do not have phones and therefore are missing from surveys that exclude door-to-door interviewing. Mail surveys have certain advantages, including cost, but it is difficult to ascertain who actually responded to the questionnaire in a multipleindividual household, and mail questionnaires assume a lingua franca which may not be the operative language for minority populations. In the developing world, postal systems are often unreliable, household addresses unreliable or unknowable, and significant proportions of the population illiterate or only semiliterate, thereby excluding them entirely from the sample. For all these reasons, face-to-face surveys in the developing world have become especially attractive; in much of Latin America, the region I know best, face-to-face interviews of an average of 45 minutes can be conducted for about $15–$25 each and sometimes less, including costs of sample design, coding, and data entry. This is in contrast to prices averaging 10 times that for comparable, in-person interviews in the United States. Cost varies, of course, by country depending on exchange rates, labor costs, country size, and population density. It is far cheaper, for example, to carry out a national sample in El Salvador (only 21,000 Km and 292 persons per square kilometer), than in Bolivia (1.1 million Km and 7 people per square kilometer) (World Bank 2000, 10). Yet this variation is trivial when compared to the cost of carrying out face-to-face interviews in advanced industrial nations. Elite interviews, which one normally expects to carry out face to face, are almost always far less costly in developing than in the advanced industrial nations—again, because of lower labor costs. In short, dollar for dollar, one can learn far more about a far wider range of national populations by carrying out research in the developing world rather than in the advanced industrial world. If we seek scientific advancement, the ability to find patterns and the information to make generalizations, then surveys in the developing world, ceteris paribus, will bring the most rapid progress.