Possible Falsification of Survey Data: An Analysis of a Mail Survey in the Soviet Union

Analysis of data from a mail survey in the Soviet Union, carried out by the agency responsible for the distribution of America Illustrated, strongly suggests that the data were falsified. Information about Soviet society (including the results of a number of other surveys of the Soviet population) and probability theory were used to generate a set of expected results for this survey. The actual results differed dramatically from the expected-the response rate was much too high, the sample was too young and contained too many workers, and too many people had managed to see almost all of the previous year's issues. These discrepancies, combined with the fact that the results served the interests of the Soviet officials and that it would have been easy for them to falsify the data, lead us to suspect that they did so. The conclusion discusses the conditions under which various kinds of falsification are most likely and general strategies for detecting falsification. Stan A. Kaplowitz is Associate Professor and Vladimir Shlapentokh is Visiting Professor in the Department of Sociology, Michigan State University. Authors are listed alphabetically. Versions of this article have been presented at meetings of the North Central Sociological Association in Cleveland in April 1981 and the American Sociological Association in Toronto in August 1981. The authors wish to thank Daniel Steinmetz, Edward L. Fink, and Bernard Finifter for their valuable comments and James P. McGregor, Office of Research at the International Communication Agency, for providing us with supplementary data from the America Illustrated mail survey. Public Opinion Quarterly Vol. 46 1-23 ? 1982 by The Trustees of Columbia University Published by Elsevier North-Holland, Inc. 0033-362X/82/0046-1/$2.50 This content downloaded from 157.55.39.127 on Mon, 27 Jun 2016 06:13:37 UTC All use subject to http://about.jstor.org/terms 2 KAPLOWITZ AND SHLAPENTOKH indicated that Newton, Mendel, and Dalton (or their assistants) "fudged" some of their data (Barber, 1976). More recently, psychologist Cyril Burt has published data which are considered falsified "beyond reasonable doubt" (Hearnshaw, 1979). Brazilian authorities appear to have falsified data to make it seem that polio had been wiped out (Keerdoja et al., 1980), and there is suspicion of falsification in parapsychology research (Barber, 1976). In sociology, we almost never question the veracity of data as historians do, and we do not have the protection via replication that natural scientists do. In social science, discrepant results can typically be attributed to differences in methodology or in the population being studied. Naturally, both the lack of attention to the problem and the difficulty of detecting it make it more likely that there will be falsification when it is in someone's interest to have it so. We believe that the detection of falsification is, however, possible, and we shall demonstrate this by arguing that the data from a mail survey done in the USSR, by an American agency, were probably falsified. The major evidence of falsification is inconsistency between the actual data and the data which would be expected on the basis of some reasonable assumptions. But agencies that engage in falsification will obviously try to disguise this fact. Hence, the assumptions used to generate the expected data may have to be based on impressionistic data and indirect evidence.' In short, demonstrating falsification requires that we be not only scientists but also detectives, building a case out of a chain of circumstantial evidence. The detection of falsification requires both sociological knowledge about the topic and society under study and mathematical/statistical knowledge. Conversely, those who falsify data can avoid detection if they know enough sociology to make their results consistent with other findings in their society and enough about probability and statistics to simulate randomness, where that is appropriate. Overview of the America Illustrated Survey America Illustrated (AI) is a monthly Russian language publication produced by a U.S. government agency, the International Communication Agency (ICA), for distribution in the Soviet Union. Its distri1 Such indirect methods were used by Trabalka et al. (1980) in arguing that the Soviet Union had experienced a serious (and unreported) nuclear accident in the late 1950s. Maps showed that a community had disappeared and other evidence showed high levels of radioactive contamination in that general area. Moreover, while a Soviet publication mentioned a planned experiment in contamination, the large size of the contaminated area convinced the investigators that deliberate contamination was very unlikely. This content downloaded from 157.55.39.127 on Mon, 27 Jun 2016 06:13:37 UTC All use subject to http://about.jstor.org/terms ANALYSIS OF A SURVEY IN THE SOVIET UNION 3 bution is governed by the 1973 USA-USSR Cultural Exchange agreement, which also permits the English language publication Soviet Life to be distributed in the United States. The agreement specified that 62,000 copies of each issue be distributed, specified the proportions to be available by subscription and at newsstands, and also specified a minimal number of cities where these publications were to be distributed. From late 1977 to early 1978, ICA undertook a mail survey of Soviet readers to learn what ATs readers thought of the publication, what kinds of people read AI, and whether AI was, in fact, being circulated outside of the largest cities. Four questionnaire forms were placed in each of the 62,000 copies of three different issues of AI. Respondents were to mail their completed responses to the American embassy. As an incentive, each respondent was to receive a set of prints by American artists. The description of the survey and its results appears as "Survey of Soviet Readers of America Illustrated," (SSRAI-see McGregor, 1978). The ICA was obviously pleased with the results. First, while the return rate, which we estimate at 2.0 percent,2 was very low by the standards of mail surveys in Western countries, "the number of returns (1,962) is by far the highest of any known survey effort undertaken by a Western governmental agency in Eastern Europe or the Soviet Union" (SSRAI:i). Moreover, the results showed a high degree of reader loyalty, as the median number of issues read of the past 12 2 The response rate depends not just on the number of responses (1,962) but on the number of people who received one or more copies of the questionnaire. We know that there were 62,000 copies of each issue, three issues with a questionnaire, and four copies of the questionnaire within each issue. To determine the number of different people who received one or more copies of the questionnaire we must estimate the mean number of issues containing a questionnaire received by each respondent. To do this, we use the data in SSRAI Table 5, which tells us the distribution of the number of issues seen by respondents in the past year. If a respondent has seen k of the past 12 issues, the expected number, of the three issues containing questionnaires, which 2(k1) he/she has received is 1 + 1 (If a person responds, then even if he or she saw only one issue, it must have been one of the issues with the questionnaire.) We then find that the mean number of those issues which a respondent read was 2.34. Dividing the total number of issues containing questionnaires (186,000) by 2.34 gives us 79,500 different recipients of those issues. We must next take into account the fact that there was more than one questionnaire in each issue and that 16 percent of the respondents (307) indicated that they had gotten their copy from a friend or relative. To do this we subtract the 307 respondents who indicated that they had gotten their copy of AI from a friend or relative from our 1,962 respondents, giving us a response rate of 2.0 percent among those who were initial recipients of an issue. We further assume that when a pass-along reader had access to a questionnaire, he or she had the same probability as an initial recipient of responding, thus leaving the overall response rate at 2.0 percent. This content downloaded from 157.55.39.127 on Mon, 27 Jun 2016 06:13:37 UTC All use subject to http://about.jstor.org/terms 4 KAPLOWITZ AND SHLAPENTOKH was between 9 and 10 (SSRAI: 12). Finally, the ICA may well have been pleased to find that AI was read by a broad cross-section of the Soviet population, reached over 200 cities, and was available at kiosks as well as by subscription. Not only did the ICA have reason to be pleased with these results, so did the Soviet government. First, the results suggested that the Soviet government is living up to its agreement regarding the distribution of AI. The Cultural Exchange agreement calls for AI to be distributed in 85 specific cities. Responses were received from 77 of those 85 (SSRAI: 10). Moreover, the Soviet distribution agency claims that one-fourth of the copies are subscription and three-fourths via kiosk (SSRAI: 15). SSRAI (p. 14, Table 6) reports that almost twice as many respondents received AI at a kiosk as via subscription. This suggests that here, too, the Soviet government has kept its word. The data on articles preferred by readers should also please the Soviet authorities, since the results show a "strong preference for lighter topics, and the avoidance of economic and political subjects"