On Checklists and Open-Ended Questions: A Reply

There is no adequate response to an irrelevant critique. Mr. van den Berghe's statement of our research findings is essentially irrelevant because he (a) ignored the stated purpose of the research, (b) ignored the major findings of the research, (c) ignored the cited research program of which this study was a part, (d) ignored the stated rationale of stereotype research-which defines the limits of the study, (e) ignored our own statement of limitations, and (f) ignored our conclusions. Even his analogical argument (of Fahrenheit versus centigrade thermometers) is as incorrect and irrelevant as what he attributes to us. As far as our study is concerned, we can only call for those professionally interested in stereotype research to return to the original report and not to rely on our critic's parody. It has been argued in the scattered methodological literature that open-ended questionnaires may decrease the volume of response: by requiring more work on the part of the respondent, by provoking greater defensiveness through the demand that the respondent be more self-revelatory, and by decreasing the flow from those with less verbal ability. These arguments are a part of the folklore of present social research traditions; and they are without any firm substantiation. Even if these arguments about the quantity of response were correct, to the best of our knowledge no one seriously argues that responses to open-ended questions are-in any scientifically meaningful manner-qualitatively different from those produced on structured answer formats. The dominant procedure in both schedule and questionnaire construction today is to begin with open-ended pretests as a means of ascertaining the basic categories of response. These original responses and response categories are then taken as the basis for the development of structured answer formats. Thus, in subsequent research a properly constructed structured questionnaire should provide fundamentally the same distribution of responses as would be obtained through an unstructured counterpart. It was the major finding of our research, among others which Mr. van den Berghe failed to note, that our two answer formats produced quite different response distributions. We displayed these results in a table which covered over four pages. The unstructured format produced not only different descriptive termns, with only 13 percent of the total number common to both answer formats, but in approximately half the sampled instances it produced terms of essentially different mnean-ings. A good part of the paper is devoted to exploring why this occurred and what the implications might be for the study of stereotypes. Van den Berghe, however, chose to ignore these substantive differences by focusing instead on consensus. In doing so he declared the lower level of consensus displayed in the open-ended responses to be an artifact of the instrument. Thus, he proposed that a correction formula be applied to the open-ended responses to increase consensus and more closely align the results of the two instruments. Had our critic not felt obliged to defend the checklist, he might have just as easily, and perhaps less incorrectly, proposed that the higher level of consensus in the checklist responses was artifactual, and that a reverse formula was necessary to decrease consensus among checklist responses. If van den Berghe means to seriously define the point at issue as being that an adequately constructed checklist would always yield results meaningfully different from an openended questionnaire, then the means of resolving that issue are clear and at hand. If we had two instruments of equal reliability, with each purported to measure the same phenomenon, then we could choose rationally between them by specifying some external criterion by which to assess their relative performance. In the case of stereotype measures, we hope that Mr. van den Berghe is not serious in defining the issue as one of checklists versus open-ended instruments. We certainly did not define the issue as such. Our critique was not even