Improving Coding Reliability for Open-Ended Questions

One of the basic measurement problems in survey research is the reliable coding of open-ended questions. A posteriori methods for improving coding reliability are distinguished from a priori methods. An a posteriori method has been shown to be of limited value for improving reliability for certain coding tasks. This article proposes and illustrates a multi-step, a priori procedure for generating coding categories for open-ended items. Preliminary evidence is presented indicating that this method may yield reliability levels far superior to those typically obtained in coding open-ended questions. Andrew C. Montgomery is a Ph.D. candidate and Kathleen S. Crittenden is an Associate Professor in the Sociology Department at the University of Illinois at Chicago Circle. The authors are indebted to Elfriede Wedam and the coding staff of the Survey Research Laboratory, University of Illinois, Chicago, for assistance in empirical evaluation of procedures proposed here. The authors express their appreciation to Robert K. LeBailly and Richard B. Warnecke for their comments on an earlier draft. The article is dedicated to the memory of Shirley A. Star, 1918-1976, who in the early 1950s at the National Opinion Research Center had developed and was using a sort procedure for coding open-ended questions. POQ 41 (1977) 235-243 This content downloaded from 207.46. 13.76 on Sat, 10 Sep 2016 05:30:43 UTC All use subject to http://about.jstor. org/terms 236 MONTGOMERY AND CRITTENDEN on a single classificatory principle; (3) adaptation to the structure of the situation; and (4) adaptation to the respondent's frame of reference. A recent empirical study of coding error suggests that the levels of intercoder reliability for certain open-ended coding tasks are unacceptably low (Crittenden and Hill, 1971). Coding tasks can be divided into several basic types. Type A coding tasks require a coder to find a specific answer to a specific question at a given place on an instrument. Type B-I coding tasks involve locating relevant information within a larger context. Finally, type B-2 coding tasks are those where the coder has not only to locate relevant information, but also to evaluate the relative importance of two or more possible responses to arrive at a single code. Crittenden and Hill have found alarmingly low levels of intercoder reliability for type B-I tasks, and even lower levels for type B-2 tasks.1 Methods for improving coding reliability may be distinguished in terms of the stage of research at which they are employed. A posteriori methods are adopted after collection of the data to be coded and after development of coding categories. A priori procedures, on the other hand, are employed after data collection but prior to the designation of final coding categories. Suitable for use with data from a pilot study, pretest, small project or sample from a large data set, a priori methods have superior potential for reducing coding error because they permit modification of coding categories where indicated. Of course, both a priori and aposteriori methods may be applied to a single data set. Coding by majority rule, an example of an a posteriori procedure, involves independent coding of a data set by three or more persons: the modal coding response is specified as the final code for any coding decision. Research assessing the levels of intercoder reliability resulting from this procedure has shown majority rule to be effective in improving reliability in certain coding tasks but not in others (Crittenden, 1972). Type A tasks can be increased to nearly perfect reliability through use of majority rule. Since these tasks have high initial reliability levels, the additional expense of majority rule may not be warranted. Even though reliability for type B-1 tasks is increased substantially, the final reliability resulting from majority rule may be unacceptable to the researcher. For type B-2 tasks, majority rule cannot be depended upon even to increase coding reliability, much less increase it to acceptable levels.2 We suspect that one reason for low reliability in type BI and B-2 tasks is failure on the part of the investigator to meet Lazarsfeld and Barton's (1955:84) fourth requirement. If the original codes, based on categories of interest to the investigator, fail to take into account the respondent's ' On a reliability measure based on Mueller and Schuessler's index of qualitative variation, eleven type A coding tasks varied from . 702 to 1.000; six type B-1 tasks, from . 488 to . 694; and five type B-2 items, from . 322 to . 413. For a discussion of I.Q. V. , see Mueller, et al (1970:174-178). 2 On the average, type A tasks increased from . 940 to . 984; type B-1, from . 592 to . 690; and type B-2, from . 371 to . 455, using the Crittenden-Hill reliability measure. This content downloaded from 207.46. 13.76 on Sat, 10 Sep 2016 05:30:43 UTC All use subject to http://about.jstor. org/terms CODING RELIABILITY FOR OPEN-ENDED QUESTIONS 237 frame of reference, then a posteriori methods can hardly be expected to eliminate this source of unreliability. On the other hand, a set of categories developed empirically from the responses to be coded will reflect the point of view of the respondent. Informed by this point of view, the investigator still has the responsibility of defining the categories in such a way as to preserve pertinent theoretical distinctions. The limited value of a posteriori methods such as majority rule for raising intercoder reliability to satisfactory levels for certain coding tasks points to the importance of pursuing the potential of a priori methods for improving the reliability of such problematic tasks. This article presents a multi-step procedure for using verbatim responses to generate coding categories for open-ended items.3 Formalization of the Method The first step is to determine whether there is a logically correct set of categories into which all responses to a question can be coded. To this end, two independent coders engage in a modified random sort of a set of N verbatim responses (either the entire set or a representative subset). Beginning with the first questionnaire (or interview), one coder lists the exact response along with the questionnaire identification number. For the second questionnaire, the coder decides whether the response is substantially the same as the first. If so, the identification number is listed after the first number; if not, the coder copies the response and forms a second category. Taking a third questionnaire, the coder compares the response with the previously formed categories. If the response does not fit an existing category, a new category is formed. This procedure is used to place all N responses into categories. Then the entire sort procedure is repeated independently by a second coder. The two sets of sorted responses are next arranged in a cross-tabulation as formalized in Table 1. Taking the entries in Table 1, any two questionnaires can be considered to be coded in the same category or different categories by one coder and the same or differently by the second coder. The distribution of these designations for N(N-1)/2 pairs of questionnaires can be represented in a fourfold table such as Table 2, which can be constructed from calculations performed on Table 1. Entries in Table 2 are computed using the following equations: