Continuous Quality Improvement for Survey Operations : General Principles and Applications

A typical sample survey consists of a number of separate, but interrelated operations that may either change the form or modify the content of the data. These operations include data collection (or interviewing), data transmission and receipt, data editing and cleaning, response encoding, and data entry. Depending on the scale of the survey, these operations may involve only a few operators or, as in the case of a census, hundreds. It is also quite common that the operators are inexperienced, lowly-paid, and minimally trained workers. The operations in which these operators are engaged may be quite complex and error prone. They may consist of repetitive and monotonous activities or, as in the case of interviewing, may require complex thinking and quick judgements on the part of the operator. Thus, almost all survey organizations employ some type of quality control for survey operations in order to ensure that the final results are of acceptable quality. In most survey organizations, quality control is usually based upon some form of acceptance sampling method. Acceptance sampling methods (also known as inspection methods) involve selecting a sample of items from a work unit, inspecting the sample to determine the number of items which are "in error" or which deviate from specified procedures, and then either rejecting the work unit if the number of errors (or defects) exceeds some threshold value or otherwise accepting the work unit. For most operations, rejected units are usually reworked to remove the errors. There are at least three potential objectives of inspection methods. One is to ensure that the error in the output of an operation does not exceed some specified level. This is possible through the use of probability methods for determining the number of units to sample, the number of items within each unit to inspect, and the threshold value for rejecting a unit. t For some implementations of acceptance sampling, this may be the sole purpose of the activity. However, for other implementations, a second objective may be to improve the skills of the operators. This is usually carried-out by providing "feedback" to the operator responsible for a rejected work unit; that is, the operator is given information on the number and types of errors found in a rejected unit. In this way, it is hoped that the operator can take whatever steps are necessary to avoid these types of errors in the future work. A third objective may be simply to "keep the operators on their toes. " That is, by providing the "threat" of inspection, the operators are dissuaded from intentionally deviating from procedures or taking "short-cuts" which may damage data quality in order to meet production goals. This use of acceptance sampling may be criticized on several grounds. First, inspection adds significantly to the costs of a product. Inspection is necessary only because the operation is prone to error. If the process could be redesigned so that the error rate were extremely small, inspection would be unnecessary and the cost of inspection would be saved. Second, acceptance sampling can ensure an acceptable level of quality only if the inspection process is nearly perfect. However, this is seldom the case as the inspection error rate may be substantial. Thus, the error in the final product may be unacceptable even with a high rate of inspection. Further, to achieve very small levels of error say, 1% or less 100% inspection is required. However, this level of inspection may be unaffordable. In addition, the inspection process may miss a large proportion of the errors in the operation since the inspection operation is as large as the operation being inspected. Third, the recently quality control literature (see, for example, Ishikawa, 1990) provides strong evidence that when inspection methods are used, operators take less responsibility for the quality of their work since quality is perceived as being the job of the inspectors. Therefore, the operators lack the motivation to improve quality. Fourth, when feedback to the operators is based upon rejected units, the implication is that they are solely responsible for the errors in their work unit. However, the quality control literature suggests that for most operations, the operator may only be responsible for 20-30% of the errors. Thus, placing full responsibility for the errors in a process on the operators risks demoralizing them. This demoralizing effect is exacerbated by an imperfect inspection process which both fails to identify true errors and erroneously classifies correct items as errors. Finally, the feedback from inspection to the operators is both time consuming and, in many cases, ineffective. Part of the reason for this is the lack of information on the "root causes" of the errors. The operator may be told that he/she is