A Quantitative Method to Test for Consistency and Correctness in Photointerpretation

A method has been developed to quantitatively test the degree of similarity between photointerpreters andlor photointerpretation variables, such as filmlfilter type, season, and scale. This method involves giving each photointerpreter the same set of photos to interpret, and then a similarity matrix is generated for each interpreter by comparing the interpretation results to the actual ground cover. It is assumed that the interpretation has been sampled in such a way as to assure that the resulting similarity matrix is representative of that interpretation. The similarity matrix is analyzed using a computer program called KAPPA, which implements a discrete multivariate analysis technique to determine if there is a significant dgference between matrices. This technique allows the comparison of individual interpreters, a test for photointerpreter consistency, and the comparison of photointerpretation variables. method of quantifying photointerpretation results consistency in his interpretation skill. Lastly, it and gives a statistical method by which interpretmay also be important to determine if varying ers and other photointerpretation variables such as types of photography (fildfilter combinations) or fildfilter types can be compared. seasons (spring versus summer) of ~hotography The procedure proposed in this paper can test result in significantly different identifications. for significant differences between photointerPROCEDURE * Presented, in part, at the 1981 Annual Convention of the American Society of Photogrammetry, WashingThe first step in quantifying photointerpretation ton, D.C. results is to cellularize the photographs. There are PHOTOCRAMMETRIC ENG NEERING AND REMOTE SENSING, Vol. 49, No. 1, January 1983, pp. 69-74. 0099-11 12/83/4901-69$02.25/0 @ 1983 American Society of Photogrammetry PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING, 1983 many ways that this may be accomplished. The identifications or delineations made by the interpreter can be digitized, and then a given cell size can be used to assign the majority land-usel land-cover category to each cell. A dot grid could also be used in which each dot defines the sample point. Obviously, the results of the assignment to categories will vary with different cell sizes or dots per square inch. Therefore, it is important to choose a cell size that is indicative of the objectives of the photointerpretation (i.e., the more detail required, the smaller the cell size). The minimum mapping unit used in the photointerpretation is a good choice for the cell size. Once a grid system has been placed over the photo and each cell has been assigned to some category, then each cell is compared with its corresponding cell from another interpretation. If one of these interpretations is assumed to be correct (reference data), then comparison of the two sets of spatially defined cells yields a measure of "photointerpretation accuracy. " This comparison is usually expressed in the form of an error matrix. It will be assumed in this paper that the error matrices generated are representative of the actual photointerpretation results. This means that an appropriate sampling scheme was chosen and an adequate number of samples were taken when compiling the matrix. ~ e c a u s e the technique proposed here analyzes the error matrices, it is necessary that this assumption hold for any results to be meaningful. An error matrix is a square array set out in rows and columns which expresses the number of cells assigned to a particular land-cover type (photointerpretation) relative to the actual land cover (reference data). The columns represent the reference data while the rows indicate the photointerpreter assigned land-cover type (Table 1). The numbers in the error matrix are the actual tallies compiled by comparing the photointerpretation with the actual land-cover type on a cell by cell basis. As can be seen in Table 1, all correct classifications are indicated on the major diagonal of the error matrix. For example, in this case 16 cells were correctly classified as forest. Also note that the offdiagonal elements indicate misclassification, which is a combination of omission and commission error. The specific method used to generate an error matrix is dependent upon what information is desired. If the similarity between two or more photointerpreters is to be determined, then each interpreter is given the same aerial photographs to interpret. A similarity matrix is tabulated for each interpreter by comparing his interpretation with a reference data set (assumed correct). If the test involves determining the consistency over time for a single interpreter, then a representative sample of the photos is selected for interpretation at the beginning of the project. At some later date, the remainder of the photos are interpreted and the two similarity matrices (Time I versus Reference Data and Time I1 versus Reference Data) are compared. Finally, if it is desired to measure the similarity of identifications made on different types of photography, a separate interpretation is performed on the same area for each set of photos by each interpreter and the appropriate similarity matrices are generated. Once the similarity matrices have been generated, a discrete multivariate analysis technique (Bishop et al., 1975) is used to test the agreement between matrices. This technique is appropriate because the data are discrete and multinomially distributed. Parametric statistical techniques such as analysis of variance assume that the data are continuous and normally distributed. Chi square analysis is possible except that problems arise when cell values in the matrix equal zero. This problem does not affect the discrete multivariate analysis technique. The statistic used to test the agreement between similarity matrices is called (KHAT). KHAT is the TABLE 1. EXAMPLE ERROR MATRIX FOR