5.0 Computational Issues

5.1 Definitions of Precision and Accuracy

Precision and accuracy are related and, in a sense, complementary
concepts. Precision can be defined as the degree of conformity among a
set of observations to a random variable (Mikhail and Ackerman 1976). As
such, precision is a measure of dispersion of the probability
distribution associated with a measurement (repeatability of a
measurement). Accuracy, on the other hand, can be defined as how close
an estimate is to its true value. Ideally, estimates should be both
precise and accurate. In other words, a point estimate (parameter)
should be as close as possible to its true value. At the same time, the
degree of variability in the point value, as derived from repeat
measurements (sampling), should be small. However, it is possible to
have highly precise estimates that are inaccurate, and highly accurate
estimates that are imprecise. The difference between precision and
accuracy lies in the possible presence of systematic effects. A precise
but inaccurate estimate is considered to be affected by systematic
error.

If an estimate is precise, it has a narrow confidence interval. Since
the width of a confidence interval is affected by sample size, it
follows that high‑precision estimates require larger sample sizes than
low‑precision ones. Although the degree of precision in an estimate can,
to some extent, be determined by sample number, the degree of accuracy
in the estimate is determined by comparing the estimate to its expected
(required) value through hypothesis testing. Whether or not the estimate
is considered to be affected by systematic effects (i.e., is
significantly different from the expected or required value), will again
depend on the precision of the estimate. With a high‑precision estimate,
even small deviations from the required value will be considered
significant. With a low‑precision estimate, comparatively large
deviations from the expected value will probably be interpreted simply
as random effects.

In terms of accuracy assessment, both the degree of precision and the
accuracy of the estimates derived from the sample will determine whether
accuracy requirements can be considered to have been met. Because of the
proposed number of samples, the precision of the per‑class estimates for
the NPS/NBS Vegetation Mapping data will be quite low. A low‑precision
estimate will be advantageous if accuracy requirements are considered to
have been met if the true value is contained within the (comparatively
wide) confidence interval. However, if the accuracy of the estimate is
required to be equal to or greater than the required value, a
high‑precision (narrow confidence interval) estimate will be preferable,
because, with this requirement, the true value must lie at the lower
edge of or outside the confidence interval.

2.  Statistical Methods for Thematic Accuracy

Many of the techniques for measuring uncertainty in mapped classes were
developed for remote sensing to provide interpreters with ways to assess
the accuracy of remotely sensed land classifications. In this kind of
accuracy assessment, it is common practice to select a sample of
locations and to compare the class assigned to each location with some
source of higher

locations and to compare the class assigned to each location with some
source of higher accuracy, usually ground truth obtained by direct
observation in the field. The results are then tabulated in the form of
an error or misclassification matrix (also referred to as a contingency
or confusion matrix), such as the one shown in Table 2 (Congalton and
Mead 1983; Congalton, Oderwald, and Mead 1983; Story and Congalton 1986;
Rosenfield and Fitzpatrick‑Lins 1986;

and Congalton 1991).

**Table 2: Sample Misclassification Matrix for Five Classes**

  ----------- ----- ----------- --------- ---------- ---------- ---------- ---------
                    Reference                                              
                    data                                                   

  Sample data A                 B         C          D          E          Total

  A           80                4         0          15         7          106

  B           2                 17        0          9          2          30

  C           12                5         9          4          8          38

  D           7                 8         0          65         0          80

  E           3                 2         1          6          38         50

  Total       104               36        10         99         55         304
  ----------- ----- ----------- --------- ---------- ---------- ---------- ---------

In this table, the columns define the classes in the reference data, and
the rows define the classes in the data being evaluated for accuracy.
The values in the cells in the table indicate how well the classified
data agree with the reference data.

The diagonal elements of the matrix indicate correct classifications.
Therefore, a crude overall measure of accuracy is the percentage of
cases that lie on the diagonal, in this case 209/304, or 68.8%. In other
words, overall accuracy can be expressed as follows:

Unfortunately, this can be misleading as an index, since a certain
number of correct classifications will occur by chance, even in the most
uncertain situations. A preferred index is the Kappa index, which has a
maximum of 1 and a minimum of zero, the latter expected under maximum
uncertainty. Kappa is computed as follows (Foody 1992):

where P~correct~ is the proportion of correctly classified entries and
P~chance~ is the proportion of samples that could be expected to be
classified correctly by chance. P~chance~ is computed as follows (Foody
1992):

where P~row(i)~ is the proportion of total entries that are in row i,
P~column\ (i)~ is the proportion of all entries that are in column i,
and n is the total number of rows or columns.

For Table 2, the Kappa index is 58.3%, which is somewhat lower than the
percent correctly classified. Some evidence exists that the Kappa index
in fact underestimates true accuracy in the data, and modifications to
account for this exist (Foody 1992). However, it is recommended that the
unmodified Kappa index be utilized to report overall classification
accuracy for the NPS/NBS Vegetation Mapping Project, primarily because
it is documented and well known within the vegetation mapping community.

An overall statement of accuracy is useful, but it does not say anything
about the accuracy of individual classes. Per‑class accuracies can be
extracted from the contingency matrix, but they can be misleading if
they have not been differentiated into producers\' and users\' accuracy
(also referred to as errors of omission and errors of commission).
Errors of omission calculate the probability that a reference sample has
been classified correctly; this is often the only type of accuracy that
is reported. This quantity is computed by dividing the number of samples
that have been classified correctly by the total number of reference
samples in that class. Errors of commission calculate the probability
that a sample from the classified data actually represents that category
on the ground. This type of error is computed by dividing the number of
correctly classified samples by the total number of samples that were
classified as belonging to that category. The users\' and producers\'
accuracies for the example in Table 2, above, are summarized in Table 3.

**Table 3: Users\' and Producers\' Accuracy for Five Classes**

  ----------------------- ----------------------- -----------------------
  Class                   Users\' accuracy        Producers\' accuracy

  A                       75.5% (80/106)          76.9% (80/104)

  B                       56.7% (17/30)           47.2% (17/36)

  C                       23.7% (9/38)            90.0% (9/10)

  D                       81.2% (65/80)           65.7% (65/99)

  E                       76.0% (38/50)           69.1% (38/55)
  ----------------------- ----------------------- -----------------------

Producers\' accuracy is an important measure because the producers of
spatial data are interested in how well a particular area on the
Earth\'s surface can be mapped. Users\' accuracy, on the other hand, is
important for users of spatial data, because users are principally
interested in knowing how well spatial data actually represent what can
be found on the ground (Story and Congalton 1986). Ideally, both users\'
and producers\' accuracy should be similar for all classes. However, as
illustrated in Table 3, users\' and producers\' accuracy may differ
considerably among classes. For example, Class C has a producers\'
accuracy of 90%, meaning that 90% of the reference samples were also
found to be classified as Class C. Users\' accuracy, on the other hand,
is only 23%. This means that only 23% of the polygons classified as
Class C in the data can be expected to be Class C when visited on the
ground. Only reporting the producers\' accuracy as part of the accuracy
assessment would give a completely misleading picture of data accuracy
to users, who would interpret the 90% figure to mean that 90% of the
polygons classified as class C in the data also have this class on the
ground.

Ideally, accuracy requirements for classification should address both
users\' and producers\' accuracy, and it is recommended that thematic
accuracy always be reported in terms of users\' and producers\' accuracy
in order to give as complete a description of the error properties of
individual classes as possible for both the users and the producers of
spatial data. The accuracy standard envisioned for the NPS/NBS
Vegetation Mapping Project is an 80% per‑class accuracy for all thematic
classes, and this implies a users\' and producers\' accuracy of 80%.
However, depending on the degree of confusion that occurs between
classes, meeting this requirement for both users\' and producers\'
accuracy may not always be possible. In addition, it may be difficult to
obtain an adequate number of samples in the case of rare classes.
Furthermore, the accuracy of a classification is confounded by the
classification scheme itself. If two classes are difficult to
differentiate, it might seem reasonable to expect large numbers of cases
in the appropriate off‑diagonal cells of the misclassification matrix.
For example, one might conclude from Table 2 that Class D is easily
confused with Class A, since 15 of the 99 objects of Class D were
wrongly classified as A. If A and D are lumped together, forming the
misclassification matrix shown in Table 4, the percentage of samples
classified correctly rises to 76.0%, indicating a higher accuracy
because of the simpler classification scheme. The Kappa index, on the
other hand, drops from 58.3% to 56.0%, since chance agreement is now
easier to achieve.

**Table 4: Aggregation of [t]{.smallcaps}wo [e]{.smallcaps}asily
[c]{.smallcaps}onfused [c]{.smallcaps}lasses from Table 2**

  ----------- ----------- ----------- ----------- ------------ -----------
              Reference                                        
              data                                             

  Sample data A + D       B           C           E            Total

  A + D       167         12          0           7            186

  B           11          17          0           2            30

  C           16          5           9           8            38

  E           9           2           1           38           50

  Total       203         36          10          55           304
  ----------- ----------- ----------- ----------- ------------ -----------

Because misclassifications must be estimated from a single set of
samples, it is common to calculate one misclassification matrix for an
entire data set. But in reality, the chances of misclassification vary
not only by class, but also across the map, because misclassification
errors are almost certainly more likely in some areas than others. As
more is learned about the nature of errors, and why errors occur, it
will be possible to produce more refined models of error calculation
that take such spatial heterogeneity into account.

With a misclassification matrix, or similar estimate of the
probabilities that a feature belongs to each class, it is possible to
simulate error using Monte Carlo simulation, and to propagate error into
products obtained from the data set. Goodchild, Sun, and Yang (1992)
demonstrate this with a soil map example.

**5.3 *Statistical Methods for Positional Accuracy***

Positional accuracy is best discussed within the framework of
measurement. Position can be thought of as a combination of two
measurements, x and y, representing the easting and northing of a pair
of UTM coordinates, or longitude and latitude, respectively. Each
measurement is subject to error. For example, if 100 people were asked
to measure the coordinates of a road intersection on a topographic map,
the results would show variation in both coordinates. In practice, the
variation in both coordinates is likely to follow a normal distribution
or bell curve, with most measurements clustered, and a few extremes in
both positive and negative directions. The amount of variation is likely
to be similar in both coordinate directions. Moreover, errors are likely
to be uncorrelated, in the sense that the direction and amount of error
in one coordinate is independent of the direction and amount of error in
the other. Errors are also likely to be unbiased, in the sense that the
average of all 100 measurements will be very close to the true location.
If these assumptions are true, and there is no obvious reason why they
should not be, the errors in both coordinates can be visualized as a
three‑dimensional bell curve, or circular normal distribution.

Several statistics of the circular normal distribution are in common use
to describe positional accuracy. Perhaps the most commonly used is the
Circular Map Accuracy Standard, or CMAS, defined as the 90th percentile
of the circular normal distribution, or 2.146 times its standard
deviation. More graphically, it forms a circle about the true location
of the point, within which the observed location is expected to lie 90%
of the time. Using the example of the topographic map, it might turn out
that 90% of the 100 people determined the road intersection\'s
coordinates to within 0.5 millimeter of their true locations at the
scale of the map, leaving 10 people with positional errors of more than
0.5 millimeter.

CMAS is a well‑defined statistic, and it forms the basis of the current
National Map Accuracy Standard for positional accuracy (Bureau of the
Budget 1947). However, map accuracy standards are currently being
revised, and the standard is likely to become the National Cartographic
Standards for Spatial Accuracy, as proposed by the American Society for
Photogrammetry and Remote Sensing (ASPRS). With this standard,
positional accuracy is reported as a standard error in each coordinate
direction; it is a measure of variability that includes the effects of
both bias and random error. It is computed as follows (Merchant 1985):

![](media/image1.png){width="6.25in" height="0.6979166666666666in"}

where RMSE~x~ is the standard error in the x‑coordinate direction, N is
the sample size, δX~i~ is the actual (measured) coordinate location, and
δX~c~ is the true coordinate location as determined in the source of
higher accuracy.

Using standard errors to report positional accuracy is not in conflict
with the CMAS, because standard errors can be related to CMAS, provided
that the assumption that errors follow circular normal distributions
holds. Since it is common practice to state positional errors at a 90%
confidence level, it is recommended that both the standard error in each
coordinate direction, and the circular error derived from these errors,
be reported. The relationship between standard error and circular error
is as follows (ASPRS 1989):

where σ~x~ and σ~y~ are the standard errors in each coordinate
direction.

CMAS and circular errors are useful measures when positional accuracy is
assessed from a sample of points. For general purposes, however, it is
necessary to use a less rigorous approach. Unless otherwise specified in
this section of the report, the term \"positional accuracy\" should be
interpreted as a linear measure approximately equal to the CMAS, and
based on the same assumptions, but not implying the same rigor of
definition.

CMAS can also be used to describe the positional accuracy of points
relative to each other. The accuracy with which two points can be
positioned relative to each other depends on the positional accuracy of
both. If it is possible to assume that errors in the two points are
independent, a simple calculation can be used to determine the error in
relative position by taking the square root of the sum of the squares of
the two positional accuracies. For example, if one point has a
positional accuracy of 1 millimeter and a second point has a positional
accuracy of 2 millimeters, a rough estimate of the error in their
relative positions is:

(1^2^ + 2^2^)^1/2^ = 2.24 millimeter

or 54 meters on the ground for a 1:24,000 product.

**5.4 *Confidence Intervals and Sample Size***

Since accuracy estimates are based on samples, a point estimate (e.g.,
of a mean or variance) requires an associated confidence interval to be
truly useful. A confidence interval is an interval within which we have
a specific level of confidence the true value of an estimate lies. The
width of a confidence interval is affected by the sample size used to
derive the point estimate, and by the confidence level itself. Larger
sample sizes will result in a more narrow confidence interval, as will
lower confidence levels. Smaller sample sizes and higher confidence
levels will widen the confidence interval. Confidence levels are often
held fixed for a given study (i.e., all values are reported to a
predetermined level of confidence), where conventionally used confidence
levels are 90%, 95%, or 99%. Therefore, the width of the confidence
interval will vary with changing sample size. If, as is recommended for
the NPS/NBS Vegetation Mapping Project, sample size varies with class
abundance, confidence intervals for rare classes will be wider than
those for abundant classes. An alternative approach would be to hold the
width of the confidence interval at a fixed level. With a given fixed
confidence interval, our certainty (confidence level) that the true
value of the estimate falls within the interval would decrease for a
smaller sample size. Rather than report that an estimate falls within a
variable‑width interval with a fixed certainty, it is considered
preferable to report that an estimate falls within a fixed interval with
varying certainty based on the sample size used to derive the estimate.

Confidence intervals may be two‑sided or one‑sided (also referred to as
two‑tailed or one‑tailed). Two‑sided confidence limits for an estimate
express the probability that a point estimate falls within the interval.
One‑sided limits express the probability that the true value of the
estimate lies outside the limit of the interval at one specified end.
For accuracy assessment purposes, investigators are typically interested
in either a two‑sided estimate or a one‑sided estimate that expresses
whether or not true value is at least equal to or exceeds the estimated
value.

The confidence interval for a binomial distribution is obtained from the
following equation (Snedecor and Cochran 1967, p. 211):

where z~α~ comes from a table of the z‑distribution at the significance
level a (this value is 1.645 for a two‑sided limit, and 1.282 for a
one‑sided limit with a 90% confidence interval. For a 95% significance
level, the value changes to 1.645 for a one‑sided limit and 1.960 for a
two‑sided limit), is the accuracy estimate, and n is the sample size.
The term 1/(2n) is the correction for continuity. The correction should
be applied to account for the fact the binomial distribution describes
discrete populations. For large sample sizes, the correction will become
very small, as should be expected, because for large populations, the
normal distribution is a good approximation of the binomial one.

Table 5 below shows the appropriate z‑values and two‑tailed and
one‑tailed confidence limits for the recommended sample sizes of 5, 20,
and 30.

# Table 5: Confidence Intervals for Given Sample Sizes and Confidence Levels

  ---------- ------- ----------- ------- ------ ------------ -------- ------- --------- ----------- ------- ------ ------------ --------
                     One‑sided                                                          Two‑sided                               

                     z‑value                    Confidence                    z‑value                              Confidence   
                                                interval                                                           interval     

  Sample     90%                 95%     90%                 95%      90%                           95%     90%                 95%
  size                                                                                                                          

  5          1.282               1.645   0.33                0.39     1.645                         1.960   0.39                0.45

  20         1.282               1.645   0.14                0.17     1.645                         1.960   0.17                0.20

  30         1.282               1.645   0.11                0.14     1.645                         1.960   0.14                0.16
  ---------- ------- ----------- ------- ------ ------------ -------- ------- --------- ----------- ------- ------ ------------ --------

The confidence intervals above can be converted from proportions to
percentages by multiplying each proportion by 100. For the 90%
confidence level, confidence intervals (in percent) are between 33% and
39% for the smallest sample size and decrease to between 11% and 14% for
a sample size of 30. If an 80% accuracy estimate is assumed for a given
class, a two‑sided interpretation of the confidence interval with a
sample size of 20 and a confidence level of 90% implies that with 90%
certainty, the true accuracy value lies between 80% ± 17%. With a
one‑sided interpretation, it can be concluded that with 90% certainty,
the estimated value is equal to or greater than 66% (80%--14%).

Confirming whether or not the estimate conforms to a previously
established accuracy requirement is solved through hypothesis testing,
which is further explained in the section below.

**5.5 *Hypothesis Testing***

The purpose of hypothesis testing is to determine whether an estimation
function supports the assumption (hypothesis) that a sample has been
drawn from a population with specified parameter values, such as a
normal distribution with a given standard deviation (Mikhail and
Ackerman 1976). Therefore, a statistical hypothesis is a statement about
the probability distribution of a random variable. In general, any
statistical test requires the formulation of a null hypothesis (H~0~)
(theoretical expectation of a random variable) and an alternative
hypothesis (H~A~). The null hypothesis always specifies the set of
theoretical distribution parameters of a random variable against which
the parameters estimated from the sample are to be compared. The outcome
of this comparison determines whether the sample parameters are in
sufficient agreement with the null hypothesis. If this is the case, one
concludes that there is sufficient evidence to accept the null
hypothesis as true. Otherwise, the null hypothesis is rejected as false
and the alternative hypothesis accepted as true.

In terms of accuracy assessment for the NPS/NBS Vegetation Mapping
Project, two parameters are of interest:

> 1\. The estimate obtained for positional accuracy. Here, the objective
> is to determine whether or not the estimate is equal to or exceeds
> National Map Accuracy Standards.
>
> 2\. The estimates obtained for thematic accuracy, both overall and on
> a per‑class basis. Here the objective is to determine whether or not
> the estimate exceeds 80%.

5.5.1 Determining [c]{.smallcaps}onformance to NMAS for
[p]{.smallcaps}ositional [a]{.smallcaps}ccuracy

It is assumed that the National Cartographic Standards for Spatial
Accuracy will become the new national map accuracy standards, and
conformance to NMAS requires that 1:24,000 products have a standard
error of no more than 6.0 meters for Class 1 products (12.0 meters for
Class 2 products). In order to determine whether the estimated standard
error in either the x‑ or y‑ coordinate direction meets this accuracy
requirement, the null hypothesis and alternative hypothesis are
formulated as follows:

H~0~: The theoretical (required) standard error and the estimated
standard error for a given coordinate direction are the same. Under
these conditions, the random variable defined below:

follows a χ^2^ distribution with n‑1 degrees of freedom. σ~x~ is the
required accuracy standard, and RMSE~x~ is the estimated standard error.

H~A~: The estimated standard error is different from the required
standard error.

The data are accepted as meeting the accuracy requirement (i.e., H~0~ is
accepted) if the following is true:

![](media/image2.png){width="6.25in" height="0.6875in"}

or, if the computed χ^2^ test statistic is less than or equal to the
theoretical χ^2^ statistic at confidence level a with n‑1 degrees of
freedom. For a 95% confidence level (5% significance level) and a sample
size of 30 points, the maximum value the computed χ^2^ statistic may
assume is 42.557.

5.5.2 Determining [c]{.smallcaps}onformance to an 80%
[a]{.smallcaps}ccuracy [r]{.smallcaps}equirement for
[t]{.smallcaps}hematic\_[a]{.smallcaps}ccuracy

The way in which hypothesis tests are applied to thematic data depends
on the way accuracy is defined. Three approaches are possible (Goodchild
et al. 1994):

1\. The hypothesis that 80% accuracy has been met will be accepted
unless the sample map accuracy is low enough so that the conclusion that
rejection is appropriate can be drawn with some predetermined degree of
certainty.

2\. The hypothesis that 80% accuracy has been met will be rejected
unless the sample accuracy is sufficiently high so that the existence of
80% accuracy can be confirmed with some degree of certainty.

3\. A given sample map is assessed for accuracy, and the probability is
that the actual map accuracy exceeds the standard of 80%. This approach
is not really different from (1) and (2); it simply rephrases the
question of accuracy in probabilistic terms.

The first two approaches toward hypothesis testing can best be
illustrated by example. Suppose classification accuracy for a specific
class has been estimated from a sample of 20 points, 15 of which (75%)
have been properly identified. With the first approach, the aim of the
hypothesis test is to determine whether or not 75% is sufficiently low
to consider it to be significantly different from the required accuracy
of 80%. Whether or not 75% is sufficiently low depends on the required
level of confidence for the estimate. Since the objective is to
determine whether or not the estimate is sufficiently different from the
required one, a two‑sided confidence limit is in order. Table 5 above
indicates that the two‑sided 90% confidence interval for a sample size
of 20 points is 15.88 and therefore ranges from 59.12% to 90.88%. Since
the required map accuracy lies within this interval, it can be concluded
that the claimed and estimated accuracy are the same. Therefore, the
estimated classification accuracy meets requirements.

In formal terms, the null and alternative hypotheses would be formulated
as follows:

Required level of confidence: 90%

Sample size: 20

Estimated accuracy: 75%

Required accuracy: 80%

Confidence interval: 75% ± 15.88

H~0~: The estimated accuracy and the claimed accuracy are the same.
Under these conditions, the random variable comparing the estimated and
claimed accuracy,

> will follow a t‑ distribution with n‑1 degrees of freedom. In the
> equation, n is the sample size, p is the required accuracy, andp is
> the estimated accuracy.

H~A~: The estimated and claimed accuracies are different.

If the calculated value for t exceeds the threshold of 1.72 set for a
two‑tailed hypothesis test at a 90% confidence level, the null
hypothesis should be rejected. Otherwise, the null hypothesis should be
accepted. For the given example, t is calculated as 1.123. The claimed
and estimated values should therefore be accepted as the same.

With this approach to hypothesis testing, it is relatively easy to meet
accuracy requirements, especially with small sample sizes (and
correspondingly wide confidence intervals), because the sample map
accuracy must be low before the assertion of 80% accuracy can be
rejected. This approach encourages the use of small sample sizes
(Goodchild et al. 1994). To limit the width of the confidence interval,
a low level of significance should be chosen (0.10 or larger), in order
to limit the acceptable error in the sample.

A much stricter interpretation of accuracy is the requirement that the
required classification accuracy be met or exceeded at a given level of
confidence. This approach is also best illustrated by example. Suppose
that classification accuracy for a given class has been estimated at 90%
with a sample size of 20. We wish to test the hypothesis that the 90%
accuracy estimate meets or exceeds the required 80%. In formal terms

Required level of confidence: 90%

Sample size: 20

Estimated accuracy: 90%

Required accuracy: 80% or greater

Confidence interval: 90% ± 12.2%

H~0~: The estimated accuracy is less than the required accuracy.

H~A~: The estimated accuracy is equal to or greater than the required
accuracy.

If the calculated value for t exceeds the 90% threshold (1.32) for this
one‑sided test, the null hypothesis should be rejected, and the
alternative accepted. However, the t‑statistic under these conditions is
1.118. Therefore, the alternative cannot be accepted. Even though the
estimated accuracy value was considerably higher, the width of the
confidence interval prevents a conclusion with sufficient certainty that
the estimated value meets or exceeds the required one. Clearly this
approach favors large sample sizes, in order to narrow confidence
intervals (and therefore increase the chance that a given point estimate
lies above the threshold established by the upper limit of the
confidence interval). For example, with a sample size of 30, the
t‑statistic increases to 1.34. This exceeds the threshold of 1.32 at a
90% confidence level and would therefore permit the alternative
hypothesis to be accepted. Again it is recommended that a low
significance level (90% or lower) be used, because this will make it
easier to achieve the required accuracy standard.

5.6 Summary of Recommendations Regarding Confidence Levels and
Hypothesis Tests

Meeting accuracy requirements at the second, more demanding level will
require that the point estimate be much higher than the required
accuracy, especially if sample sizes are small. At the same time, it is
unlikely that sample sizes on a per‑class basis can be increased much
beyond 30_samples per class, even for the most abundant classes. In
fact, it is strongly recommended that sample sizes of less than 30 be
used for rare classes. Since the majority of classes are in fact
expected to be rare, sample sizes of 20 or less will be the more common
scenario. With these sample sizes, meeting or exceeding the required
accuracy will be nearly impossible. On a per‑class basis, it is
therefore recommended that the less stringent approach be used to
confirm accuracy. In other words, the accuracy requirement will be
considered met, if it can be concluded with sufficient certainty that
the sample accuracy and the required accuracy are in fact the same
(two‑sided approach). In addition, it is recommended that 90% confidence
intervals be computed on a per‑class basis.

Although sample sizes on a per‑class basis are small, it is likely that
overall sample size for thematic accuracy assessment will be
sufficiently large to draw more rigorous conclusions regarding overall
classification accuracy. It is therefore recommended that conditions be
defined for meeting overall accuracy by using the more demanding
requirement of either meeting or exceeding the established accuracy
standard. Again it is recommended that 90% be used as the required level
of confidence.