# Tests of Fit for the Logarithmic Distribution

Authors: D. J. Best; J. C. W. Rayner; O. Thas
Journal: Journal of Applied Mathematics and Decision Sciences
 (2008)

---

## Abstract

Smooth tests for the logarithmic distribution are compared with three tests: the first is a test due to Epps and is based on a probability generating function, the second is the Anderson-Darling test, and the third is due to Klar and is based on the empirical integrated distribution function. These tests all have substantially better power than the traditional Pearson-FisherX2 test of fit for the logarithmic. These traditional chi-squared tests are the only logarithmic tests of fit commonly applied by ecologists and other scientists.


---

## Body

## 1. Introduction

Species
diversity data can sometimes be modeled by a zero-truncated negative binomial
distribution with index parameter near zero. Fisher et al. [1] examined the
limit as the index parameter of this distribution approached zero and so
derived the logarithmic distribution. A random variable X has this distribution if and only if
(1.1)P(X=x)=px=γβxx,x=1,2,3,… in which 0 < β < 1 and γ=−1/ln(1−β). The logarithmic or log-series distribution is often applied to
species diversity data.As an example of  species diversity data which the
logarithmic distribution may fit, consider the following data on insect catches
from the Sierra Tarahuma, Mexico, reported by Aldrete [2]. Ten species were
caught precisely once, three species were caught precisely twice, and so on
according to Table 1. The expected line in Table 1 shows the expected counts on
fitting a logarithmic distribution. For these data, the alpha index (AI) of diversity is 9.01, where AI=n(1−β^)/β^ in which n is the total number of insects and β^ is the maximum likelihood estimator of β. The AI quoted here is defined in Krebs [3, 12.13]; for a discussion of the index of diversity see Krebs [3, Section 12.4.1]. Note that larger AI implies more diversity while smaller AI implies less.Table 1
Catch frequencies per species and corresponding expected
values assuming a logarithmic model for the Aldrete [2] data.
Times caught12345681011121316256995At least 13No. of species10342221111112116Expected8.784.282.782.031.581.281.070.910.620.55—————7.63It would seem sensible to test the data for consistency with the
logarithmic distribution before quoting anAI value. However, the only statistic that appears to be commonly used by
ecologists as a test of fit for the logarithmic distribution is the so-called
chi-squared test, which, as Krebs [3, Section 12.4.1] notes, may not always
have good power.“The goodness-of-fit of the logarithmic series… can be
tested by the usual chi-squared goodness-of-fit test … this means low power ….
Thus in most cases the decision to use the logarithmic series … must be made on
ecological grounds, rather than statistical goodness-of-fit criteria.” [3, page 429].In this paper, we will examine a number of statistical tests which
are considerably more powerful than the traditional Pearson-FisherX2 test. These include tests of fit based on components of Neyman’s smooth test
statistic, the Anderson-Darling test discussed by Lockhart et al. [4], an
empirical integrated distribution function test given by Klar [5], and a test
due to Epps [6] based on a probability generating function (pgf). We suggest
that these could be used to help make a decision as to whether or not to use
the logarithmic series based on statistical as well as ecological criteria. In
particular, the dispersion statistic, V^2,
defined subsequently, should be useful for identifying the not infrequent case
of data for which the abundance species are more abundant than predicted by the
logarithmic series.Our second example is included for its somewhat curious interest and
is not involved with conventional species diversity. Collins and Hand [7]
have counted the number of times, in the period 1983 to 2000, that a Hollywood
film won one Oscar, two Oscars, three Oscars, and so on, giving the data in
Table 2. The film with 11 Oscars was “Titanic.”Table 2
Oscar frequencies per film 1983 to 2000 and corresponding
expected values assuming a logarithmic model.
Oscars1234567891011At least 9No. of films1112914961412013Expected103361795321———2


## 2. Tests of Fit for the Logarithmic

A discussion of smooth tests of fit and their components,
particularly when testing for the logarithmic, is given in AppendixA. These
tests may be derived as a routine application of Rayner and Best [8, Theorem
6.1.1]. The first-order component V^1 is identically zero when β is estimated by maximum likelihood, or,
equivalently, by method of moments. The test based on the component V^2 suggests whether or not the data are
consistent with the logarithmic variance while the test based on V^3 suggests whether or not the data are
consistent with logarithmic moments up to the third. To find p-values for these
tests, it is suggested that the parametric bootstrap is to be used as convergence to the
asymptotic standard normal distribution is very slow. See Gürtler and Henze [9] and Appendix B for details of the parametric bootstrap in a goodness of
fit context.In Section3, we give powers for the Anderson-Darling
test based on the statistic
(2.1)A2=∑j=1∞Zj2pjhj(1−hj) in which Zj=∑x=1j(Ox−npx),hj=∑x=1jp^x,
and Ox is the number of
observations equal to x. Summation is
halted when x is the minimum such
that Ox = 0 and ∑j=x∞p^j<10−3/n. We also give powers for a test given by Klar [5], based on
the empirical integrated distribution function with test statistic
(2.2)Tn=nsup1≤k≤M|∑j=1kZj| in which M is the largest
observation. Finally, for comparison purposes, we quote powers of the pgf and X2 tests given by Epps
[6].


## 3. Power Comparisons

Random deviates from the logarithmic (L), positive Poisson (P+), and
positive geometric (G+) distributions were generated using IMSL [10] routines
RNLGR, RNPOI, and RNGEO. Random zeta deviates (Z) and random Yule deviates (Y) were
found using algorithms of Devroye [11, pages 551 and 553]. Table 3 gives powers
for the same alternatives as used by Epps [6], but with the addition of two
Yule alternatives. For convenience, we reproduce the powers given by Epps for
his pgf and X2 tests. The
powers we give for A2, Tn, V^22, V^32, and V^22 + V^32 were found using parametric bootstrap with 1000
simulations both for the inner and the outer loops. Note that the calculation
of V^3 can involve large numbers, and calculation of
the pgf and A2 statistics
can involve small numbers. Care with rounding error may be needed. The
statistics Tn and V^22 are less prone to rounding error. Klar [5]
notes that the smooth tests, the X2 test, and the pgf test are not consistent against all alternatives.Table 3
Powers of some tests for the logarithmic distribution withn = 50 and α = 0.05.
AlternativeV^22V^32V^22 + V^32PGFX2TnA2P+(1.05)0.390.540.510.560.430.500.57P+(1.2)0.490.680.630.720.580.610.73P+(1.3)0.570.780.690.790.630.680.79G+(0.25)0.450.070.050.780.510.520.77G+(0.33)0.320.370.300.610.390.380.58G+(0.4)0.210.390.330.460.300.300.48Z(1.0)0.850.370.540.840.080.730.72Z(1.3)0.740.630.720.720.160.680.69Z(2.0)0.500.380.430.440.170.480.42Y(2.75)0.550.380.50——0.490.43Y(3.0)0.500.350.45——0.460.35From Table3 our powers for Tn are a little greater than those of Klar [5], and
we observe that the power for the Z(1.0) alternative is 0.73, somewhat larger
than the 0.40 reported by Klar [5]. Also from Table 3, we see that the X2 test is not generally
competitive with the other tests.The test based onV^22 + V^32 performs reasonably well. The test based on
the Tn statistic has power
a little less than that for the pgf- and A2-based
tests. An advantage of the test based on Tn is that Klar [5] showed it is consistent.The test based on the dispersion statisticV^22 has good power for the zeta and Yule alternatives,
while the A2 and pgf tests
generally have competitive powers for all alternatives. Clearly, the test based
on V^22 will not have good power for alternatives with
similar dispersion to the logarithmic distribution. If the test based on V^22 is not significant but that based on V^32 is, this suggests a skewness departure from the logarithmic distribution. However, if the test
based on V^22 is significant, then this suggests that the test based on V^32 may be significant due to either a dispersion
or a skewness departure of the data from the logarithmic distribution. Notice that
we say the test based on V^32suggests how the data deviate from the logarithmic. We do not claim that the data
actually do deviate in this manner. See the comments of Henze and Klar [12].On the basis of Table3 powers, we suggest that the tests
based on A2 and Klar’s Tn are considered as tests of
fit for the logarithmic distribution. These tests have good power and are consistent.
We recommend that the tests based on Tn and A2 are augmented by
the use of V^22 and V^32 in a data analytic fashion.


## 4. Examples

In the following parametric
bootstrap,p-values for the tests based on V^22, V^32, A2, and Tn are given. These
use 1000 random samples of the logarithmic distribution with parameter β^ as given below. We
give V^2 and V^3 values because they may suggest how the data deviate from the logarithmic. We give the A2 and Tn values because the tests based on these statistics
are consistent and have good power.


### 4.1. Insect Data

From the data in Table1, we find β^=0.9743,AI=9.0104,V^2=0.4879 with p-value 0.52, V^3=−0.8791 with
p-value 0.19, A2=0.2613
with p-value 0.82, and Tn=204.8192 with p-value 0.33. It appears that the logarithmic distribution is a
good fit. In agreement with this, the Pearson-Fisher statistic takes the value
4.56 on 11 degrees of freedom when data greater than 12 have been combined.


### 4.2. Oscars Data

We find thatV^2=0.49,V^3=−1.27, and X2=7.40 on 7 degrees of
freedom if the classes greater than or equal to 9 are combined. The corresponding
p-values are 0.58, 0.08, and 0.62. It appears that a logarithmic distribution
with β^ = 0.7044 fits the data
reasonably well. However, the p-value for V^3 suggests that the data may not be quite as
skewed as would be expected for the logarithmic distribution. Collins and Hand
[7] suggest a Yule distribution fits the data well. In addition, we note
that A2=0.9727 with
p-value 0.12 and Tn=120.0203 with p-value 0.26.


## 4.1. Insect Data

From the data in Table1, we find β^=0.9743,AI=9.0104,V^2=0.4879 with p-value 0.52, V^3=−0.8791 with
p-value 0.19, A2=0.2613
with p-value 0.82, and Tn=204.8192 with p-value 0.33. It appears that the logarithmic distribution is a
good fit. In agreement with this, the Pearson-Fisher statistic takes the value
4.56 on 11 degrees of freedom when data greater than 12 have been combined.


## 4.2. Oscars Data

We find thatV^2=0.49,V^3=−1.27, and X2=7.40 on 7 degrees of
freedom if the classes greater than or equal to 9 are combined. The corresponding
p-values are 0.58, 0.08, and 0.62. It appears that a logarithmic distribution
with β^ = 0.7044 fits the data
reasonably well. However, the p-value for V^3 suggests that the data may not be quite as
skewed as would be expected for the logarithmic distribution. Collins and Hand
[7] suggest a Yule distribution fits the data well. In addition, we note
that A2=0.9727 with
p-value 0.12 and Tn=120.0203 with p-value 0.26.


## 5. Conclusion

In this paper, we have examined a number of statistical tests which
are considerably more powerful than the traditional Pearson-FisherX2 test. We suggest that these could be used to help make a decision as to whether
or not to use the logarithmic series based on statistical as well as ecological
criteria. A test of fit could be done before quoting the index of diversity. In
particular, the dispersion statistic, V^2,
should be useful for identifying the not infrequent case of data for which the
abundance species are more abundant than predicted by the logarithmic series.



---

*Source: 463781-2008-03-19.xml*