# A Cluster Truncated Pareto Distribution and Its Applications

Authors: Mei Ling Huang; Vincenzo Coia; Percy Brill
Journal: ISRN Probability and Statistics
 (2013)

---

## Abstract

The Pareto distribution is a heavy-tailed distribution with many applications in the real
world. The tail of the distribution is important, but the threshold of the distribution is difficult
to determine in some situations. In this paper we consider two real-world examples with heavy-tailed observations, which leads us to propose a mixture truncated Pareto distribution (MTPD)
and study its properties. We construct a cluster truncated Pareto distribution (CTPD) by using
a two-point slope technique to estimate the MTPD from a random sample. We apply the MTPD
and CTPD to the two examples and compare the proposed method with existing estimation
methods. The results of log-log plots and goodness-of-fit tests show that the MTPD and the
cluster estimation method produce very good fitting distributions with real-world data.


---

## Body

## 1. Introduction

There are many real-world problems modelled as heavy-tailed distributions, especially the Pareto distribution [1, 2]. However, there are some difficulties in estimation of Pareto distributions. First, the Pareto distribution has infinite moments in some heavy-tailed cases. Therefore the moment estimation method for the shape parameter cannot be used in these situations. It is a loss for the estimation process since the moment estimator is a robust estimator. Several authors suggest using a truncated Pareto distribution which always has finite moments (e.g., [3–5]).In some situations, data will behave differently within different thresholds. For example, losses from hurricane damage can be classified into small, medium, and large hurricane groups. The data in these classes may have different distributions, or by grouping, data with self-similarity may have the same kind of distribution but with different parameters. A cluster method for data is needed to determine these groups when dealing with real data sets. In this paper we study an example of 49 most damaging Atlantic hurricanes occurring between years 1900 and 2005 [6]. The costs are standardized to 2005 USD; see Figure 1.Figure 1
The 49 costliest Atlantic hurricanes between the years 1900 and 2005.Coia and Huang [7] applied Pareto and truncated Pareto models to fit the hurricane data set. The maximum likelihood estimator (MLE) and the moment estimator for the shape parameter were used. The results are shown in a log-log plot in Figure 2. Coia and Huang [7] also used Kolmogorov-Smirnov, Anderson-Darling, and Cramer-von-Mises goodness-of-fit tests. We note that the two estimated (by MLE and moment method) truncated Pareto curves fit the data set quite well; they fit much better in the tail than the original Pareto distribution (which is in a straight line). But the truncated Pareto curves do not fit the data uniformly well, especially for the middle value data. We observed that the pattern of data can be classified into three groups. The data in these three groups may still be Pareto distributed but with different shape parameters. In the literature, researchers study similar data sets by using cluster methods; for example, Coia and Huang [7] proposed a sieve model.Figure 2
Log-log plot of hurricane example with estimated distribution curves. The red circles are the data; the black straight line is the original Pareto distribution; the green dot line is the MLE estimated truncated Pareto distribution; the brown dash line is the moment estimated truncated Pareto distribution.In this paper, we propose a more generalized method—mixture truncated Pareto distribution (MTPD)—in Section2. We study the properties of the MTPD in Section 3. In Section 4, we propose a cluster method by using a two-point slope technique to estimate the MTPD from data which utilizes a cluster truncated Pareto distribution (CTPD). In Section 5, we review the nonparametric kernel density estimation method. In Section 6, we analyze the hurricane data and a second example regarding sizes of fish caught off the Atlantic shore of Massachusetts, USA, by using the CTPD, nonparametric kernel estimation method and three other existing semiparametric estimation methods in log-log plots (see Figure 3 in Section 6). We also perform Kolmogorov-Smirnov, Anderson Darling, and Cramer-von Mises goodness-of-fit tests on these two data sets. The results show that the proposed cluster method and nonparametric method are superior to other existing estimation methods, in both examples.Figure 3
Log-log plot of hurricane example with estimated distribution curves. The red circles are the data; the black straight line is the original Pareto distribution; the green dot line is the MLE estimated truncated Pareto distribution; the brown dash line is the moment estimated truncated Pareto distribution; the thick green dash line is the kernel estimated distribution; the thick blue line is the cluster truncated Pareto distribution.


## 2. Mixture Truncated Pareto Distribution

Definition 1.
The probability density function (p.d.f.) and the cumulative distribution function (c.d.f.) of a random variableY having the Pareto distribution are given by
(1)fP(y;γ,α)=αγαy(α+1),0<γ≤y<∞,α>0,(2)FP(y;γ,α)=1-(γy)α,0<γ≤y<∞,α>0,
where α is the shape parameter.When0<α≤1, which is a heavy-tailed case, the mean and variance of Y are infinite, and the distribution is heavier in the right tail as α decreases.The truncated Pareto distribution (TPD) was originally used to describe the distribution of oil fields by size. It has a lower limitγ, an upper limit ν, and a shape parameter α. In fact, it has been shown that the truncated Pareto distribution fits better than the nontruncated Pareto distribution for some positively skewed populations [3].Definition 2.
The p.d.f. and c.d.f. of a random variableX having the truncated Pareto distribution are given by
(3)f(x;γ,ν,α)=αγαx-α-11-(γ/ν)α,0<γ≤x≤ν<∞,α>0,(4)F(x;γ,ν,α)=1-γα(x-α-ν-α),1-(γ/ν)α,0<γ≤x≤ν<∞,α>0,
where γ and ν are the left and right truncation points.The quantile function of the truncated Pareto distribution is(5)F-1(u)=(1-uγα+uνα)-1/α,0≤u≤1,α>0.The mean, second moment, and variance ofX are, respectively,
(6)μ=αγα(γ1-α-ν1-α)(α-1)(1-(γ/ν)α),0<γ<ν<∞,α>1,(7)μ(2)=αγα(ν2-α-γ2-α)(2-α)(1-(γ/ν)α)2,0<γ<ν<∞,α>2,(8)σ2=αγα(ν2-α-γ2-α)(2-α)(1-(γ/ν)α)2-α2γ2α(ν1-α-γ1-α)2(1-α)2(1-(γ/ν)α)2,0<γ<ν<∞,α>2.We consider a vector of thresholds(9)T=(t0,t1,…,tk)T,where0<a=t0<t1⋯<tk=b<∞,a,b∈ℜ,k=1,2,….
Consider a vector Λ=(α1,α2,…,αk)T, αi>0, i=1,…,k. We define a mixture truncated Pareto distribution as follows.Definition 3.
The c.d.f. of a random variableX having a mixture truncated Pareto distribution (MTPD) is given by
(10)FMTP(x;T,Λ;W)=∑i=1k‍wiFi(x;ti-1,ti,αi),0<a≤x≤b<∞,a=t0,b=tk,
where Fi(x;ti-1,ti,αi) is the c.d.f. of the truncated Pareto distribution in (4), the truncation points ti-1, ti, are related to thresholds T=(t0,t1,…,tk)T, and W is a vector of weights
(11)W=(w1,w2,…,wk)T,0≤wi≤1,∑i=1k‍wi=1.
The p.d.f. of a mixture truncated Pareto distribution is given by
(12)fMTP(x;T,Λ;W)=∑i=1k‍wifi(x;ti-1,ti,αi),0<a≤x≤b<∞,a=t0,b=tk,
where fi(x;ti-1,ti,αi) is the p.d.f. of the truncated Pareto distribution in (3).


## 3. Properties of Mixture Truncated Pareto Distribution

Proposition 4.
The meanμMTP and variance VarMTP of a mixture truncated Pareto distribution are given by
(13)μMTP=∑i=1k‍wiμi,VarMTP=∑i=1k‍wiμ(2)i-(∑i=1k‍wiμi)2,
where μi and μ(2)i are the mean and the second moment of the fi(x;ti-1,ti,αi) in (6) and (7), i=1,2,…,n.Proof.
(14)
μ
MTP
=
∫
γ
ν
x
∑
i
=
1
k
‍
w
i
f
i
(
x
;
t
i
-
1
,
t
i
,
α
i
)
d
x
μ
MTP
=
∑
i
=
1
k
‍
w
i
∫
t
i
-
1
t
i
x
f
i
(
x
;
t
i
-
1
,
t
i
,
α
i
)
d
x
μ
MTP
=
∑
i
=
1
k
‍
w
i
μ
i
,
μ
(
2
)
MTP
=
∫
γ
ν
x
2
∑
i
=
1
k
‍
w
i
f
i
(
x
;
t
i
-
1
,
t
i
,
α
i
)
d
x
μ
(
2
)
MTP
=
∑
i
=
1
k
‍
w
i
∫
t
i
-
1
t
i
x
2
f
i
(
x
;
t
i
-
1
,
t
i
,
α
i
)
d
x
μ
(
2
)
MTP
=
∑
i
=
1
k
‍
w
i
μ
(
2
)
i
.Proposition 5.
For givenp,  0<p<1, the quantile function of a mixture truncated Pareto distribution in (10) is given by solving for x in the following equation:
(15)∑i=1k‍wiFi(x;ti-1,ti,αi)=p,0<a≤x≤b<∞.


## 4. A Cluster Truncated Pareto Distribution Estimator

Consider a random sampleX1,X2,…,Xn from the MTPD in (10), and let X1,n≤X2,n≤⋯≤Xn,n denote its order statistics. We divide data into k clusters by the domains (ti,ti+1), i=0,1,…,k-1,  T=(t0,t1,…,tk)T, where 0<a=t0<t1<⋯<tk=b<∞, a,b∈ℜ. We define a cluster truncated Pareto distribution (CTPD) as an estimator of the MTPD.Definition 6.
The c.d.f. of a random variableX having the cluster truncated Pareto distribution (CTPD) is given by
(16)FC(x;T,Λ;W)=∑i=1k‍(nin)Fi(x;ti-1,ti,αi),0<a≤x≤b<∞,
where FC(x;T,Λ;W) is a c.d.f. of the MTPD in (10) and ni is the sample size in the ith cluster in the ith domain: (ti-1,ti):
(17)W=(n1n,n2n,…,nkn)T,wi=nin,i=1,2,…,k,
where ni’s  depend on the vector C=(c0,c1,…,ck)T, where 0=c0<c1<⋯<ck=n,  ci is the number of data less than or equal to the threshold ti. Note that ci is a function of ti and the random sample (X1,X2,…Xn). Thus
(18)ci(ti;X1,X2,…,Xn)=∑j=1n‍I(-∞,ti](Xj),i=1,2,…,k-1,ni=ci-ci-1,i=1,2,…,k,
where IA is the indicator function of set A.We also note thatci and ni all depend on ti, i=1,…,k in (16). The key point of applying the CTPD in (16) is to determine T=(t0,t1,…,tk)T from the random sample. In this paper, we propose a two-point slope technique in the log-log plot to estimate thresholds T=(t0,t1,…,tk)T.Definition 7.
A two-point slope is defined as(19)Si(X1,X2,…,Xn)={log(1-(i+1)/n)-log(1-i/n)log(Xi+1,n)-log(Xi,n),log(Xi+1,n)-log(Xi,n)≠0,i=1,…,n-1;0,log(Xi+1,n)-log(Xi,n)=0,i=1,…,n-1.Then we construct(n-1) order statistics S1,n-1≤S2,n-1≤⋯≤Sn-1,n-1 from the absolute values of the two-point slopes |Si(X1,X2,…,Xn)|, i=1,2,…,n-1. The cluster threshold points can be estimated by t^1(X1,X2,…,Xn),…,t^k-1(X1,X2,…,Xn) which are determined by the (k-1) largest absolute values of the two-point slopes
(20)Sn-k+1,n-1≤Sn-k+2,n-1≤⋯≤Sn-1,n-1,
where k depends upon empirical observations of differences between successive Si,n-1’s. This usually occurs when |Sn-k+2,n-1-Sn-k+1,n-1| is large compared with previous differences.We propose seven steps to construct a cluster truncated Pareto distribution in (16).Step
1. Compute (n-1) two-point slopes Si(X1,X2,…,Xn) in (19), i=1,…,n-1.Step
2. Determine k by using (20); there are two main factors:(1)
Determiningk depends upon empirical observations of differences between successive Si,n-1’s, when |Sn-k+2,n-1-Sn-k+1,n-1| is much larger than the previous difference |Sn-k+1,n-1-Sn-k,n-1|. (This technique is used on the two examples in Section 6.)(2)
We also ensure that the sample sizeni within each group is sufficiently large (usually ni≥5).Step
3. Find the (k-1) estimated threshold points t^1,…,t^k-1 by using the values of the (k-1) largest absolute slopes of the order statistics of |Si(X1,X2,…,Xn)| in (20), i=1,…,n-1, corresponding to the (k-1) values {X1*,X2*,…,Xk-1*} of the original sample, which now have been ordered as new order statistics
(21)X1,k-1*≤X2,k-1*≤⋯≤Xk-1,k-1*.
Then we let
(22)t^i(X1,X2,…,Xn)=Xi,k-1*,i=1,…,k-1,t^0=X1,n=a,t^k=Xn,n=b.Step
4. Determine C=(c0,c1,…,ck)T, where 0=c0<c1<⋯<ck=n, ci(ti;X1,X2,…,Xn)=∑j=1n‍I(-∞,ti](Xj). Thus
(23)ni=ci-ci-1,i=1,2,…,k,t^i=Xci,n,i=1,…,k-1,(notethatwereplaceXci,n=Xi,k-1*in(22)),t^0=X1,n=a,t^k=Xn,n=b.
Then we have k clusters:
(24){a=t^0,…,Xc1,n},{Xc2,n,…,Xc3,n},…,{Xck-1,n,…,t^k=Xn,n=b}.
This construction is shown in Box 1.Box 1:Construction of cluster truncated Pareto distribution from the data.Step
5. Construct FC~(x;T^,Λ;W^)=∑i=1k(ni/n)Fi(x;t^i-1,t^i,αi), in (16).Step
6. Estimate α^i,Moment. We suggest using the moment estimator α^M in (26) since it has robust properties, but there are other estimators available in (25) and (27).Step
7. Construct an estimator FC^(x;T^,Λ^;W^)=∑i=1k(ni/n)Fi(x;t^i-1,t^i,α^i), for (16).Remark 8.
There are three estimation methods for the shape parametersαi for all sub-samples, given by the following.
(1) Hill Estimator (original Pareto distribution in (1)). The Hill [8] MLE α^Hill for α is defined as
(25)α^Hill=[r-1∑‍ri=1{lnXn-i+1,n-lnXn-r,n}]-1,
where Xi,n is the ith smallest order statistic and r is the cut-off point.
(2) Moment Estimator (truncated Pareto distribution in (3)). A moment estimator α^M for α can be obtained by solving the following equation:
(26)1n∑‍ni=1Xi=α^Mγα^M(γ1-α^M-ν1-α^M)(α^M-1)(1-(γ/ν)α^M),
where 0<γ≤Xi≤ν<∞, α^M>0.
(3) MLE Method (truncated Pareto distribution in (3)). The Aban MLE α^Aban [4] for α is obtained by solving the following equation:
(27)nα^Aban+n(γ/ν)α^Abanln(γ/ν)1-(γ/ν)α^Aban-∑‍ni=1[lnXn-i+1,n-lnγ]=0,
where Xi,n is the ith smallest order statistic,  γ≤Xi,n≤ν, i=1,2,…,n.
Note that, before using (26) and (27), the parameters γ and ν may be estimated by γ^=X1,n, ν^=Xn,n, which are used in this paper. There are other nonparametric methods in the literature. Cooke [9] proposed estimators
(28)γ^=2X1,n-∑i=0n-1‍[(1-in)n-(1-i+1n)n]Xi+1,n,ν^=2Xn,n-∑i=0n-1‍[(1-in)n-(1-i+1n)n]Xn-i,n.


## 5. Nonparametric Kernel Density Estimation

We apply the kernel density estimation method (KDE) [10]. It is a smoothing technique for estimating distributions. It is well known that the classical kernel density estimator from a random sample X1,X2,…,Xn for true probability density function (p.d.f.) f is
(29)fn^(x)=1nh∑i=1n‍K(x-Xih),x∈ℜ,
where K(•) is a symmetric density function and h>0 is a bandwidth.We will compare the KDE estimator and the CTPD estimator in the next section.


## 6. Applications

Now we apply the cluster truncated Pareto distribution to the hurricane example in Section1.


### 6.1. Cluster Method

By using 48 two-point slopes in (19) and the seven steps in Section 4, we construct k=3 clusters. We select the k-1=2 largest absolute values of the two-point slopes Sn-2,n-1≤Sn-1,n-1 in (20) as
(30)S48,48=14.2027,S47,48=5.6574,
since the third largest two-point slope S46,48=5.2276 does not have a large change compared with  S47,48=5.6574.Then we determinet^i’s and k=3 groups as
(31){a=t^0=X1,n,…,Xc1,n},{Xc1+1,n,…,Xc2,n},{Xc2+1,n,…,t^3=Xn,n=b},
where  t^0=X1,n=3.7=a, t^1=Xc1,n=13.7, t^2=Xc2,n=21.29, and t^3=Xc3,n=157=b; c0=0, c1=27, c2=40, and c3=49; n1=27, n2=13, and n3=9; n1+n2+n3=n=49; then we have an estimated CTPD in (16) as
(32)FC^(x;T^,Λ^;W^)=∑i=13‍(nin)Fi(x;t^i-1,t^i,α^i).
This construction is shown in Box 2.Box 2:Construction of cluster truncated Pareto distribution from the data.


### 6.2. Kernel and Other Estimation Methods in Log-Log Plot

We apply the kernel estimator in (29) which is normalized to the hurricane data. Here, we use a standard normal kernel and optimal bandwidth [10, p. 45]
(33)hopt=(43n)1/5≈1.05922n-1/5.
We ensure that the bandwidth is sufficiently large such that the estimated tail distribution is smooth enough.Table1 gives α^, μ^, Median, 5% Value-at-Risk (VaR), and 1% VaR of each of four estimation methods. We note that the cluster method and kernel method give the largest medians. The cluster method gives the smallest VaRs.Table 1
Five estimation methods for hurricane example.
Method
α
^
μ
^
Median
5
% VaR
1
% VaR
Pareto(Hill)
0.8126
∞
8.68 billion
147.68 billion
1070.30 billion
TPD(Aban)
0.6206
21.10 billion
9.73 billion
85.15 billion
136.17 billion
TPD(moment)
0.6476
20.48 billion
9.47 billion
82.55 billion
134.90 billion
Kernel
—
19.88 billion
12.88 billion
80.58 billion
156.14 billion
Cluster
α
^
1 = 0.6476
27.65 billion
11.19 billion
77.24 billion
132.41 billion
α
^
2 = 5.6498
α
^
3 = 0.8416Figure3 is a log-log plot which exhibits data and five estimated distribution curves. We note that the original Pareto distribution does not fit data well in the right tail. The moment and Aban estimated truncated Pareto distribution (TPD) fit data well in the right tail, but not so well in the smaller or middle values data. The cluster truncated Pareto distribution overcomes this problem and has the best fit to the data over the whole range. Figure 3 suggests a single distribution may not totally represent how natural data is distributed. We may consider grouping data by using the cluster method. Figure 3 also shows that the nonparametric kernel estimated distribution fits the data well.Figure3 provides a visual observation. It is necessary to run goodness-of-fit tests mathematically to confirm which estimated distribution best fits the hurricane data.


### 6.3. Goodness-of-Fit Tests

In this section we conduct three goodness-of-fit tests, Kolmogorov-Smirnov, Anderson Darling, and Cramer-von Mises. All three tests are based on the distance between the empirical distribution function and the proposed distribution function: original Pareto distribution in (1) or truncated Pareto distribution in (3) or mixture truncated Pareto distribution in (10).Each test considers the same null and alternative hypothesis:(34)H0:F(x)=F*(x)versusH1:F(x)≠F*(x),
where F(x) is the unknown true distribution of the sample data and F*(x) is one of our proposed four estimated distributions:(1)
Pareto distribution in (1) with Hill estimator α^Hill in (25);(2)
truncated Pareto distribution (TPD) in (3) with Aban estimator α^Aban in (27);(3)
truncated Pareto distribution in (TPD) (3) with moment estimator α^M in (26);(4)
kernel estimated distribution in (29);(5)
cluster truncated Pareto distribution in (16) with moment estimator α^M(i) in (26).We ran a test for each estimated distribution as F*(x).(1) The Kolmogorov-Smirnov (K-S) Test [11]. The test statistic is given by
(35)T=supx|F*(x)-Sn(x)|,-∞<x<∞,
where Sn(x)=(1/n)∑i=1n‍I(-∞,x](Xi) is the empirical distribution function. Under H0 the two-tailed P value for the K-S test is as follows:
(36)Pvalue=1-[1-T∑j=0[n(1-T)]‍n!j!(n-j)!(1-T-jn)n-j(T+jn)j-1]2,
where [n(1-T)] is the integer part of n(1-T).(2) Anderson and Darling Test (A-D Test). Anderson and Darling [12] introduced a measure of “distance” between the empirical distribution Sn(x) and the proposed c.d.f. F*(x) by using a metric function space,
(37)Wn2=n∫-∞∞[Sn(x)-F*(x)]2ψ(F*(x))dF*,
where ψ(u)=1/(u(1-u)), with u=F*(x). Let ui=F*(Xi,n); assume under H0 the test statistic and P value are given by
(38)Wn2=-n-1n∑j=1n‍(2j-1)(log(uj)+log(1-un-j+1)),Pvalue=1-2z∑j=0∞‍{(-1)jΓ(j+1/2)(4j+1)j!e-(4j+1)2π2/8/z·∫0∞e(z/8/(w2+1)-(4j+1)2π2w2/8/z)dw(-1)jΓ(j+1/2)(4j+1)j!},
where z is the observed value of  Wn2  and  Γ(x)=∫0∞tx-1e-tdt,  x>0, is the Gamma function.(3) Cramer-von Mises Test (C-v-M Test). Anderson and Darling [12] proposed this test by using ψ(u)=1 in (37). Thus under H0 the test statistic and P value are given by
(39)nω2=112n+∑j=1n‍(uj-2j-12n)2,Pvalue=1-1πz∑j=0∞{(-1)2jΓ(j+1/2)(4j+1)1/2j!πe-(4j+1)2/16/z‍·K1/4((4j+1)216z)},
where K1/4(x) is the modified Bessel function of the second kind,
(40)Kα(x)=π2I-α(x)-Iα(x)sin(απ),Iα(x)=∑m=0∞‍1m!Γ(m+α+1)(x2)2m+α.Table2 gives the values of the test statistics and P values of three goodness-of-fit tests. Note that the cluster truncated Pareto distribution has the smallest test statistics in the K-S test (i.e., the smallest errors) and the largest P values. The kernel estimated distribution gives the smallest test statistics in the A-D test and C-v-M test, respectively. This means the cluster truncated Pareto distribution and kernel estimated distribution have the best fit to the hurricane data.Table 2
Goodness-of-fit testsn=49 for hurricane example.
Method
Goodness-of-fit tests
K-S test
A-D test
C-v-M test
Test statistic
P value
Test statistic
P value
Test statistic
P value
Pareto(Hill)
0.1340
0.2900
2.7141
0.0383
0.2057
0.2568
TPD(Aban)
0.0948
0.6282
2.3126
0.0622
0.0964
0.6030
TPD(moment)
0.1053
0.5308
2.3672
0.0582
0.1095
0.5402
Kernel
0.0890
(
2
)
0.6805
(
2
)
2.2344
(
1
)
0.0685
(
1
)
0.0427
(
1
)
0.9187
(
1
)
Cluster
0.0741
(
1
)
0.8035
(
1
)
2.2584
(
2
)
0.0665
(
2
)
0.0467
(
2
)
0.8958
(
2
)
Note. In this paper, we use (1) to denote the best values and (2) to denote the second-best values in the tables.In Table3, we took the r largest data in the sample. The absolute error and integrated error are defined by
(41)AE=supx|F*(x)-Sn(x)|,-∞<x<∞,IE=1(Xn,n-Xn-r+1,n)[∫Xn-r+1,nXn,n‍(Sn(x)-F*(x))2dx]1/2.Table 3
Errors of goodness-of-fit testsn=49 for hurricane example.
Method
Goodness-of-fit tests
Absolute error (AE)
Integrated error (IE)
r
=
49
r
=
18
r
=
10
r
=
49
r
=
18
r
=
10
Pareto(Hill)
0.1340
0.0584
0.0584
0.0032
0.0027
0.0027
TPD(Aban)
0.0948
0.0839
0.0832
0.0020
0.0018
0.0016
TPD(moment)
0.1053
0.0738
0.0737
0.0019
0.0015
0.0013
Kernel
0.0890
(
2
)
0.0271
(
1
)
0.0138
(
1
)
0.0009
(
1
)
0.0007
(
1
)
0.0007
(
1
)
Cluster
0.0741
(
1
)
0.0510
(
2
)
0.0278
(
2
)
0.0011
(
2
)
0.0010
(
2
)
0.0009
(
2
)Table3 gives absolute errors and integrated errors of the five estimation methods in r=49,18,10 cases. We note that the cluster truncated Pareto distribution has the smallest errors in all 6 cases. This means the cluster method is superior in fitting the hurricane data compared with the other existing methods.


### 6.4. Fishing Example

Another example is determining the chance of catching a fish of record length (size). Overfishing is a serious issue that has been known to collapse many fish populations. To help control the population, limits are set upon anglers to determine the size of fish which can be kept.The data is from a fishing trip, May 29–June 3, 2011, to Buzzard’s Bay of the Cape Cod area in Massachusetts, USA, by Coia [13]. The largest black sea bass lengths were measured out of a total of 326 black sea bass caught on the trip, using 43.0 cm as a lower-limit threshold. The threshold of 43.0 cm was chosen to be conservative. When sampling, locations producing smaller fish were avoided and the largest fish were targeted. A time-series plot in Figure 4 shows the lengths of these largest 72 fish in the order of which they were caught.Figure 4
Lengths of 72 black sea bass (Centropristis striata) caught at least 43 cm long, fishing in Buzzard’s Bay, May 29–June 3, 2011.By using 71 two-point slopes in (19) and the seven steps in Section 4, we construct k=4 clusters. We select the (k-1)=3 largest absolute values of the two-point slopes Sn-3,n-1≤Sn-2,n-1≤Sn-1,n-1 in (20) as
(42)S71,71=88.6994,S70,71=63.8948,S69,71=36.3351,
since the fourth largest two-point slope S68,71=35.3847 does not have a large change compared with  S69,71=36.3351.Then we determinet^i’s and k=4 groups as
(43){a=t^0,…,Xc1,n},{Xc1+1,n,…,Xc2,n},{Xc2+1,n,…,Xc3,n},{Xc3+1,n,…,t^n=b},
where t^0=3.7=a, t^1=Xc1,n=13.7, t^2=Xc2,n=21.29, t^3=Xc3,n=157, and t^4=b; c0=0, c1=46, c2=59, c3=65, and c4=72; n1=46, n2=13, n3=6, and n4=7; n1+n2+n3+n4=n=72;   then we have an estimated CTPD in (16) as
(44)Fc^(x;T^,Λ^;W^)=∑i=14‍(nin)Fi(x;t^i-1,t^i,α^i).
This construction is shown in Box 3.Box 3:Construction of cluster truncated Pareto distribution from the data.We also apply the kernel estimator in (29) and bandwidth in (33) to the fish data. Figure 5 is a log-log plot which exhibits data and five estimated distribution curves. We note that the original Pareto distribution does not fit the data well in the right tail. The moment and Aban estimated truncated Pareto distribution (TPD) fit the data well in the right tail, but not so well in the smaller or middle values data. The cluster truncated Pareto distribution overcomes this problem and has the best fit to the data over the whole range. Figure 5 suggests a single distribution may not totally represent how natural data is distributed. We group the data by using the cluster method. Figure 5 also shows that the nonparametric kernel estimated distribution fits the data well.Figure 5
Log-log plot of fishing example with estimated distribution curves. The red circles are the data; the black straight line is the original Pareto distribution; the green dot line is the MLE estimated truncated Pareto distribution; the brown dash line is the moment estimated truncated Pareto distribution; the thick green dash line is the kernel estimated distribution; the thick blue line is the cluster Pareto distribution.Table4 gave α^, μ^, Median, 5% VaR, and 1% VaR of each of four estimation methods. We note that the cluster method and kernel method give the largest median and the smallest VaRs.Table 4
Five estimation methods for fishing example.
Method
α
^
μ
^
Median
5
% VaR
1
% VaR
Pareto(Hill)
16.3582
45.6433
44.8612
51.6419
56.9812
TPD(Aban)
15.6493
45.7437
44.9108
51.3511
54.7664
TPD(moment)
15.7782
45.7250
44.8961
51.2993
54.7273
Kernel
—
45.7484
45.4767
49.3010
54.5943
Cluster
α
^
1 = 2.7781
43.4179
45.2801
50.3432
54.0029
α
^
2 = 162.6216
α
^
3 = 64.5429
α
^
4 = 17.0537We also ran goodness-of-fit tests mathematically to confirm which estimated distribution best fits the fish data.Table5 gives the values of the test statistics and P value of each of three goodness-of-fit tests. We note that the cluster truncated Pareto distribution has the smallest test statistics (i.e., the smallest errors) and the largest P values in the K-S test. The kernel estimated distribution has the smallest test statistics in the A-D test and C-v-M test, respectively. Thus the cluster truncated Pareto distribution and kernel estimated distribution fit best to the fish data.Table 5
Goodness-of-fit testsn=72 for fishing example.
Method
Goodness-of-fit tests
K-S test
A-D test
C-v-M test
Test statistic
P value
Test statistic
P value
Test statistic
P value
Pareto(Hill)
0.1496
0.0702
2.8568
0.0324
0.4956
0.0409
TPD(Aban)
0.1407
0.1021
2.8413
0.0330
0.4442
0.0554
TPD(moment)
0.1434
0.0916
2.8672
0.0320
0.4560
0.0516
Kernel
0.0697
(
2
)
0.7240
(
2
)
0.7034
(
1
)
0.5560
(
1
)
0.0379
(
1
)
0.9438
(
1
)
Cluster
0.0687
(
1
)
0.7348
(
1
)
0.8528
(
2
)
0.4445
(
2
)
0.0590
(
2
)
0.8202
(
2
)Table6 gives absolute errors and integrated errors of five estimation methods in r=72,50,10 cases. We note that the cluster truncated Pareto distribution and the kernel estimated distribution have the smallest errors in all 6 cases. The cluster method and the kernel estimation method are superior in fitting the fish data compared with the other existing semiparametric estimation methods.Table 6
Errors of goodness-of-fit testsn=72 for fishing example.
Method
Goodness-of-fit tests
Absolute error (AE)
Integrated error (IE)
r
=
72
r
=
50
r
=
10
r
=
72
r
=
50
r
=
10
Pareto(Hill)
0.1496
0.1496
0.0739
0.0154
0.0148
0.0109
TPD(Aban)
0.1407
0.1407
0.0770
0.0149
0.0144
0.0108
TPD(moment)
0.1434
0.1434
0.0747
0.0149
0.0143
0.0105
Kernel
0.0697
(
2
)
0.0561
(
1
)
0.0179
(
1
)
0.0040
(
1
)
0.0035
(
1
)
0.0022
(
1
)
Cluster
0.0687
(
1
)
0.0687
(
2
)
0.0332
(
2
)
0.0055
(
2
)
0.0055
(
2
)
0.0046
(
2
)


## 6.1. Cluster Method

By using 48 two-point slopes in (19) and the seven steps in Section 4, we construct k=3 clusters. We select the k-1=2 largest absolute values of the two-point slopes Sn-2,n-1≤Sn-1,n-1 in (20) as
(30)S48,48=14.2027,S47,48=5.6574,
since the third largest two-point slope S46,48=5.2276 does not have a large change compared with  S47,48=5.6574.Then we determinet^i’s and k=3 groups as
(31){a=t^0=X1,n,…,Xc1,n},{Xc1+1,n,…,Xc2,n},{Xc2+1,n,…,t^3=Xn,n=b},
where  t^0=X1,n=3.7=a, t^1=Xc1,n=13.7, t^2=Xc2,n=21.29, and t^3=Xc3,n=157=b; c0=0, c1=27, c2=40, and c3=49; n1=27, n2=13, and n3=9; n1+n2+n3=n=49; then we have an estimated CTPD in (16) as
(32)FC^(x;T^,Λ^;W^)=∑i=13‍(nin)Fi(x;t^i-1,t^i,α^i).
This construction is shown in Box 2.Box 2:Construction of cluster truncated Pareto distribution from the data.


## 6.2. Kernel and Other Estimation Methods in Log-Log Plot

We apply the kernel estimator in (29) which is normalized to the hurricane data. Here, we use a standard normal kernel and optimal bandwidth [10, p. 45]
(33)hopt=(43n)1/5≈1.05922n-1/5.
We ensure that the bandwidth is sufficiently large such that the estimated tail distribution is smooth enough.Table1 gives α^, μ^, Median, 5% Value-at-Risk (VaR), and 1% VaR of each of four estimation methods. We note that the cluster method and kernel method give the largest medians. The cluster method gives the smallest VaRs.Table 1
Five estimation methods for hurricane example.
Method
α
^
μ
^
Median
5
% VaR
1
% VaR
Pareto(Hill)
0.8126
∞
8.68 billion
147.68 billion
1070.30 billion
TPD(Aban)
0.6206
21.10 billion
9.73 billion
85.15 billion
136.17 billion
TPD(moment)
0.6476
20.48 billion
9.47 billion
82.55 billion
134.90 billion
Kernel
—
19.88 billion
12.88 billion
80.58 billion
156.14 billion
Cluster
α
^
1 = 0.6476
27.65 billion
11.19 billion
77.24 billion
132.41 billion
α
^
2 = 5.6498
α
^
3 = 0.8416Figure3 is a log-log plot which exhibits data and five estimated distribution curves. We note that the original Pareto distribution does not fit data well in the right tail. The moment and Aban estimated truncated Pareto distribution (TPD) fit data well in the right tail, but not so well in the smaller or middle values data. The cluster truncated Pareto distribution overcomes this problem and has the best fit to the data over the whole range. Figure 3 suggests a single distribution may not totally represent how natural data is distributed. We may consider grouping data by using the cluster method. Figure 3 also shows that the nonparametric kernel estimated distribution fits the data well.Figure3 provides a visual observation. It is necessary to run goodness-of-fit tests mathematically to confirm which estimated distribution best fits the hurricane data.


## 6.3. Goodness-of-Fit Tests

In this section we conduct three goodness-of-fit tests, Kolmogorov-Smirnov, Anderson Darling, and Cramer-von Mises. All three tests are based on the distance between the empirical distribution function and the proposed distribution function: original Pareto distribution in (1) or truncated Pareto distribution in (3) or mixture truncated Pareto distribution in (10).Each test considers the same null and alternative hypothesis:(34)H0:F(x)=F*(x)versusH1:F(x)≠F*(x),
where F(x) is the unknown true distribution of the sample data and F*(x) is one of our proposed four estimated distributions:(1)
Pareto distribution in (1) with Hill estimator α^Hill in (25);(2)
truncated Pareto distribution (TPD) in (3) with Aban estimator α^Aban in (27);(3)
truncated Pareto distribution in (TPD) (3) with moment estimator α^M in (26);(4)
kernel estimated distribution in (29);(5)
cluster truncated Pareto distribution in (16) with moment estimator α^M(i) in (26).We ran a test for each estimated distribution as F*(x).(1) The Kolmogorov-Smirnov (K-S) Test [11]. The test statistic is given by
(35)T=supx|F*(x)-Sn(x)|,-∞<x<∞,
where Sn(x)=(1/n)∑i=1n‍I(-∞,x](Xi) is the empirical distribution function. Under H0 the two-tailed P value for the K-S test is as follows:
(36)Pvalue=1-[1-T∑j=0[n(1-T)]‍n!j!(n-j)!(1-T-jn)n-j(T+jn)j-1]2,
where [n(1-T)] is the integer part of n(1-T).(2) Anderson and Darling Test (A-D Test). Anderson and Darling [12] introduced a measure of “distance” between the empirical distribution Sn(x) and the proposed c.d.f. F*(x) by using a metric function space,
(37)Wn2=n∫-∞∞[Sn(x)-F*(x)]2ψ(F*(x))dF*,
where ψ(u)=1/(u(1-u)), with u=F*(x). Let ui=F*(Xi,n); assume under H0 the test statistic and P value are given by
(38)Wn2=-n-1n∑j=1n‍(2j-1)(log(uj)+log(1-un-j+1)),Pvalue=1-2z∑j=0∞‍{(-1)jΓ(j+1/2)(4j+1)j!e-(4j+1)2π2/8/z·∫0∞e(z/8/(w2+1)-(4j+1)2π2w2/8/z)dw(-1)jΓ(j+1/2)(4j+1)j!},
where z is the observed value of  Wn2  and  Γ(x)=∫0∞tx-1e-tdt,  x>0, is the Gamma function.(3) Cramer-von Mises Test (C-v-M Test). Anderson and Darling [12] proposed this test by using ψ(u)=1 in (37). Thus under H0 the test statistic and P value are given by
(39)nω2=112n+∑j=1n‍(uj-2j-12n)2,Pvalue=1-1πz∑j=0∞{(-1)2jΓ(j+1/2)(4j+1)1/2j!πe-(4j+1)2/16/z‍·K1/4((4j+1)216z)},
where K1/4(x) is the modified Bessel function of the second kind,
(40)Kα(x)=π2I-α(x)-Iα(x)sin(απ),Iα(x)=∑m=0∞‍1m!Γ(m+α+1)(x2)2m+α.Table2 gives the values of the test statistics and P values of three goodness-of-fit tests. Note that the cluster truncated Pareto distribution has the smallest test statistics in the K-S test (i.e., the smallest errors) and the largest P values. The kernel estimated distribution gives the smallest test statistics in the A-D test and C-v-M test, respectively. This means the cluster truncated Pareto distribution and kernel estimated distribution have the best fit to the hurricane data.Table 2
Goodness-of-fit testsn=49 for hurricane example.
Method
Goodness-of-fit tests
K-S test
A-D test
C-v-M test
Test statistic
P value
Test statistic
P value
Test statistic
P value
Pareto(Hill)
0.1340
0.2900
2.7141
0.0383
0.2057
0.2568
TPD(Aban)
0.0948
0.6282
2.3126
0.0622
0.0964
0.6030
TPD(moment)
0.1053
0.5308
2.3672
0.0582
0.1095
0.5402
Kernel
0.0890
(
2
)
0.6805
(
2
)
2.2344
(
1
)
0.0685
(
1
)
0.0427
(
1
)
0.9187
(
1
)
Cluster
0.0741
(
1
)
0.8035
(
1
)
2.2584
(
2
)
0.0665
(
2
)
0.0467
(
2
)
0.8958
(
2
)
Note. In this paper, we use (1) to denote the best values and (2) to denote the second-best values in the tables.In Table3, we took the r largest data in the sample. The absolute error and integrated error are defined by
(41)AE=supx|F*(x)-Sn(x)|,-∞<x<∞,IE=1(Xn,n-Xn-r+1,n)[∫Xn-r+1,nXn,n‍(Sn(x)-F*(x))2dx]1/2.Table 3
Errors of goodness-of-fit testsn=49 for hurricane example.
Method
Goodness-of-fit tests
Absolute error (AE)
Integrated error (IE)
r
=
49
r
=
18
r
=
10
r
=
49
r
=
18
r
=
10
Pareto(Hill)
0.1340
0.0584
0.0584
0.0032
0.0027
0.0027
TPD(Aban)
0.0948
0.0839
0.0832
0.0020
0.0018
0.0016
TPD(moment)
0.1053
0.0738
0.0737
0.0019
0.0015
0.0013
Kernel
0.0890
(
2
)
0.0271
(
1
)
0.0138
(
1
)
0.0009
(
1
)
0.0007
(
1
)
0.0007
(
1
)
Cluster
0.0741
(
1
)
0.0510
(
2
)
0.0278
(
2
)
0.0011
(
2
)
0.0010
(
2
)
0.0009
(
2
)Table3 gives absolute errors and integrated errors of the five estimation methods in r=49,18,10 cases. We note that the cluster truncated Pareto distribution has the smallest errors in all 6 cases. This means the cluster method is superior in fitting the hurricane data compared with the other existing methods.


## 6.4. Fishing Example

Another example is determining the chance of catching a fish of record length (size). Overfishing is a serious issue that has been known to collapse many fish populations. To help control the population, limits are set upon anglers to determine the size of fish which can be kept.The data is from a fishing trip, May 29–June 3, 2011, to Buzzard’s Bay of the Cape Cod area in Massachusetts, USA, by Coia [13]. The largest black sea bass lengths were measured out of a total of 326 black sea bass caught on the trip, using 43.0 cm as a lower-limit threshold. The threshold of 43.0 cm was chosen to be conservative. When sampling, locations producing smaller fish were avoided and the largest fish were targeted. A time-series plot in Figure 4 shows the lengths of these largest 72 fish in the order of which they were caught.Figure 4
Lengths of 72 black sea bass (Centropristis striata) caught at least 43 cm long, fishing in Buzzard’s Bay, May 29–June 3, 2011.By using 71 two-point slopes in (19) and the seven steps in Section 4, we construct k=4 clusters. We select the (k-1)=3 largest absolute values of the two-point slopes Sn-3,n-1≤Sn-2,n-1≤Sn-1,n-1 in (20) as
(42)S71,71=88.6994,S70,71=63.8948,S69,71=36.3351,
since the fourth largest two-point slope S68,71=35.3847 does not have a large change compared with  S69,71=36.3351.Then we determinet^i’s and k=4 groups as
(43){a=t^0,…,Xc1,n},{Xc1+1,n,…,Xc2,n},{Xc2+1,n,…,Xc3,n},{Xc3+1,n,…,t^n=b},
where t^0=3.7=a, t^1=Xc1,n=13.7, t^2=Xc2,n=21.29, t^3=Xc3,n=157, and t^4=b; c0=0, c1=46, c2=59, c3=65, and c4=72; n1=46, n2=13, n3=6, and n4=7; n1+n2+n3+n4=n=72;   then we have an estimated CTPD in (16) as
(44)Fc^(x;T^,Λ^;W^)=∑i=14‍(nin)Fi(x;t^i-1,t^i,α^i).
This construction is shown in Box 3.Box 3:Construction of cluster truncated Pareto distribution from the data.We also apply the kernel estimator in (29) and bandwidth in (33) to the fish data. Figure 5 is a log-log plot which exhibits data and five estimated distribution curves. We note that the original Pareto distribution does not fit the data well in the right tail. The moment and Aban estimated truncated Pareto distribution (TPD) fit the data well in the right tail, but not so well in the smaller or middle values data. The cluster truncated Pareto distribution overcomes this problem and has the best fit to the data over the whole range. Figure 5 suggests a single distribution may not totally represent how natural data is distributed. We group the data by using the cluster method. Figure 5 also shows that the nonparametric kernel estimated distribution fits the data well.Figure 5
Log-log plot of fishing example with estimated distribution curves. The red circles are the data; the black straight line is the original Pareto distribution; the green dot line is the MLE estimated truncated Pareto distribution; the brown dash line is the moment estimated truncated Pareto distribution; the thick green dash line is the kernel estimated distribution; the thick blue line is the cluster Pareto distribution.Table4 gave α^, μ^, Median, 5% VaR, and 1% VaR of each of four estimation methods. We note that the cluster method and kernel method give the largest median and the smallest VaRs.Table 4
Five estimation methods for fishing example.
Method
α
^
μ
^
Median
5
% VaR
1
% VaR
Pareto(Hill)
16.3582
45.6433
44.8612
51.6419
56.9812
TPD(Aban)
15.6493
45.7437
44.9108
51.3511
54.7664
TPD(moment)
15.7782
45.7250
44.8961
51.2993
54.7273
Kernel
—
45.7484
45.4767
49.3010
54.5943
Cluster
α
^
1 = 2.7781
43.4179
45.2801
50.3432
54.0029
α
^
2 = 162.6216
α
^
3 = 64.5429
α
^
4 = 17.0537We also ran goodness-of-fit tests mathematically to confirm which estimated distribution best fits the fish data.Table5 gives the values of the test statistics and P value of each of three goodness-of-fit tests. We note that the cluster truncated Pareto distribution has the smallest test statistics (i.e., the smallest errors) and the largest P values in the K-S test. The kernel estimated distribution has the smallest test statistics in the A-D test and C-v-M test, respectively. Thus the cluster truncated Pareto distribution and kernel estimated distribution fit best to the fish data.Table 5
Goodness-of-fit testsn=72 for fishing example.
Method
Goodness-of-fit tests
K-S test
A-D test
C-v-M test
Test statistic
P value
Test statistic
P value
Test statistic
P value
Pareto(Hill)
0.1496
0.0702
2.8568
0.0324
0.4956
0.0409
TPD(Aban)
0.1407
0.1021
2.8413
0.0330
0.4442
0.0554
TPD(moment)
0.1434
0.0916
2.8672
0.0320
0.4560
0.0516
Kernel
0.0697
(
2
)
0.7240
(
2
)
0.7034
(
1
)
0.5560
(
1
)
0.0379
(
1
)
0.9438
(
1
)
Cluster
0.0687
(
1
)
0.7348
(
1
)
0.8528
(
2
)
0.4445
(
2
)
0.0590
(
2
)
0.8202
(
2
)Table6 gives absolute errors and integrated errors of five estimation methods in r=72,50,10 cases. We note that the cluster truncated Pareto distribution and the kernel estimated distribution have the smallest errors in all 6 cases. The cluster method and the kernel estimation method are superior in fitting the fish data compared with the other existing semiparametric estimation methods.Table 6
Errors of goodness-of-fit testsn=72 for fishing example.
Method
Goodness-of-fit tests
Absolute error (AE)
Integrated error (IE)
r
=
72
r
=
50
r
=
10
r
=
72
r
=
50
r
=
10
Pareto(Hill)
0.1496
0.1496
0.0739
0.0154
0.0148
0.0109
TPD(Aban)
0.1407
0.1407
0.0770
0.0149
0.0144
0.0108
TPD(moment)
0.1434
0.1434
0.0747
0.0149
0.0143
0.0105
Kernel
0.0697
(
2
)
0.0561
(
1
)
0.0179
(
1
)
0.0040
(
1
)
0.0035
(
1
)
0.0022
(
1
)
Cluster
0.0687
(
1
)
0.0687
(
2
)
0.0332
(
2
)
0.0055
(
2
)
0.0055
(
2
)
0.0046
(
2
)


## 7. Conclusions

Overall, after the studies in this paper, we may conclude the following.(1)
Truncated Pareto models are useful for analyzing real-world data.(2)
Cluster truncated Pareto models are useful for grouped data.(3)
The two-point slope technique is innovate and useful for determining the threshold points in clustering.(4)
A nonparametric kernel estimated distribution fits heavy-tailed data very well.(5)
It is a difficult problem to determinek—the number of groups which is based on the two-point slope in (20) in Step 2 of the construction of the CTPD in (16). The k very much depends on the empirical observations. We displayed this technique in the two examples in Section 6. We plan to do further studies on more complex data sets in the future.



---

*Source: 265373-2013-09-18.xml*