# Inference about the Tail ofa Distribution: Improvementon the Hill Estimator

Authors: Jean Nuyts
Journal: International Journal of Mathematics and Mathematical Sciences
 (2010)

---

## Abstract

The Hill estimator is often used to infer the power behavior in tails of
experimental distribution functions. This estimator is known to produce bad results in certain situations which have lead to the so-called Hill horror plots. In this brief note, we propose an improved estimator which is simple and coherent and often provides an efficient remedy in the bad situations, especially when the distribution is decreasing slowly, when the data is restricted by external cuts to lie within a finite domain, or even when the distribution is increasing.


---

## Body

## 1. Introduction

It has been advocated that self-organization, first discovered to dominate sand pile formation [1], may very well apply to many financial, economic, traffic control, or social phenomena. The general outcome of these systems is in general an asymptotic power like behavior of the experimental distribution x-μ in some variable x. The value of the critical exponent μ is often a major prediction of the models. Let us also mention models leading to Pareto [2, 3], Zipf [4], Levy-flight [5, 6], or Padé [7–9] type distributions which also provide asymptotic power like behaviors with some definite critical exponent.A particularly simple estimator of the critical exponent for a set of data measures [10] has been provided by Hill [11]. It has proved to be very useful and has been widely used. See, for example, the KU Leuven web page [12]. Following Hill, the estimator is first applied to a subset of measures corresponding to the highest values of the variable. In order to increase the statistics, this subset is then extended to include progressively further experimental values of the variable, every set providing a value of μ. These values are then plotted in a so-called Hill plot [13]. The subset is extended as long as it includes values which can be considered to lie in the asymptotic domain where the distribution is power like. The values obtained when they are stable provide the best value of the Hill estimator, as usually seen on the Hill plot. It is then obvious that the estimator usually takes into account all the highest experimental values of the variable larger than a certain value x≥L. The data points with x<L do not belong to the asymptotic region or turn out to be unsafe to use.Various properties, aspects, and generalizations of the Hill estimator have been discussed and studied in numerous publications. One should mention that many questions about the asymptotic properties, about asymptotic normality, and about the volatility of the index have been addressed (see, e.g., [14–18]). Some authors have proposed, in order to improve the high volatility of the Hill plots, to smooth the result by averaging the Hill estimator values corresponding to different numbers of order statistics [19]. Other authors have smoothed the Hill estimator by convoluting the experimental random variables by a kernel function together with a bandwidth parameter [20]. An optimal choice of the kernel and of the parameter improves the situation greatly.However, it should be stressed that, in many cases, experimental distributions are related to discrete phenomena, which carry their own natural limits. In other cases, large values in the data may be biased, unsafe or unreachable for technical reasons. Let us give two examples.One example of considerable importance relates to the evaluation of risk in finance, in particular those related to the variations of interest rates. Starting from their known daily variation in the past, one may try to evaluate the probability of a major overnight variation of say 3 or 4 percent in the future. Using a normal law, as often used by risk managers, the catastrophe should occur of the order of once every 10,000 years. Starting from the same available data but with a power law this may happen tree or four times per century. The precise evaluation of the parameterμ is obviously of paramount importance. In data of the Federal Reserve System, if any day the variation is larger than a certain predefined limit the quotation is suspended. Hence the data is artificially cut for high x. The relevant Xi must be restricted in a finite domain extending from some value L where asymptotics begins to some value R where the data is artificially cut.Another example occurs in sociology. Suppose that some phenomena depends on the number of inhabitants of cities, it is clear that no data on earth today will be obtained for towns larger than say thirty millions inhabitants or smaller than say ten inhabitants. The available data will be restricted to a finite domain.To conclude, in various cases, the variable is not only discrete but also restricted to lie within a finite domainDL≤x≤DR. Data outside the domain is either not available or unsafe to use or does not correspond to the asymptotic region. Remark that the domain may not only have a lowest (left) bound DL but also a highest (right) bound DR. Though the lower bound DL is taken into account by the Hill estimator, the possible presence of the highest bound DR has not been addressed by the Hill.In this paper, we thus wish to provide a very simple improvement of the Hill estimator (and of the Hill plot) which takes into account in a perfectly symmetrical way of a lowest valueL (DL≤L) and of a highest value R (R≤DR) defining a safe domain [L,R] where the power behavior is at work, where data exists, and from which the critical exponent should be inferred.The problem of existing limits on the data has been subject to a much lower scrutiny than the Hill estimator itself. We would like to cite the work of Beirlant and Guillou [21] and Beirlant et al. [22] dealing in particular with insurance policies where policy provisions (deduction, limits) constrain the data. They suppose censored data, that is, that the number of events above the constrain value is known and discuss the influence of censoring. This is not exactly the aim of our article.We have focused our attention on data which exist in a finite range only and have obtained the simplest estimator which takes into account in a symmetrical way the lower and higher bounds of the domain.Finally, we would like to mention the related problems of variance and bias of the estimators and especially the question of their experimental determination. For the Hill estimator, this is delicate and it has given rise to much research as is testified by numerous references [23–26]. Starting from the basic definition of the variance-bias parameters applied to the improved estimator, a detailed discussion following the paths set up for the Hill estimator are certainly worth further work and publications, both in theory and when actual data, which carry their own uncertainty, are used. Indeed, the experimental precision of each data point is not always easy to determine, may in fact depend of the value of the Xi itself and has to be taken into account precisely. On the other hand, in simulations, the result depends on the details of the random number generators.


## 2. The Hill Estimator

One supposes that, based on some theoretical model, one predicts a distribution densityf(x) which behaves asymptotically as(2.1)f(x)≈g(x)x-μwhenx→∞,
where f and x are real and the critical exponent μ is a positive real parameter. When x→∞, the function g(x) is supposed to be a rather smooth function which, for large x (say x>L), is often assumed to become essentially a constant λ. In those cases and within a certain margin of error, the distribution is thus approximated in the form(2.2)f(x)≈λx-μwhenx→∞,
which is scale free. As explained in the introduction, it has been conjectured by different authors that this type of power like distributions and in particular those based on self-organized critical models, rather than the often used Gauss like exponential forms, could very well dominate certain financial, economic, or social phenomena.LetYi (Yi>0) be a random sample obtained from experimental data for a phenomenon which is supposed to follow a distribution law f satisfying the requirements (2.1) and/or (2.2). The question is to draw inference on the critical exponent μ from the random sample.This most important question was discussed very carefully by Hill [11] both from a Bayesian and a frequentist approach. He showed that, in a first approach, both points of view lead to the same very useful answer. His recipe for estimating μ can be outlined as follows.  (i)
Let the setXi be the set Yi reordered (reversed order statistics) in such a way that
(2.3)Xi≥Xjfori<j,
that is, the set Xi is ordered in a decreasing fashion, X1 being the highest Y value.(ii)
Construct the sets of numbersHk, αk, and μk for k>2. The set Hk is defined by
(2.4)Hk=1k∑j=1kln(Xj)-ln(Xk),
which is identical to the set proposed by Hill (see [11])
(2.5)Ĥr=1r+1∑j=1rln(Xj)-rr+1ln(Xr+1)(withĤr=Hr+1).
The set αk is defined by
(2.6)αk=1Hk,
and the set μk defined by(2.7)μk=αk+1.(iii)
For quite many distributions, Hill showed thatμk is an estimator of μ improving when k is increased until “it seems unwise to proceed.” It is the point Xk≈L at the left of which the form of the actual distribution is not anymore approximated safely enough by  (2.2). In other words, this occurs when the variable x is not large enough anymore to be in the asymptotic region of the distribution. Both the phenomenon and the limitation can easily be seen by constructing specific examples.Unfortunately, there are simple but somewhat more subtle distributions for which the setμk does not converge well when k is increased. This type of situation has led to what has been called “horror Hill plots.” Such horror plots can easily be constructed by considering simple examples or by looking at [27, page 194 ]. As argued in the introduction, there are also cases when the discrete experimental distribution carry their own domain, bounded on the left and on the right DL<x<DR.In this short note, we show how to remedy some of these unfortunate situations in a very straightforward way. We construct some numerical examples to show this explicitly. This result will be achieved by taking into account, not only a left boundaryL but also a right boundary R. It is supposed that the data cannot be trusted and/or is not power like at the left of L or at the right of R.


## 3. Improvement on the Hill Estimator

We first derive a simple heuristic formula and then show how to apply it to improve the Hill estimator.


### 3.1. A Simple and Exact Formula

Let us first suppose thatf(x) is exactly, for x>0, of a power law form(3.1)f(x)=λx-μ,
with an arbitrary normalization constant λ. Take two arbitrary positive numbers 0<L<R and define the average value 〈ln(x)〉LR of ln(x) on the interval [L,R] as(3.2)〈ln(x)〉LR=∫LRln(x)f(x)dx∫LRf(x)dx.
After some algebra, defining(3.3)μ=α+1,
and for later convenience(3.4)H=1α,
one finds, from  (3.1), the exact relation(3.5)〈ln(x)〉LR=1α+ln(L)L-α-ln(R)R-αL-α-R-α,
which depends on the correction function C(α,L,R)(3.6)C(α,L,R)=ln(L)L-α-ln(R)R-αL-α-R-α.
Equation (3.5) is our basic equation which is exactly valid for an exact power law distribution  (3.1) and approximately correct for a distribution which satisfies  (2.1) and/or  (2.2).


### 3.2. The Basic Hill Estimator

The basic Hill estimator is obtained when the upper limitR is chosen at infinity. Indeed, when μ is larger than 1 and R→∞,  (3.5) reduces to(3.7)〈ln(x)〉L∞=H+ln(L).The formula  (3.7) can easily be used to draw inference for α and μ from the k highest values of the random sample X with the chosen order statistics which we consider as the conditional event. Take 〈ln(x)〉LR as the experimental average value of ln(x) from L=Xk to the highest empirical value X1,
(3.8)〈ln(x)〉L∞≡1k∑j=1kln(Xj),(3.9)1k∑j=1kln(Xj)=Hk+ln(Xk).
We find the formula for the Hill Hk estimator (2.4) exactly.Remark that by using a Simpson rule an experimental average, slightly better than  (3.8), is obtained by(3.10)〈ln(x)〉L∞≡12(X1+Xk)+1k∑j=2k-1ln(Xj).
The difference for the Hill estimator using the slightly better  (3.10) rather than  (3.8) is usually too minute to care.


### 3.3. The Improvement

From  (3.5), we see that the Hill estimator can be improved by taking into account not only a lowest value L=Xl in the reduced empirical set {Xl,Xl-1,…,Xr} but also a highest value R=Xr. Taking again a sample estimate for 〈ln(x)〉LR in the reduced set, one obtains the solution αlr of(3.11)1l-r+1∑j=rlln(Xj)=1αlr+ln(Xl)Xl-αlr-ln(Xr)Xr-αlrXl-αlr-Xr-αlr
as an inference for μ,(3.12)μlr=1+αlr.This obviously provides an easy generalization of the Hill procedure. It makes sense when it is known theoretically that the density functionf(x) behaves as x-μ times a constant λ or times a slowly varying function g(x) and that the random sample is secure for DL≤L≤x≤R≤DR. It takes into account the facts that the exact theoretical form of f(x) is not known on the left of x=L and that the data points do not extend beyond R=Xr because of limited statistics or when the data is poorly known outside a finite DL,DR domain. In this case, the set of Xi derived from the random sample Y should include the values inside the interval only and thus take the Xi's between the two limit points (L<Xl≤Xi≤Xr<R) into account.The generalised Hill plot is obtained by taking firstl=r+1, then increasing l until “t seems unwise to proceed” and plotting the μlr as a function of l. Otherwise l can be increased and r decreased until “it seems unwise to proceed.” When the data is not biased for the largest values of Xi, the choice r=1 is optimal.It is finally worth noting the very important fact that our basic equation (3.11) is perfectly symmetrical under the exchange Xl↔Xr. As a result, it will be apply and provide meaningful answer for distribution which increase (μ negative) rather than decrease with x, as will be seen in the examples.Our fundamental equation (3.11) is transcendental and hence requires a further treatment. The estimation of the value of αlr inferred from  (3.11) can, for example, be obtained numerically in the two following ways.Method(1). Direct Evaluation
By using standardad hoc computer programs, (3.11) can be solved numerically for αlr.Method(2). Iteration
A simple way to achieve the same result is as follows. Define the functionD(α,L,R) as the derivative of the correction function C(α,L,R) with respect to α(3.13)D(α,L,R)=∂∂αC(α,L,R)=RαLα(ln(L)-ln(R))2(Lα-Rα)2.
Take the first order αlr[1] as the Hill solution
(3.14)αlr[1]=1∑j=rlln(Xj)/(l-r+1)-ln(Xl),
and define the successive approximations αlr[p],p=2,… by iteration as
(3.15)αlr[p+1]=αlr[p](1+αlr[p](∑j=rlln(Xj)/(l-r+1))-αlr[p]C(αlr[p],Xl,Xr)-1αlr[p]2D(αlr[p],Xl,Xr)-1).
In the right hand side, the correction function C and its derivative D are estimated for the value of α of the preceding iteration. Then αlr[p]→αlr for p→∞. Empirically, for [p] as low as [5, 6] or [7–9], the approximation for αlr and hence for μlr=αlr+1 is already very good.


## 3.1. A Simple and Exact Formula

Let us first suppose thatf(x) is exactly, for x>0, of a power law form(3.1)f(x)=λx-μ,
with an arbitrary normalization constant λ. Take two arbitrary positive numbers 0<L<R and define the average value 〈ln(x)〉LR of ln(x) on the interval [L,R] as(3.2)〈ln(x)〉LR=∫LRln(x)f(x)dx∫LRf(x)dx.
After some algebra, defining(3.3)μ=α+1,
and for later convenience(3.4)H=1α,
one finds, from  (3.1), the exact relation(3.5)〈ln(x)〉LR=1α+ln(L)L-α-ln(R)R-αL-α-R-α,
which depends on the correction function C(α,L,R)(3.6)C(α,L,R)=ln(L)L-α-ln(R)R-αL-α-R-α.
Equation (3.5) is our basic equation which is exactly valid for an exact power law distribution  (3.1) and approximately correct for a distribution which satisfies  (2.1) and/or  (2.2).


## 3.2. The Basic Hill Estimator

The basic Hill estimator is obtained when the upper limitR is chosen at infinity. Indeed, when μ is larger than 1 and R→∞,  (3.5) reduces to(3.7)〈ln(x)〉L∞=H+ln(L).The formula  (3.7) can easily be used to draw inference for α and μ from the k highest values of the random sample X with the chosen order statistics which we consider as the conditional event. Take 〈ln(x)〉LR as the experimental average value of ln(x) from L=Xk to the highest empirical value X1,
(3.8)〈ln(x)〉L∞≡1k∑j=1kln(Xj),(3.9)1k∑j=1kln(Xj)=Hk+ln(Xk).
We find the formula for the Hill Hk estimator (2.4) exactly.Remark that by using a Simpson rule an experimental average, slightly better than  (3.8), is obtained by(3.10)〈ln(x)〉L∞≡12(X1+Xk)+1k∑j=2k-1ln(Xj).
The difference for the Hill estimator using the slightly better  (3.10) rather than  (3.8) is usually too minute to care.


## 3.3. The Improvement

From  (3.5), we see that the Hill estimator can be improved by taking into account not only a lowest value L=Xl in the reduced empirical set {Xl,Xl-1,…,Xr} but also a highest value R=Xr. Taking again a sample estimate for 〈ln(x)〉LR in the reduced set, one obtains the solution αlr of(3.11)1l-r+1∑j=rlln(Xj)=1αlr+ln(Xl)Xl-αlr-ln(Xr)Xr-αlrXl-αlr-Xr-αlr
as an inference for μ,(3.12)μlr=1+αlr.This obviously provides an easy generalization of the Hill procedure. It makes sense when it is known theoretically that the density functionf(x) behaves as x-μ times a constant λ or times a slowly varying function g(x) and that the random sample is secure for DL≤L≤x≤R≤DR. It takes into account the facts that the exact theoretical form of f(x) is not known on the left of x=L and that the data points do not extend beyond R=Xr because of limited statistics or when the data is poorly known outside a finite DL,DR domain. In this case, the set of Xi derived from the random sample Y should include the values inside the interval only and thus take the Xi's between the two limit points (L<Xl≤Xi≤Xr<R) into account.The generalised Hill plot is obtained by taking firstl=r+1, then increasing l until “t seems unwise to proceed” and plotting the μlr as a function of l. Otherwise l can be increased and r decreased until “it seems unwise to proceed.” When the data is not biased for the largest values of Xi, the choice r=1 is optimal.It is finally worth noting the very important fact that our basic equation (3.11) is perfectly symmetrical under the exchange Xl↔Xr. As a result, it will be apply and provide meaningful answer for distribution which increase (μ negative) rather than decrease with x, as will be seen in the examples.Our fundamental equation (3.11) is transcendental and hence requires a further treatment. The estimation of the value of αlr inferred from  (3.11) can, for example, be obtained numerically in the two following ways.Method(1). Direct Evaluation
By using standardad hoc computer programs, (3.11) can be solved numerically for αlr.Method(2). Iteration
A simple way to achieve the same result is as follows. Define the functionD(α,L,R) as the derivative of the correction function C(α,L,R) with respect to α(3.13)D(α,L,R)=∂∂αC(α,L,R)=RαLα(ln(L)-ln(R))2(Lα-Rα)2.
Take the first order αlr[1] as the Hill solution
(3.14)αlr[1]=1∑j=rlln(Xj)/(l-r+1)-ln(Xl),
and define the successive approximations αlr[p],p=2,… by iteration as
(3.15)αlr[p+1]=αlr[p](1+αlr[p](∑j=rlln(Xj)/(l-r+1))-αlr[p]C(αlr[p],Xl,Xr)-1αlr[p]2D(αlr[p],Xl,Xr)-1).
In the right hand side, the correction function C and its derivative D are estimated for the value of α of the preceding iteration. Then αlr[p]→αlr for p→∞. Empirically, for [p] as low as [5, 6] or [7–9], the approximation for αlr and hence for μlr=αlr+1 is already very good.


## 4. Theoretical Comparison between the Hill Estimator and  the New Estimator

When should the new estimator obviously be preferred to the Hill estimator? We remark that the latter (see (2.4)) depends critically on two quantities only. These are the average value of the logarithm of the data points and the smallest Xl (highest l) included in the sample. The improved estimator depends on these two quantities but also crucially on the third one: the highest attained value Xr.Let us justify this by discussing more carefully than in the introduction three examples of situations, pertaining to three completely different domains of research (sociology, economy, high energy physics), where both limitsXl and Xr should be taken into account.(i)
Suppose that some sociological theory predicts an asymptotic power law for some phenomenon (crime rate, economic growth, power consumption) as a function of the number of inhabitantsNinh in towns, either when this number is high or when it is small. Obviously, since every country has a finite number of inhabitants, the number of people in any city on earth is less than some number, actually 11.914×106 inhabitants for Bombay (India). There can be no data for larger numbers. The data is cut artificially on this higher side essentially by the fact that the earth has a finite population. On the other end of the statistics, when Ninh is small, no aggregation of human beings is called a city if it contains less than say about ten inhabitants, depending on the rules for defining towns in different countries. Even more, no town will ever be defined with a fraction of one inhabitant. Hence the data is cut on the lower side by country depending administrative decisions.(ii)
Suppose that some economic theory predicts a power law for the daily variation of the interest rates (or of the price of some share) when they are large (the tail of the variations). Usually, if this variation on some day exceeds some predefined number the floor has rules to suspend the quotation. Hence, no data will ever be produced with higher variations. This artificial limit, which is usually not taken into account in the models, is included in the analysis of the data by the dependence of the new estimator in the highest valueXr.(iii)
In high energy physics, when two beams of initial particles are made to scatter head on at very high speed along a certain direction, final particles emerge from the scattering region at an angle with respect to the direction of the initial beams. The probability distribution is often plotted as a function of the transverse momentum which is related to this angle. The interaction region is surrounded by detectors. In the forward and in the backward directions there is a blind cone where the scattered particles cannot be identified among those of the intensive initial beams. Hence there are artificial limits both at low and at high transverse momenta outside which no data is produced because of the physics of the measuring devices. When the tails of the distributions of the scattered particles with respect to beam are studied, these artificial limitations have to be taken into account.(iv)
In practical situations, to obtain the improved estimator the parameters boundsXl and Xr, which limit on the left and on the right the data points, have to be chosen in an appropriate manner. There is, a priori, no unique method for choosing them. For actual phenomena, related to some definite model (e.g., to solutions of a differential equation), the model itself should provide some information on the bounds. The onset of the asymptotic domain of the model provides an indication on Xl. On the other hand, the measure themselves may carry a natural right bound Xr where the measurements become meaningless (e.g., a town of more than 5×107 inhabitants). The estimator should be evaluated for various values of the bounds Xl,Xr and educated guesses have to be applied. In general, it is convenient to simply use the highest data point as Xr unless there are good reasons to reject it. In high energy physics, “good experimentalists” are known to apply suitable and correct cuts to their data, depending on a deep inside knowledge of their apparatus.


## 5. Tests of the New Estimator

In order to test the new estimator, we have used it for some trial functions. The new estimator performs as well or even slightly better than the original Hill estimator in cases where the latter is known to be good. When applied to cases where the Hill estimator is known to perform more poorly, the improvement is often spectacular and the new estimator converges to a much better approximate value. Using the method (2) above (see (3.15)), a good value αlr is usually obtained after three steps only and always after four steps, that is, from αlr≈αlr[5].Let us give here a few examples where we have used the following procedure. The examples are divided in sets. The examples (1)–(13) are related to very simple distributions where the best value of μ is obtained by letting l become the largest available value in the data l=Nrand and r the lowest, that is, r=1. In the examples (14)–(17), the Hill plot is drawn for somewhat more complicated distributions where the deviation from a purely power behavior shows up for small x. In all the cases, r=1 (see (3.11)) has been chosen.(1)
We have used FORTRAN to do numerical calculations.(2)
We have used the DRNGCS module of the International Mathematical and Statistical Libraries (IMSL) to generate randomly a certain numberNrand of Xi, inside a predefined interval [DL,DR] and following a given distribution P. It should be noted that the normalization of P is inessential. As required by DRNGCS, this normalization is chosen in such a way that the cumulated distribution N(y)(5.1)N(y)=∫DLyP(x)dx
has N(DR)=1. In our examples, the distribution P and the cumulated distribution N have always been defined on a grid of 1000 to 10,000 points regularly separated between DL and DR. This is precise enough for our purpose.(3)
For each example (1)–(13), we give the results in Table 1 which includes the following.
(i)
The trial distributionP, up to an arbitrary factor, fed to the random generator.(ii)
The predefined limitsDL and DR chosen for the data DL≤Xi≤DR produced by the random generator and ordered according to (2.3).(iii)
The numberNrand of randomly produced Xi.(iv)
The smallest valueL=XNrand and highest value R=X1 obtained from the random generator. Obviously DL≤L<R≤DR.(v)
The average valueσ of the natural logarithm of the Xi as produced by the random generator
(5.2)σ=1Nrand∑i=1Nrandln(Xi).(vi)
The value expected forμ, the exponent of the power decrease of the starting distribution P.(vii)
The value produced by the Hill estimatorμHill=μ[1]=α[1]+1 (see (3.14)).(viii)
The valuesμ[5] produced by the 4d iteration. Quite generally, the values μ[4] produced by the 3d iteration are already very good except in example (13) below where we had to proceed to μ[5]). In almost all cases (except in example (13)), it turns out that μ[j]≈μ[4] for j>4 with at least four significant digits well within the errors of the method.Table 1PNrandDLDRLRσinputμμHillμ[5](1)x-510003.0150.03.000414.371.33955.1575.115(2)x-510003.04.03.00153.9621.208510.1755.784(3)x-550003.04.03.00023.9991.21459.6345.142(4)1/x10003.0150.03.017149.533.6420.51.3940.511(5)1/x10003.01500.03.0581494.75.5850.51.2230.508(6)1/x10003.015000.03.115149457.6820.51.1530.518(7)1/(1+p2x2+p4x4)10001.02.01.00011.9920.234≈45.2693.977(8)1/(1+p2x2+p4x4)10001.05.01.00014.6860.320≈44.1263.980(9)ln(x)/x1000100.0400.0100.07399.115.322≈12.3970.849(10)ln(x)/x10002000.05000.02001.79973.28.423≈12.2170.914(11)1/ln(x)x50008000.010000.08000.89999.09.098≈110.0541.247(12)1/ln(x)x50003000.06000.03000.95998.18.346≈13.9441.164(13)x3.510003.010000.01828.39995.98.989−3.51.677−3.503Example(1)
Let us first give an example where the old and the new estimators perform both well. We have supposed a power law  (3.1) with μ=5, DL=3, and DR=150. In this case, DR is large enough and R can be essentially ignored in all the formulas. Following the procedure outlined above we obtain the line (1) of Table 1. We see that in the values produced by the Hill estimator and by the new estimator are almost equal and close to the expected value μ=5. The new estimator is slightly better. The same behavior is found when μ and the related DR are large enough.Example(2)
The second example is almost identical to the first example except that the data is artificially reduced to the intervalDL=3, DR=4. We find the line (2) of Table 1. Here we see that in this extreme case, where the interval allowed has been reduced drastically, the Hill estimator has produced a very bad result μHill≈10 while the new estimator converges toward a much better value close to the expected one.Example(3)
The third example in line (3) of Table 1 is identical to the second except that the number of Xi, Nrand is increased to 5000. The new estimator behaves even better while the Hill one is still very bad. This behavior was expected. It should be noted that, when DR is increased from 4 to over 100, the two estimators tend gradually to produce compatible results. There is a weak dependency in the value chosen for Nrand provided that it is chosen large enough.Examples(4), (5), and (6)
When the probability distribution is chosen to decrease very slowly, hereP=λ/x, even if DR is chosen rather large DR=150, DR=1500 or DR=15000, we see in line (4), (5), and (6) of Table 1 that the new estimator is definitively better than the Hill one. An increase of Nrand and/or of DR does not alter this fact drastically.Examples(7) and (8)
Let us now give an example where the initial distribution involves a slowly varying function as the consequence of a Padé (1,4) type distribution in the positivex region x>0:
(5.3)P=λ11+p2x2+p4x4.
We expect a μ=4 type behavior provided that x is chosen sufficiently large. The actual left limit L to be used depends on the values of the parameters pi. We analyze the problem in the case of an experimental symmetric distribution of variation of interest rates of a Padé type (see [7–9] and especially the results obtained in the first reference). Data which were extracted from the Federal Reserve System lead to p2=494.7 and p4=4886.0 for a lag of one day and a maturity of one year. Here, x is expressed in the convenient unit of percent/year. The parameters DR and DR as well as L and R are given percent/year. It is easy to see that for the term in p4 to dominate over the term p2 in the denominator, a value of x of about one percent in needed. Hence L has to be chosen greater than one percent. We do not expect a x-4 behavior for smaller values. In lines (7) and (8) of Table 1, the left boundary DL is taken to be one percent while the right one DR, when the floor would be supposed to suspend the quotation, is respectively two and five percent.Examples(9), (10), (11), and (12)
Let us now turn to two examples when there is a logarithmic factor, namely,(5.4)f=λln(x)x,
and when there is an inverse logarithmic factor
(5.5)f=λ1ln(x)x.
In both cases the distribution decreases rather slowly so that taking into account the right boundary has usually a major effect. In these cases, we have increased the value of Nrand to 5000. The results depend rather weakly on this choice. In lines (11) and (12) of Table 1, the presence of the ln(x) factor in the denominator leads to an effective decrease of the probability faster than 1/x. Hence, the effective μ[5] is expected and turns out to be somewhat higher than one. In the opposite way, in lines (9) and (10), when the logarithm appears in the numerator, the effective μ[5] is expected to be somewhat lower than one and it is.Example(13)
Since the treatment of the new estimator  (3.11) is perfectly symmetrical under the exchange Xl↔Xr of two limits Xr and Xl, it can be applied directly without any change to infer the asymptotic power behavior of distributions which increase with x and hence have a power behavior with negative μ up to a smooth multiplication function. Let us give a last example for μ=-3.5, DL=3.0 and DR=10000.0. Obviously the Hill estimator produces a wrong sign for μHill. The new estimator is perfect but the iterations had to be carried effectively to the fourth level since the starting μ[1]=μHill is so bad.Example(14). Figure 1
In Figure1, we show the original Hill plot and the improved plot related to the Padé example (8). The parameters in (5.3) are again p2=494.7 and p4=4886.0. Here we have chosen DL=1 percent and DR=3 percent and generated 2000 data points. The continuous line is the original Hill plot, the dotted line is the improved plot, and the dashed line is the expected asymptotic value μ=4.Figure 1
A Padé[0,4] behavior (see Example14).Example(15). Figure 2
We have taken a distribution(5.6)f(x)=3.0x-4.0+1.0x-2.5.
The random data with Nrand=10000 has been collected in the interval DL=10 and DR=30. The original Hill estimate and the improved estimate are plotted in terms of l ranging up to 10000 in Figure 2. The continuous line is the original Hill plot, the dotted line is the improved plot and the dashed line is the expected asymptotic value μ=2.5.Figure 2
A sum of two power like behaviors (see Example15).Example(16). Figure 3
In Figure3, the distribution is taken as ln(x)/x related to example (10). The random data with Nrand=10000 has been collected in the interval DL=100 and DR=400. The continuous line is the original Hill plot, the dotted line is the improved plot and the dashed line is the expected asymptotic value μ=1 (regardless of the ln(x)). In fact one sees that when k reaches about 4000, the value of μ is close to 1. For larger values of k, μ decreases below μ=1. This is due to the fact that the slowly varying function ln(x) in the numerator leads to a slightly decreased value of μ whose effects increases with k. Indeed, since ln(x) is an increasing function of x, the distribution is expected to decrease somewhat more slowly than 1/x.Figure 3
Aln(x)/x behavior (see Example 16).Example(17). Figure 4
In Figure4, the distribution is taken as 1/x related to Examples (4), (5), and (6). The random data with Nrand=10000 has been collected in the interval DL=3 and DR=1500. The continuous line is the original Hill plot, the dotted line is the improved plot and the dashed line is the expected asymptotic value μ=0.5.Figure 4
A1/x behavior (see Example17).


## 6. Conclusion

We have shown that, by a very simple alteration, the Hill estimator, which provides a useful inference for the power behavior of the tail of a distribution, can be easily improved to cover cases where it performs badly. This includes cases when the tail is not of the form of a power law but is multiplied by a slowly varying functionsg(x) including logarithms for example. It also applies when the power μ is of the order or smaller than one. It even works for inferring the tails of increasing distributions, that is, when μ is negative.Needless to say, the approach outlined in this paper is one of efficiency. From the frequentist point of view, it is perfectly justified. However, in some way, the Bayesian point of view is also, but granted not completely, met as the theoretical and empirical knowledge of extreme tails of the distribution is somehow better taken into account. It should be stressed that the new estimator applies particularly when the data is confined by artificial external conditions to lie within a finite domain bounded not only by a lowest value of the variable but also by a highest value.



---

*Source: 924013-2010-06-30.xml*