Kullback–Leibler 情報量を変換し境界値問題と見て重み付き残差法で母数推定する
Takuya KAWANISHI
Kanazawa University

September 7, 2022

https://t-kawanishi.w3.kanazawa-u.ac.jp/ Takuya Kawanishi

1Introduction∙ When the support of the distribution function depend on theparameter, MLE may not exist or is not consistent. For
example, for the three-parameter Weibull (TPW) distribution,
when the shape parameter is less than unity, no MLE exists(e.g., Smith, 1985).

∙ In such cases, the maximum product of spacings (MPS)
method (Cheng and Amin, 1983; Ranneby 1984) can be used.

∙ Jiang (2013) proposed a modiﬁcation of MPS (Jiang’smodiﬁed MPS, JMMPS) which reduces the bias of the shapeand scale parameters of the TPW. But the method has not
attracted much attention.

∙ Kawanishi (2020) found that the MLE, MPS and JMMPS canbe derived from the same framework: Transforming KLD tointegral over the closed interval, (then minimizing the KLD canbe seen a boundary value problem), and then applying themethod of weighted residuals.
https://t-kawanishi.w3.kanazawa-u.ac.jp/ Takuya Kawanishi2

Why transformed KLD? Why MWR?
Transformed KLD∙ Integral over a compact set [0, 1] is easier to handle than theintegral over entire ℝ.
∙ homogeneous partition 𝑝0 = 0, 𝑝1 , … , 𝑝𝑛+1 = 1 of the interval[0, 1].
MWR∙ Various method under the theory of MWR can be applied;∙ the Galerkin method leads to the formation of Jiang’s modiﬁed
MPS (JMMPS);∙ the ML, MPS and JMMPS can be constructed under one
theory.

https://t-kawanishi.w3.kanazawa-u.ac.jp/ Takuya Kawanishi

3

Continuous empirical quantile function
We deﬁne 𝑄E (𝑝, 𝜃) ↦ 𝑥 as follows.
(a) As the values of 𝑄E at points 0, 𝑝1 , … , 𝑝𝑛 , 1,𝑄E (0, 𝜃) = ∗ 𝑥;
𝑄E (𝑝𝑖 , 𝜃) = 𝑋(𝑖) ,𝑄E (1, 𝜃) = 𝑥∗ ;𝑝𝑖 =𝑖,𝑛+1

𝑖 = 1, … 𝑛;where ∗ 𝑥 = inf{𝑥 ∶ 𝐹 (𝑥, 𝜃) > 0}, 𝑥∗ = sup{𝑥 ∶ 𝐹 (𝑥; 𝜃) < 1}.
(b) the function 𝑄E is continuous with respect to 𝑝;(c) on each interval (𝑝𝑖−1 , 𝑝𝑖 ), 𝑖 = 1, … , 𝑛 + 1, 𝑄E is twice
continuously differentiable with respect to 𝑝 and 𝜃 ;(d) ﬁx 𝜃 , 𝑄E is an increasing function of 𝑝.

https://t-kawanishi.w3.kanazawa-u.ac.jp/ Takuya Kawanishi

4Transformed KLD∙ Transformed continuous edf
𝐹̃ E ∶= 𝐹 ◦(𝑄E , 𝜋 𝜃 ),(𝜋 𝜃 ∶ (𝑝, 𝜃) ↦ 𝜃 is a projection.)(1)∙ Let 𝐹0 be the true distribution and let
𝐹 E ∶= 𝐹̃ E ◦(𝐹0 , 𝜋 𝜃 ),we have𝑓E =𝜕𝐹 E(𝑥, 𝜃),
𝜕𝑥(2)

𝑓0
1= ̃EE𝜕
𝐹𝑓𝜕𝑝∙ Transformed KLD(∞−𝐷KL (𝜃, 𝜃0 ) = −∫∞log

𝑓0𝑓E)(1𝑓0 𝑑𝑥 =∫0log

𝜕 𝐹̃ E𝜕𝑝)𝑑𝑝(3)

https://t-kawanishi.w3.kanazawa-u.ac.jp/ Takuya Kawanishi

5Minimizing KLD as a BVP∙ To ﬁnd the minimizer, we look for the stationary points.
𝑛+11 𝐹̃ E𝑝𝑖 𝐹̃ E∑𝑝𝜃𝜃𝑝
𝜕𝑑𝑝 =𝑑𝑝.
0 = − 𝐷KL =EE
̃
̃
∫
∫
𝜕𝜃
0 𝐹𝑝
𝑖=1 𝑝𝑖−1 𝐹𝑝(4)∙ Boundary conditions:𝐹̃𝜃E (0; 𝜃) = 𝐹̃𝜃E (1; 𝜃) = 0∙ But lim𝑝↓0 𝐹 E (𝑝; 𝜃) or lim𝑝↑1 𝐹 E (𝑝; 𝜃) can be nonzero, including𝜃𝜃
inﬁnity.

https://t-kawanishi.w3.kanazawa-u.ac.jp/ Takuya Kawanishi6KLDMWR–collocation estimator (CE)∙ In the collocation method of MWR, we use the function itself
as the trial function and use Dirac’s delta function as theweighting function.
𝑛𝑛 𝐹̃ E |∑∑𝑓𝜃 (𝑋(𝑖) )𝜃𝑝 ||== 0,(𝑅, 𝑤) =|̃E𝑓 (𝑋(𝑖) )𝑖=1
𝑖=1 𝐹𝑝 ||𝑝𝑖(5)

which is nothing but the likelihood equation.
∙ 𝑋(𝑖) represents the order statistics of the sample 𝑋 .

∙ The collocation method leads to the ML.

https://t-kawanishi.w3.kanazawa-u.ac.jp/ Takuya Kawanishi7KLDMWR–subdomain estimator (SE)∙ In the subdomain method, the trial function is a simple
function: constant over each interval; the weighting functions
are indicator function of the interval. The trial functions areE𝐹̃𝜃𝑝𝐹̃𝑝E=

𝐹𝜃 (𝑋(𝑖) ) − 𝐹𝜃 (𝑋(𝑖−1) )𝐹 (𝑋(𝑖) ) − 𝐹 (𝑋(𝑖−1) )∙ Making the weighted residual zero (𝑅, 𝑤) = 0 yields
𝑛+11 ∑ 𝐹𝜃 (𝑋(𝑖) ) − 𝐹𝜃 (𝑋(𝑖−1) )(𝑅, 𝑤) =𝑛 + 1 𝑖=1 𝐹 (𝑋(𝑖) ) − 𝐹 (𝑋(𝑖−1) )=𝑛+11 ∑ 𝐷𝜃,𝑖=0
𝑛 + 1 𝑖=1 𝐷𝑖(6)∙ The subdomain method leads to the maximum product ofspacings (MPS) method.
https://t-kawanishi.w3.kanazawa-u.ac.jp/ Takuya Kawanishi

8

KLDMWR–Galerkin estimator (GE) 1∙ We use the same basis function for trial and weighting
functions.

∙ Basis function 𝑖 = 0, … , 𝑛 + 1,
⎧ 𝑦 − 𝑝𝑖−1
⎪𝑝 − 𝑝𝜙𝑖 = ⎨ 𝑝𝑖𝑖+1 −𝑖−1𝑦⎪⎩ 𝑝𝑖+1 − 𝑝𝑖
(d𝜙d𝑝)

1⎧⎪ 𝑝𝑖 − 𝑝𝑖−1
= ⎨ −1𝑖⎪⎩ 𝑝𝑖+1 − 𝑝𝑖

𝑝𝑖−1 ≤ 𝑦 ≤ 𝑝𝑖 , (𝑖 ≥ 1);𝑝𝑖 ≤ 𝑦 ≤ 𝑝𝑖+1 , (𝑖 ≤ 𝑛).

(7)

𝑝𝑖−1 ≤ 𝑦 ≤ 𝑝𝑖 , (𝑖 ≥ 1);𝑝𝑖 ≤ 𝑦 ≤ 𝑝𝑖+1 , (𝑖 ≤ 𝑛).

https://t-kawanishi.w3.kanazawa-u.ac.jp/ Takuya Kawanishi(8)

9

KLDMWR–Galerkin estimator (GE) 2∙ Trial functions()
𝑛+1( )ℎ ∑𝜕𝜙𝑖EẼ̃
𝐹𝑝=𝐹 (𝑝𝑖 )𝜕𝑝𝑖=0)(𝑛+1( )ℎ ∑
𝜕𝜙𝑖
EE
̃̃
𝐹𝜃𝑝 =𝐹𝜃 (𝑝𝑖 )𝜕𝑝𝑖=0∙ Weighting function
𝑤(𝑝) =𝑛∑𝑖=1𝜙𝑖 (𝑝)(9)enditemize∙ This weighting function satiﬁes the boundary conditions
𝑤(0) = 𝑤(1) = 0. (Note 𝐹̃𝜃E (0; 𝜃) = 𝐹̃𝜃E (1; 𝜃) = 0.)
https://t-kawanishi.w3.kanazawa-u.ac.jp/ Takuya Kawanishi

10

KLDMWR–Galerkin estimator (GE) 3∙ Let the weighting function be a linear combination of 𝜙𝑖 , then𝑝𝑖(∫𝑝𝑖−1=𝜕 𝐹̃𝜃E 𝜕𝑝
𝜕𝑝 𝜕 𝐹̃ E)ℎ𝑤 d𝑡𝑝𝑖 𝐹̃ E − 𝐹̃ E
𝜃,𝑖
𝜃,𝑖−1

∫𝑝𝑖−1

𝑝𝑖 − 𝑝𝑖−1

𝑝𝑖 − 𝑝𝑖−1𝐹̃ E − 𝐹̃ E{𝑡 − 𝑝𝑖−1
𝑝 −𝑡+ 𝑖𝑝𝑖 − 𝑝𝑖−1 𝑝𝑖 − 𝑝𝑖−1}
𝑑𝑡𝑖𝑖−1E − 𝐹̃ E𝐹̃𝜃,𝑖𝜃,𝑖−11 𝐹𝜃 (𝑋(𝑖) ) − 𝐹𝜃 (𝑋(𝑖−1) )= (𝑝𝑖 − 𝑝𝑖−1 ) E=;
E𝑛 + 1 𝐹 (𝑋(𝑖) ) − 𝐹 (𝑋(𝑖−1) )𝐹̃𝑖 − 𝐹̃𝑖−1()ℎ𝜕 𝐹̃𝜃E 𝜕𝑝𝐹𝜃 (𝑋(1) )1;𝑤 𝑑𝑡 =E∫0𝜕𝑝 𝜕 𝐹̃
2(𝑛 + 1) 𝐹 (𝑋(1) )( E)ℎ1
𝜕 𝐹̃𝜃 𝜕𝑝−𝐹𝜃 (𝑋(𝑛) )1𝑤 𝑑𝑡 =.
∫𝑝𝑛𝜕𝑝 𝜕 𝐹̃ E
2(𝑛 + 1) 1 − 𝐹 (𝑋(𝑛) )𝑝1

https://t-kawanishi.w3.kanazawa-u.ac.jp/ Takuya Kawanishi

11

KLDMWR–Galerkin estimator (GE) 4∙ Making the weighted residual zero,()ℎ⎛ 𝜕 𝐹̃ E⎞𝜕𝑝
𝜃⎜, 𝑤⎟ = 0,⎟⎜ 𝜕𝑝 𝜕 𝐹̃ E⎝⎠yields((𝑅, 𝑤) =1𝑛+1

1 𝐷𝜃,12 𝐷1+𝑛∑
𝐷𝜃,𝑖𝑖=2𝐷𝑖+1 𝐷𝜃,𝑛+12 𝐷𝑛+1)=0(10)∙ where 𝐷𝑖 ∶= 𝐹 (𝑋(𝑖) ) − 𝐹 (𝑋(𝑖−1) ), 𝐷𝜃,𝑖 ∶= 𝐹𝜃 (𝑋(𝑖) ) − 𝐹𝜃 (𝑋(𝑖−1) )∙ and 𝐹 (𝑋(0) ) = 𝐹𝜃 (𝑋(0) ) = 𝐹𝜃 (𝑋(𝑛+1) ) = 0, 𝐹 (𝑋(𝑛+1) ) = 1,

∙ The Galerkin method leads to Jiang’s modiﬁed maximum
product of spacings (JMMPS) method.
https://t-kawanishi.w3.kanazawa-u.ac.jp/ Takuya Kawanishi

12Advantages of MPS and JMMPS over ML∙ Applicable for nonregular cases.
∙ Let us assume that 𝑄E is linear in each interval [𝑝𝑖−1 , 𝑝𝑖 ), amodest assumption. Then for a ﬁxed 𝜃 ,⎧⎫⎪1 ⎪ ̃EẼ
𝑑𝑝 =⎨𝐹𝜃 (𝑝1 ; 𝜃) − 𝐹𝜃 (0; 𝜃)⎬EẼ̃
𝐹𝑝𝐹𝑝 ⎪⏟⏞⏟⏞⏟⎪⎩⎭=0𝑝1 𝐹̃ E𝜃𝑝∫0⎫⎧⎪⎪⎪1 ⎪ ̃EẼ𝐹𝑝 (1; 𝜃) −𝐹𝜃 (𝑝1 ; 𝜃)⎬
𝑑𝑝 =⎨EẼ̃∫𝑝𝑛 𝐹𝐹𝑝 ⎪⏟⏞⏟⏞⏟⎪𝑝⎪⎪ =0⎭⎩1 𝐹̃ E𝜃𝑝

∙ Even if 𝑓𝜃 ∕𝑓 is unbounded, ∫ 𝑝𝑖 𝐹̃ E ∕𝐹̃𝑝E 𝑑𝑝 can be calculated:𝑝𝑖−1 𝜃𝑝
𝐹̃ E is usually bounded.
𝜃

https://t-kawanishi.w3.kanazawa-u.ac.jp/ Takuya Kawanishi

13As boundary value problems∙ Problem of the dependence of distribution function on theparameter is converted to the singularity in boundary values.

∙ For example, for the three parameter Weibull (TPW)
distribution, ML does not work when the shape parameter𝜉 < 1. In that case, the boundary value is𝐹̃𝜃E (0, 𝜃) = 0|||lim 𝐹̃ E (𝑝, 𝜃)| = ∞| 𝑝↓0 𝜃|||∙ BVP in statistics? The boundary conditions are the propertiesof distribution functions.

https://t-kawanishi.w3.kanazawa-u.ac.jp/ Takuya Kawanishi

14Conclusions and future work

∙ With the KLDMWR framework we deal with the estimators, (atleast) MLE, MPSE, and JMMPSE under the same theory.

∙ The problem of the dependence of the support of distribution
on the parameter is converted to the singularity of theboundary conditions.

∙ Effects of the scheme of solving BVP sample performance ofestimators for small samples are to be investigated.

∙ Advantages and disadvantages of the estimators under thelight of the methods for BVP are to be explored.

https://t-kawanishi.w3.kanazawa-u.ac.jp/ Takuya Kawanishi

15

References

R. C. H. Cheng and N. a. K. Amin, Journal of the Royal StatisticalSociety: Series B (Methodological) 45, 394 (1983).
M. Ekström, Journal of Statistical Planning and Inference 138,
1778 (2008).
T. Kawanishi, Comp. Appl. Math. 39, (2020).
B. Ranneby, Scandinavian Journal of Statistics 11, 93 (1984).
R. L. SMITH, Biometrika 72, 67 (1985).

https://t-kawanishi.w3.kanazawa-u.ac.jp/ Takuya Kawanishi

16