# Some Notes on Concordance between Optimization and Statistics

Authors: Weiyan Mu; Qiuyue Wei; Shifeng Xiong
Journal: Mathematical Problems in Engineering
 (2019)

---

## Abstract

Many engineering problems require solutions to statistical optimization problems. When the global solution is hard to attain, engineers or statisticians always use the better solution because we intuitively believe a principle, called better solution principle (BSP) in this paper, that a better solution to a statistical optimization problem also has better statistical properties of interest. This principle displays some concordance between optimization and statistics and is expected to widely hold. Since theoretical study on BSP seems to be neglected by statisticians, this paper presents a primary discussion on BSP within a relatively general framework. We demonstrate two comparison theorems as the key results of this paper. Their applications to maximum likelihood estimation are presented. It can be seen that BSP for this problem holds under reasonable conditions; i.e., an estimator with greater likelihood is better in some statistical sense.


---

## Body

## 1. Introduction

Experimental design and data analysis in engineering rely on statistical methods that usually require solutions to optimization problems [1]. A notable example is maximum likelihood estimation, whose objective is to maximize the likelihood function. In fact, optimization methods are ubiquitous in almost all statistical areas and play a vital role in modern statistics. In the meanwhile, statisticians have to face the common difficulty in the optimization community; i.e., it is often extremely hard to obtain the global solution to a nonconvex optimization problem. A number of global optimization algorithms have been proposed, including the simulated annealing algorithm [2] and the genetic algorithm [3]. However, they can attain the global solution only in the probabilistic sense and often take an unrealistically long time to approach it in practice [4]. When handling large-scale data, the problem of multiple extrema becomes more serious. In fact, for such cases, it is also hard to obtain the solution to a convex optimization problem due to the unaffordable computational time and memory [5]. Another difficulty from the problem of multiple extrema is that we can rarely know whether a solution at hand is the global solution [6].When the global solution is hard to attain and/or to verify, for minimization problems, statisticians always take the solution whose objective value is as small as possible as the final solution. In other words, for two solutions, we always use the better one, where “better" should be understood in the sense of optimization. This seems reasonable in that the better solution is more likely to be the global solution, whose statistical properties of interest usually have been well established. From the statistical perspective, we use the better solution because we intuitively believe a principle, called better solution principle (BSP) in this paper, that a better solution to a statistical optimization problem also has better statistical properties of interest. This principle shows some concordance, or monotonicity, between optimization and statistics, and is expected to widely hold. Strictly speaking, a better solution can safely be used only after the corresponding BSP is verified. However, it is surprising that statisticians seem to neglect this problem, although we have actually made decisions following it ever since complex optimization problems appeared in statistics. To the best of the author’s knowledge, no paper has formally discussed BSP. For example, in the maximum likelihood problem, it is not clear to us whether a better solution with greater likelihood has higher estimation accuracy. Fairly recently, Xiong [7] introduced the better-fitting better-screening rule when discussing variable screening in high-dimensional linear models. This rule tells us that, under reasonable conditions, a subset with smaller residual sums of squares possesses better asymptotic screening properties, i.e., more likely to include the true submodel asymptotically. Such a subset is a better solution to the l
0-norm constrained least squares problem. Therefore, the better-fitting better-screening rule is actually the BSP for this problem.In this paper we provide some theoretical discussion on BSP in relatively general statistical optimization problems. The rest of the paper is organized as follows. We first present examples where BSP immediately holds in Section2. Such examples widely exist in experimental designs. They can help us understand BSP and the reason why we introduce the theorems in the following text. In Section 3, we demonstrate two comparison theorems which state that a better solution is more likely to have good statistical properties if the optimization problem possesses certain separation properties. These theorems, which look very simple and understandable, are effective to establish BSP in a general setting. Section 4 applies these results to maximum likelihood estimation. We can see that BSP for this problem holds under reasonable conditions. Section 5 concludes with some discussion.


## 2. Known Examples Where BSP Holds

We first present examples where BSP immediately holds. Such examples widely exist in experimental designs. They can help us understand BSP and the reason why we introduce the theorems in the following text. We take the D-optimal design [8] for example. Consider a regression model(1)
y
=
∑
i
=
1
d
θ
i
r
i
x
+
ε
,where the control variable x lies in a subset D of R
p, r
i’s are specified functions, θ
=
θ
1
,
…
,
θ
d
′ is the vector of unknown parameters, and ε is the random error. Given the sample size n, denote the experimental design by P
=
{
x
1
,
…
,
x
n
}. The information matrix of this design is M
(
P
)
=
R
P
′
R
(
P
), where (2)
R
P
=
r
1
x
1
⋯
r
d
x
1
⋮
⋱
⋮
r
1
x
n
⋯
r
d
x
n
.The D-optimal design minimizes the generalized variance of the least squares estimate of θ; i.e., it is the solution to the optimization problem(3)
m
i
n
x
i
∈
D
ψ
P
=
d
e
t
M
P
-
1
,where d
e
t denotes determinant. For two designs P
1 and P
2 with ψ
(
P
1
)
⩽
ψ
(
P
2
), it is clear that P
1 leads to a better estimator whose generalized variance is smaller. Therefore, if estimation accuracy, which is justified by generalized variance, is the statistical property of interest, then the BSP for problem (3) holds.The objective function in (3) itself is a statistical criterion, which does not involve any random variables. This is the reason why the BSP for (3) automatically holds. The same conclusion can be drawn for other model-based optimal designs [8] and minimum aberration designs [9]. For criterion-based space-filling designs (Santner, Williams, and Notz 2003), the geometric or discrepancy criteria used as objective functions seem not to have clear statistical interpretation. However, most of them relate to some desirable statistical properties. For example, the criteria for constructing the minimax distance design [10] and uniform design [11] can act as factors in the upper bounds of some estimation errors [12, 13]. If such estimation errors are used to evaluate the corresponding estimators, we can say that BSP holds.Design of experiments is a pre-sampling work (we do not consider sequential designs here), and thus the objective functions used in this area do not involve the random sample. In statistical inference, we have to deal with objective functions depending on the sample, which makes the problem of BSP more complicated. From the next section, we study whether BSP holds for sample-based optimization problems through introducing new definitions and theorems.


## 3. The Comparison Theorems

Let(
Ω
,
F
,
P
) be a probability space. For simplicity, it is assumed that all sets and maps throughout this paper are measurable with respect to according σ-fields. For each n
∈
N, the sample X
n of size n is a map from Ω to a space X
n. Let D denote the decision space that contains all statistical decisions of interest. Suppose that we need to make inferences based on the global solution to the optimization problem(4)
m
i
n
x
∈
D
ψ
n
x
,
X
n
,where the objective function ψ
n is a map from D
×
X
n to R. In general, the problem in (4) is proposed because its solution can asymptotically lie in a desirable subset A of D that contains all good decisions. This property can be viewed as a consistency property of the global solution.Consider the situations where the global solution to (4) is difficult to obtain. Suppose that there are K candidate solutions, ξ
n
(
1
)
,
…
,
ξ
n
(
K
). In practice, we always use ξ
n
∗, which denotes the one that takes the smallest value of ψ
n
(
·
,
X
n
) among them, as the final decision. For each ξ
n
(
k
), ξ
n
∗ is a better solution since the inequality (5)
ψ
n
ξ
n
∗
,
X
n
⩽
ψ
n
ξ
n
k
,
X
nalways holds. This section discusses whether BSP holds, i.e., whether such a better solution is more likely to lie in A.LetB be another subset of D, which contains relatively bad decisions compared to A.Definition 1.
We say that{
ψ
n
} strongly separates A from B or {
ψ
n
} has the strong separation property with respect of A and B, if as n
→
∞,(6)
P
s
u
p
x
∈
A
ψ
n
x
,
X
n
<
i
n
f
y
∈
B
ψ
n
y
,
X
n
→
1
.We say that {
ψ
n
} weakly separates A from B or {
ψ
n
} has the weak separation property with respect of A and B, if for all x
∈
A
,
y
∈
B,(7)
l
i
m
s
u
p
n
→
∞
ψ
n
x
,
X
n
-
ψ
n
y
,
X
n
<
0
a
.
s
.
,where “a.s." denotes “almost surely."The strong separation property needs not to imply its weak analogue. It is generally more difficult to verify and can lead to stronger results.Remark 2.
For convenience in asymptotic analysis, we often consider a scaled objective function. It should be pointed out that (6) holds if(8)
P
s
u
p
x
∈
A
ψ
n
x
,
X
n
a
n
<
i
n
f
y
∈
B
ψ
n
y
,
X
n
a
n
→
1for arbitrary sequence of positive numbers {
a
n
} and that (7) holds if (9)
lim
sup
n
→
∞
⁡
ψ
n
x
,
X
n
-
ψ
n
y
,
X
n
a
n
<
0
a.s. for a sequence of positive numbers {
a
n
} with a
n
-
1
=
O
(
1
).Roughly speaking, the strong separation property requires that the level set corresponding to smaller objective values is asymptotically identical to the set of good decisions. It shows the consistency of the objective function to the statistical properties of interest. We can immediately prove the following result that this property implies BSP.Theorem 3 (strong comparison theorem).
Suppose that{
ψ
n
} strongly separates A from B. For all n
∈
N, ξ
n and η
n are statistics valued in D satisfying P
(
ξ
n
∈
A
∪
B
,
η
n
∈
A
∪
B
)
→
1 as n
→
∞. If ψ
n
(
ξ
n
,
X
n
)
⩽
ψ
n
(
η
n
,
X
n
) for all n, then(10)
l
i
m
i
n
f
n
→
∞
P
ξ
n
∈
A
-
P
η
n
∈
A
⩾
0
.Proof.
Forω
∈
{
η
n
∈
A
,
ξ
n
∈
B
}
⊂
Ω, if ω
∈
s
u
p
x
∈
A
ψ
n
(
x
,
X
n
)
<
i
n
f
y
∈
B
ψ
n
(
y
,
X
n
)
∩
{
ξ
n
∈
A
∪
B
,
η
n
∈
A
∪
B
}, then (11)
ψ
n
η
n
X
n
ω
,
X
n
ω
⩽
s
u
p
x
∈
A
ψ
n
x
,
X
n
ω
<
i
n
f
y
∈
B
ψ
n
y
,
X
n
ω
⩽
ψ
n
ξ
n
X
n
ω
,
X
n
ω
,which leads to a contradiction. Therefore, ω
∉
s
u
p
x
∈
A
ψ
n
(
x
,
X
n
)
<
i
n
f
y
∈
B
ψ
n
(
y
,
X
n
)
∩
{
ξ
n
∈
A
∪
B
,
η
n
∈
A
∪
B
}, which implies P
(
η
n
∈
A
,
ξ
n
∈
B
)
→
0. We have P
(
η
n
∈
A
)
=
P
(
η
n
∈
A
,
ξ
n
∈
B
)
+
P
(
η
n
∈
A
,
ξ
n
∈
A
)
+
P
(
η
n
∈
A
,
ξ
n
∉
A
∪
B
)
=
P
(
ξ
n
∈
A
)
-
P
(
ξ
n
∈
A
,
η
n
∈
B
)
+
o
(
1
). This completes the proof.Theorem3 indicates that, for two candidate solutions ξ
n and η
n to the statistical optimization problem (4), the better one is asymptotically more likely to be a good decision, and thus the BSP holds. Recall that, for a decision ξ
n, the property that P
(
ξ
n
∈
A
)
→
1 can be viewed as a consistency property of ξ
n. The following theorem shows that the strong separation property of {
ψ
n
} is often stronger than the consistency of the minimum of ψ
n.Theorem 4.
Suppose that{
ψ
n
} strongly separates A from B. If ξ
n
=
a
r
g
m
i
n
x
∈
D
ψ
n
(
x
,
X
n
) exists and P
(
ξ
n
∈
A
∪
B
)
→
1 as n
→
∞, then (12)
l
i
m
n
→
∞
P
ξ
n
∈
A
→
1
.Proof.
This theorem follows from{
s
u
p
x
∈
A
ψ
n
(
x
,
X
n
)
<
i
n
f
y
∈
B
ψ
n
(
y
,
X
n
)
}
∩
{
ξ
n
∈
A
∪
B
}
⊂
{
ξ
n
∈
A
}.We next discuss BSP with the weak separation property. This weaker property cannot directly imply BSP, and more conditions are needed.Theorem 5 (weak comparison theorem).
Suppose that{
ψ
n
} weakly separates A from B. Denote the set of probability one where (7) holds by E
(
x
,
y
) and write E
=
∩
x
∈
A
,
y
∈
B
E
(
x
,
y
). For all n
∈
N, ξ
n and η
n are statistics valued in D satisfying P
(
ξ
n
∈
A
∪
B
,
η
n
∈
A
∪
B
)
→
1 as n
→
∞. If ψ
n
(
ξ
n
,
X
n
)
⩽
ψ
n
(
η
n
,
X
n
) for all n, then(13)
l
i
m
i
n
f
n
→
∞
P
ξ
n
∈
A
-
P
η
n
∈
A
⩾
-
P
Ω
\
E
.Proof.
Forω
∈
{
η
n
∈
A
,
ξ
n
∈
B
}
⊂
Ω, if ω
∈
E
∩
{
ξ
n
∈
A
∪
B
,
η
n
∈
A
∪
B
}, then (14)
l
i
m
s
u
p
n
→
∞
ψ
n
η
n
X
n
ω
,
X
n
ω
-
ψ
n
ξ
n
X
n
ω
,
X
n
ω
<
0
.This is a contradiction. Therefore, ω
∉
E
∩
{
ξ
n
∈
A
∪
B
,
η
n
∈
A
∪
B
} for sufficiently large n, which implies P
(
η
n
∈
A
,
ξ
n
∈
B
)
⩽
P
(
Ω
\
E
)
+
1
-
P
(
ξ
n
∈
A
∪
B
,
η
n
∈
A
∪
B
) for sufficiently large n. It follows that P
(
η
n
∈
A
)
=
P
(
η
n
∈
A
,
ξ
n
∈
B
)
+
P
(
η
n
∈
A
,
ξ
n
∈
A
)
+
P
(
η
n
∈
A
,
ξ
n
∉
A
∪
B
)
⩽
P
(
ξ
n
∈
A
)
-
P
(
ξ
n
∈
A
,
η
n
∈
B
)
+
P
(
η
n
∈
A
,
ξ
n
∈
B
)
+
1
-
P
(
ξ
n
∈
A
∪
B
)
⩽
P
(
ξ
n
∈
A
)
+
P
(
Ω
\
E
)
+
1
-
P
(
ξ
n
∈
A
∪
B
,
η
n
∈
A
∪
B
)
+
1
-
P
(
ξ
n
∈
A
∪
B
) for sufficiently large n, which completes the proof.IfP
(
Ω
\
E
) in (13) equals zero, then BSP holds. Nevertheless, it is impossible to verify this condition in practice. A way for avoiding this problem is to consider countable subsets, and we immediately obtain the following theorem.Theorem 6.
Suppose that{
ψ
n
} weakly separates A from B. For n
∈
N, ξ
n and η
n are statistics valued in a countable subset of D satisfying P
(
ξ
n
∈
A
∪
B
,
η
n
∈
A
∪
B
)
→
1 as n
→
∞. If ψ
n
(
ξ
n
,
X
n
)
⩽
ψ
n
(
η
n
,
X
n
) for all n, then(15)
l
i
m
i
n
f
n
→
∞
P
ξ
n
∈
A
-
P
η
n
∈
A
⩾
0
.Remark 7.
For many cases,D is a separable set. It is usually enough to consider the decisions in its countable and dense subset in practice. For example, to estimate a scalar parameter, we can always consider the estimators valued in the set of all rational numbers, which is countable and dense in R. In this sense, BSP follows from the weak separation property of {
ψ
n
}.In many problems like high-dimensional variable selection, the decision spaceD may depend on n. The definitions of the separation properties and the comparison theorems for such cases can be found in Xiong [14]. They have slightly different notation since sequences of decisions are involved, whereas the proofs are almost the same.In the rest of this paper, we omit the sampleX
n in ψ
n
(
·
,
X
n
) and write ψ
n
(
·
) for emphasizing the decision variable.


## 4. Greater Likelihood Principle

### 4.1. Separation Properties of the Likelihood Function

We have shown in Section3 that the separation properties of an objective function can imply the corresponding BSP. Despite simplicity, these results are effective to establish BSP since many objective functions indeed possess these properties under reasonable conditions. This section discusses the problem associated with maximum likelihood estimation.Let the dataX
1
,
…
,
X
n be i.i.d. (independently and identically distributed) from a probability density function f
(
·
,
θ
) with respect to a σ-finite measure ν on R
p, where θ lies in the parameter space Θ
⊂
R
q. The likelihood function is(16)
l
n
θ
=
∏
i
=
1
n
f
X
i
,
θ
,and the maximum likelihood estimator (MLE) is the solution to the optimization problem(17)
m
a
x
θ
∈
C
Θ
l
n
θ
,where C
(
Θ
) denotes the closure of Θ. For convenience, we write (17) as the problem of minimizing the negative log-likelihood(18)
m
i
n
θ
∈
C
Θ
-
l
o
g
l
n
θ
.The MLE is commonly used because of its well-known high asymptotic efficiency. However, when the negative log-likelihood has multiple local minima, the MLE is difficult to compute [6].When estimation accuracy is concerned, it is common to use the probability of lying in a neighborhood of the true parameter to evaluate an estimator. For a consistent estimator, this probability converges to one as the sample size goes to infinity. Following this way, we define good decisions in discussing the BSP for (18) and show that, for two estimators, the better one with greater likelihood has larger probability of lying in a sufficiently small neighborhood of θ
0 under regularity conditions, where θ
0 denotes the true parameter. This result, called greater likelihood principle in this paper, is a special case of BSP and can be viewed as a supplementary of the maximum likelihood principle.By the results in Section3, we can establish BSP via the separation properties of the objective function. Some assumptions and lemmas are needed.Denote(19)
s
θ
,
θ
0
=
-
∫
l
o
g
f
x
,
θ
f
x
,
θ
0
d
ν
x
.Assumption 8.
For allθ
1
,
θ
2
∈
Θ, f
(
·
,
θ
1
)
=
f
(
·
,
θ
2
) (a.s.) implies θ
1
=
θ
2.Assumption 9.
For allθ
∈
Θ, ∫
‍
|
l
o
g
(
f
(
x
,
θ
)
)
|
f
(
x
,
θ
)
d
ν
(
x
)
<
∞.Assumption 10.
For allθ
0
∈
Θ, s
(
·
,
θ
0
) is continuous on Θ and l
i
m
i
n
f
x
→
b
s
(
x
,
θ
0
)
>
s
(
θ
0
,
θ
0
) for all b
∈
C
∗
(
Θ
)
\
Θ, where C
∗
(
Θ
)
=
C
(
Θ
) if Θ is bounded and C
∗
(
Θ
)
=
C
(
Θ
)
∪
{
∞
} otherwise.Lemma 11.
Leth be a continuous function defined in D
⊂
R
q. Suppose that h has a unique minimum x
0; i.e., for all x
≠
x
0, h
(
x
)
>
h
(
x
0
). Furthermore, for all b
∈
C
∗
(
D
)
\
D, l
i
m
i
n
f
x
→
b
h
(
x
)
>
h
(
x
0
). Then for all ϵ
>
0, there exists δ
>
0 such that {
x
∈
D
:
h
(
x
)
-
h
(
x
0
)
⩽
δ
}
⊂
B
(
x
0
,
ϵ
), where B
(
x
0
,
ϵ
)
=
{
x
∈
R
q
:
x
-
x
0
⩽
ϵ
}.Proof.
For any sequence of positive numbers{
a
n
} with a
n
→
0 as n
→
∞, assume that there exist x
n
∈
D and ϵ
0
>
0 such that h
(
x
n
)
-
h
(
x
0
)
⩽
a
n but |
x
n
-
x
0
|
>
ϵ
0. Therefore h
(
x
n
)
→
h
(
x
0
). Since any limit point of {
x
n
} in C
∗
(
D
) cannot be x
0, this is in contradiction to the condition that x
0 is the unique minimum of h.Lemma 12.
If Assumptions8 and 9 hold, then for all θ
0
∈
Θ, s
(
·
,
θ
0
) in (19), as a function defined on Θ, attains its minimum uniquely at θ
0.The above lemma and its proof can be found in many places; see, e.g., Wald [15] and Van der Vaart [16].Under Assumptions8–10, by Lemmas 11 and 12, for all ϵ
>
0, there exists δ
(
ϵ
)
>
0 such that {
θ
∈
Θ
:
s
(
θ
,
θ
0
)
-
s
(
θ
0
,
θ
0
)
⩽
δ
(
ϵ
)
}
⊂
B
(
θ
0
,
ϵ
). Denote B
s
(
θ
0
,
ϵ
)
=
{
θ
∈
Θ
:
s
(
θ
,
θ
0
)
-
s
(
θ
0
,
θ
0
)
⩽
δ
(
ϵ
)
} and consider(20)
A
ϵ
=
B
s
θ
0
,
ϵ
,
B
ϵ
=
Θ
\
B
s
θ
0
,
ϵ
.Note that, for all θ
∈
Θ,(21)
-
l
o
g
l
n
θ
n
=
-
l
o
g
f
X
1
,
θ
+
⋯
+
l
o
g
f
X
n
,
θ
n
→
s
θ
,
θ
0
a
.
s
.
.We can immediately obtain the following theorem by Definition 1 and Remark 2.Theorem 13.
Under Assumptions8–10, for all ϵ
>
0, {
-
l
o
g
(
l
n
)
} weakly separates A
ϵ from B
ϵ in (20).Remark 14.
The conditions above are weaker than those for the consistency of maximum likelihood estimators in Wald [15]. Furthermore, our results in this section neither rely on the existence of an MLE nor require that Θ is an open or closed subset.Although the weak separation property is sufficient for BSP in practical use by Remark7, the strong separation property is still of theoretical interest. We next discuss it for the likelihood function. Some stronger conditions are needed.Assumption 15.
The family{
f
(
·
,
θ
)
}
θ
∈
Θ has a common support set S
=
{
x
∈
R
p
:
0
<
f
(
x
,
θ
)
<
∞
}. For all x
∈
S, f
(
x
,
·
) is continuous on Θ.Assumption 16.
For anyθ
∈
Θ and any compact subset K of Θ, (22)
∫
s
u
p
ϕ
∈
K
l
o
g
f
x
,
ϕ
f
x
,
θ
d
ν
x
<
∞
.TakeA
ϵ as in (20). Instead of B
ϵ in (20), take B
∗
ϵ as any compact subset of Θ
\
B
s
(
θ
0
,
ϵ
).Theorem 17.
Under Assumptions8, 10, 15, and 16, for all ϵ
>
0, {
-
l
o
g
(
l
n
)
} strongly separates A
ϵ from B
∗
ϵ.Proof.
Consider the Banach space of all continuous function onB
s
(
θ
0
,
ϵ
), which is separable since B
(
θ
0
,
ϵ
) is a compact subset of R
q. By Assumption 15, -
l
o
g
(
f
(
X
1
,
θ
)
)
,
…
,
-
l
o
g
(
f
(
X
n
,
θ
)
) are i.i.d. random variables valued in this Banach space. By Assumption 16 and the law of large numbers in Banach spaces (see, e.g., Corollary 7.10 in [17]),(23)
s
u
p
θ
∈
A
ϵ
-
l
o
g
l
n
θ
-
s
θ
,
θ
0
→
0
a
.
s
.
,which implies(24)
s
u
p
θ
∈
A
ϵ
-
l
o
g
l
n
θ
→
s
u
p
θ
∈
A
ϵ
s
θ
,
θ
0
a
.
s
.
.Similarly, we have(25)
i
n
f
θ
∈
B
∗
ϵ
-
l
o
g
l
n
θ
→
i
n
f
θ
∈
B
∗
ϵ
s
θ
,
θ
0
a
.
s
.
.Since B
∗
ϵ is compact, there exists δ
1
>
0 such that s
(
θ
,
θ
0
)
⩾
s
(
θ
0
,
θ
0
)
+
δ
+
δ
1 for all θ
∈
B
∗
ϵ. Consequently, by (24) and (25),(26)
P
s
u
p
θ
∈
A
ϵ
-
l
o
g
l
n
θ
<
i
n
f
θ
∈
B
∗
ϵ
-
l
o
g
l
n
θ
→
1
, which completes the proof.Remark 18.
If Assumptions15 and 16 hold, it can be proved that s
(
·
,
θ
0
) is continuous on Θ, which is assumed in Assumption 10.The strong separation property of the objective function provides a more strict guarantee of BSP than its weak analogue. However, at a price of this strictness, more restrictive conditions are required for verifying the strong separation property. By Theorem3, for comparing two estimators ξ
n and η
n via the strong separation property stated in Theorem 17, we require P
(
ξ
n
∈
A
ϵ
∪
B
∗
ϵ
)
→
1 and P
(
η
n
∈
A
ϵ
∪
B
∗
ϵ
)
→
1.


### 4.2. A Counter Example

A prerequisite of BSP is that the global solution has desirable statistical properties. Based on examples of inconsistent MLEs, we can find counter examples of the greater likelihood principle. The example discussed here is taken from Chen and Wu [18].LetX
1
,
…
,
X
n be i.i.d. from the following distribution: (27)
P
X
1
=
1
=
θ
i
f
θ
i
s
a
r
a
t
i
o
n
a
l
n
u
m
b
e
r
,
1
-
θ
o
t
h
e
r
w
i
s
e
,
P
X
1
=
0
=
1
-
P
X
1
=
1
,where θ
∈
[
0,1
] is the unknown parameter. It is not hard to show that the MLE of θ is the sample mean X
¯. However, if the true parameter θ
0 is an irrational number, as n
→
∞, X
¯
→
1
-
θ
0 (a.s.), which cannot be θ
0. Consider another estimator(28)
θ
^
=
1
-
X
¯
I
U
⩽
1
2
+
X
¯
I
U
>
1
2
,where U, independent of the sample, is drawn from the uniform distribution on [
0,1
] and I is the indicator function. We can show that if θ
0 is an irrational number, θ
^ has larger probability of lying in a sufficiently small neighborhood of θ
0 than X
¯ asymptotically, whereas it always produces smaller value of the likelihood function.


### 4.3. A Simulation Study

In this subsection we conduct a small simulation study to verify the greater likelihood principle in finite-sample cases. Consider a location family with the density function(29)
f
x
,
θ
=
f
0
x
-
θ
,where θ
∈
R is the unknown parameter that we want to estimate based on the i.i.d. observations X
1
,
…
,
X
n. Three types of f
0 are used: the standard normal distribution, t distribution with 5 degrees of freedom, and the Cauchy distribution with density f
0
(
x
)
=
[
π
(
x
2
+
1
)
]
-
1. It is known that the likelihood functions for the latter two cases often have multiple maximum. We compare three simple methods, the sample median, the trimmed mean removing 50
% extreme values, and the method that selects the better one of the two estimators with greater likelihood as the final estimator. Given the true parameter θ
0
=
0, we repeat 10,000 times to compute mean squares errors (MSEs) of the three estimators for various sample sizes, and the results are displayed in Table 1. It can be seen that the results follow the greater likelihood principle well: the better solution always yields the smallest MSEs among the three estimators.Table 1
MSE comparisons in Section4.3.
n
10
15
20
25
30
35
40
Normal
median
0.1361
0.1019
0.0728
0.0623
0.0502
0.0447
0.0373
trimmed mean
0.1113
0.0798
0.0588
0.0472
0.0393
0.0343
0.0291
better
0.1093
0.0776
0.0574
0.0459
0.0382
0.0333
0.0284
t
5
median
0.1588
0.1159
0.0824
0.0701
0.0568
0.0508
0.0392
trimmed mean
0.1393
0.0961
0.0698
0.0559
0.0465
0.0409
0.0316
better
0.1383
0.0951
0.0694
0.0555
0.0463
0.0405
0.0312
Cauchy
median
0.3360
0.2056
0.1427
0.1109
0.0905
0.0804
0.0061
trimmed mean
0.4929
0.2236
0.1628
0.1221
0.1027
0.0827
0.0224
better
0.3260
0.1857
0.1333
0.1001
0.0845
0.0720
0.0061


## 4.1. Separation Properties of the Likelihood Function

We have shown in Section3 that the separation properties of an objective function can imply the corresponding BSP. Despite simplicity, these results are effective to establish BSP since many objective functions indeed possess these properties under reasonable conditions. This section discusses the problem associated with maximum likelihood estimation.Let the dataX
1
,
…
,
X
n be i.i.d. (independently and identically distributed) from a probability density function f
(
·
,
θ
) with respect to a σ-finite measure ν on R
p, where θ lies in the parameter space Θ
⊂
R
q. The likelihood function is(16)
l
n
θ
=
∏
i
=
1
n
f
X
i
,
θ
,and the maximum likelihood estimator (MLE) is the solution to the optimization problem(17)
m
a
x
θ
∈
C
Θ
l
n
θ
,where C
(
Θ
) denotes the closure of Θ. For convenience, we write (17) as the problem of minimizing the negative log-likelihood(18)
m
i
n
θ
∈
C
Θ
-
l
o
g
l
n
θ
.The MLE is commonly used because of its well-known high asymptotic efficiency. However, when the negative log-likelihood has multiple local minima, the MLE is difficult to compute [6].When estimation accuracy is concerned, it is common to use the probability of lying in a neighborhood of the true parameter to evaluate an estimator. For a consistent estimator, this probability converges to one as the sample size goes to infinity. Following this way, we define good decisions in discussing the BSP for (18) and show that, for two estimators, the better one with greater likelihood has larger probability of lying in a sufficiently small neighborhood of θ
0 under regularity conditions, where θ
0 denotes the true parameter. This result, called greater likelihood principle in this paper, is a special case of BSP and can be viewed as a supplementary of the maximum likelihood principle.By the results in Section3, we can establish BSP via the separation properties of the objective function. Some assumptions and lemmas are needed.Denote(19)
s
θ
,
θ
0
=
-
∫
l
o
g
f
x
,
θ
f
x
,
θ
0
d
ν
x
.Assumption 8.
For allθ
1
,
θ
2
∈
Θ, f
(
·
,
θ
1
)
=
f
(
·
,
θ
2
) (a.s.) implies θ
1
=
θ
2.Assumption 9.
For allθ
∈
Θ, ∫
‍
|
l
o
g
(
f
(
x
,
θ
)
)
|
f
(
x
,
θ
)
d
ν
(
x
)
<
∞.Assumption 10.
For allθ
0
∈
Θ, s
(
·
,
θ
0
) is continuous on Θ and l
i
m
i
n
f
x
→
b
s
(
x
,
θ
0
)
>
s
(
θ
0
,
θ
0
) for all b
∈
C
∗
(
Θ
)
\
Θ, where C
∗
(
Θ
)
=
C
(
Θ
) if Θ is bounded and C
∗
(
Θ
)
=
C
(
Θ
)
∪
{
∞
} otherwise.Lemma 11.
Leth be a continuous function defined in D
⊂
R
q. Suppose that h has a unique minimum x
0; i.e., for all x
≠
x
0, h
(
x
)
>
h
(
x
0
). Furthermore, for all b
∈
C
∗
(
D
)
\
D, l
i
m
i
n
f
x
→
b
h
(
x
)
>
h
(
x
0
). Then for all ϵ
>
0, there exists δ
>
0 such that {
x
∈
D
:
h
(
x
)
-
h
(
x
0
)
⩽
δ
}
⊂
B
(
x
0
,
ϵ
), where B
(
x
0
,
ϵ
)
=
{
x
∈
R
q
:
x
-
x
0
⩽
ϵ
}.Proof.
For any sequence of positive numbers{
a
n
} with a
n
→
0 as n
→
∞, assume that there exist x
n
∈
D and ϵ
0
>
0 such that h
(
x
n
)
-
h
(
x
0
)
⩽
a
n but |
x
n
-
x
0
|
>
ϵ
0. Therefore h
(
x
n
)
→
h
(
x
0
). Since any limit point of {
x
n
} in C
∗
(
D
) cannot be x
0, this is in contradiction to the condition that x
0 is the unique minimum of h.Lemma 12.
If Assumptions8 and 9 hold, then for all θ
0
∈
Θ, s
(
·
,
θ
0
) in (19), as a function defined on Θ, attains its minimum uniquely at θ
0.The above lemma and its proof can be found in many places; see, e.g., Wald [15] and Van der Vaart [16].Under Assumptions8–10, by Lemmas 11 and 12, for all ϵ
>
0, there exists δ
(
ϵ
)
>
0 such that {
θ
∈
Θ
:
s
(
θ
,
θ
0
)
-
s
(
θ
0
,
θ
0
)
⩽
δ
(
ϵ
)
}
⊂
B
(
θ
0
,
ϵ
). Denote B
s
(
θ
0
,
ϵ
)
=
{
θ
∈
Θ
:
s
(
θ
,
θ
0
)
-
s
(
θ
0
,
θ
0
)
⩽
δ
(
ϵ
)
} and consider(20)
A
ϵ
=
B
s
θ
0
,
ϵ
,
B
ϵ
=
Θ
\
B
s
θ
0
,
ϵ
.Note that, for all θ
∈
Θ,(21)
-
l
o
g
l
n
θ
n
=
-
l
o
g
f
X
1
,
θ
+
⋯
+
l
o
g
f
X
n
,
θ
n
→
s
θ
,
θ
0
a
.
s
.
.We can immediately obtain the following theorem by Definition 1 and Remark 2.Theorem 13.
Under Assumptions8–10, for all ϵ
>
0, {
-
l
o
g
(
l
n
)
} weakly separates A
ϵ from B
ϵ in (20).Remark 14.
The conditions above are weaker than those for the consistency of maximum likelihood estimators in Wald [15]. Furthermore, our results in this section neither rely on the existence of an MLE nor require that Θ is an open or closed subset.Although the weak separation property is sufficient for BSP in practical use by Remark7, the strong separation property is still of theoretical interest. We next discuss it for the likelihood function. Some stronger conditions are needed.Assumption 15.
The family{
f
(
·
,
θ
)
}
θ
∈
Θ has a common support set S
=
{
x
∈
R
p
:
0
<
f
(
x
,
θ
)
<
∞
}. For all x
∈
S, f
(
x
,
·
) is continuous on Θ.Assumption 16.
For anyθ
∈
Θ and any compact subset K of Θ, (22)
∫
s
u
p
ϕ
∈
K
l
o
g
f
x
,
ϕ
f
x
,
θ
d
ν
x
<
∞
.TakeA
ϵ as in (20). Instead of B
ϵ in (20), take B
∗
ϵ as any compact subset of Θ
\
B
s
(
θ
0
,
ϵ
).Theorem 17.
Under Assumptions8, 10, 15, and 16, for all ϵ
>
0, {
-
l
o
g
(
l
n
)
} strongly separates A
ϵ from B
∗
ϵ.Proof.
Consider the Banach space of all continuous function onB
s
(
θ
0
,
ϵ
), which is separable since B
(
θ
0
,
ϵ
) is a compact subset of R
q. By Assumption 15, -
l
o
g
(
f
(
X
1
,
θ
)
)
,
…
,
-
l
o
g
(
f
(
X
n
,
θ
)
) are i.i.d. random variables valued in this Banach space. By Assumption 16 and the law of large numbers in Banach spaces (see, e.g., Corollary 7.10 in [17]),(23)
s
u
p
θ
∈
A
ϵ
-
l
o
g
l
n
θ
-
s
θ
,
θ
0
→
0
a
.
s
.
,which implies(24)
s
u
p
θ
∈
A
ϵ
-
l
o
g
l
n
θ
→
s
u
p
θ
∈
A
ϵ
s
θ
,
θ
0
a
.
s
.
.Similarly, we have(25)
i
n
f
θ
∈
B
∗
ϵ
-
l
o
g
l
n
θ
→
i
n
f
θ
∈
B
∗
ϵ
s
θ
,
θ
0
a
.
s
.
.Since B
∗
ϵ is compact, there exists δ
1
>
0 such that s
(
θ
,
θ
0
)
⩾
s
(
θ
0
,
θ
0
)
+
δ
+
δ
1 for all θ
∈
B
∗
ϵ. Consequently, by (24) and (25),(26)
P
s
u
p
θ
∈
A
ϵ
-
l
o
g
l
n
θ
<
i
n
f
θ
∈
B
∗
ϵ
-
l
o
g
l
n
θ
→
1
, which completes the proof.Remark 18.
If Assumptions15 and 16 hold, it can be proved that s
(
·
,
θ
0
) is continuous on Θ, which is assumed in Assumption 10.The strong separation property of the objective function provides a more strict guarantee of BSP than its weak analogue. However, at a price of this strictness, more restrictive conditions are required for verifying the strong separation property. By Theorem3, for comparing two estimators ξ
n and η
n via the strong separation property stated in Theorem 17, we require P
(
ξ
n
∈
A
ϵ
∪
B
∗
ϵ
)
→
1 and P
(
η
n
∈
A
ϵ
∪
B
∗
ϵ
)
→
1.


## 4.2. A Counter Example

A prerequisite of BSP is that the global solution has desirable statistical properties. Based on examples of inconsistent MLEs, we can find counter examples of the greater likelihood principle. The example discussed here is taken from Chen and Wu [18].LetX
1
,
…
,
X
n be i.i.d. from the following distribution: (27)
P
X
1
=
1
=
θ
i
f
θ
i
s
a
r
a
t
i
o
n
a
l
n
u
m
b
e
r
,
1
-
θ
o
t
h
e
r
w
i
s
e
,
P
X
1
=
0
=
1
-
P
X
1
=
1
,where θ
∈
[
0,1
] is the unknown parameter. It is not hard to show that the MLE of θ is the sample mean X
¯. However, if the true parameter θ
0 is an irrational number, as n
→
∞, X
¯
→
1
-
θ
0 (a.s.), which cannot be θ
0. Consider another estimator(28)
θ
^
=
1
-
X
¯
I
U
⩽
1
2
+
X
¯
I
U
>
1
2
,where U, independent of the sample, is drawn from the uniform distribution on [
0,1
] and I is the indicator function. We can show that if θ
0 is an irrational number, θ
^ has larger probability of lying in a sufficiently small neighborhood of θ
0 than X
¯ asymptotically, whereas it always produces smaller value of the likelihood function.


## 4.3. A Simulation Study

In this subsection we conduct a small simulation study to verify the greater likelihood principle in finite-sample cases. Consider a location family with the density function(29)
f
x
,
θ
=
f
0
x
-
θ
,where θ
∈
R is the unknown parameter that we want to estimate based on the i.i.d. observations X
1
,
…
,
X
n. Three types of f
0 are used: the standard normal distribution, t distribution with 5 degrees of freedom, and the Cauchy distribution with density f
0
(
x
)
=
[
π
(
x
2
+
1
)
]
-
1. It is known that the likelihood functions for the latter two cases often have multiple maximum. We compare three simple methods, the sample median, the trimmed mean removing 50
% extreme values, and the method that selects the better one of the two estimators with greater likelihood as the final estimator. Given the true parameter θ
0
=
0, we repeat 10,000 times to compute mean squares errors (MSEs) of the three estimators for various sample sizes, and the results are displayed in Table 1. It can be seen that the results follow the greater likelihood principle well: the better solution always yields the smallest MSEs among the three estimators.Table 1
MSE comparisons in Section4.3.
n
10
15
20
25
30
35
40
Normal
median
0.1361
0.1019
0.0728
0.0623
0.0502
0.0447
0.0373
trimmed mean
0.1113
0.0798
0.0588
0.0472
0.0393
0.0343
0.0291
better
0.1093
0.0776
0.0574
0.0459
0.0382
0.0333
0.0284
t
5
median
0.1588
0.1159
0.0824
0.0701
0.0568
0.0508
0.0392
trimmed mean
0.1393
0.0961
0.0698
0.0559
0.0465
0.0409
0.0316
better
0.1383
0.0951
0.0694
0.0555
0.0463
0.0405
0.0312
Cauchy
median
0.3360
0.2056
0.1427
0.1109
0.0905
0.0804
0.0061
trimmed mean
0.4929
0.2236
0.1628
0.1221
0.1027
0.0827
0.0224
better
0.3260
0.1857
0.1333
0.1001
0.0845
0.0720
0.0061


## 5. Discussion

When the global solution to a statistical optimization problem is difficult to obtain, BSP theoretically supports to the method of using the solution whose objective value is as small as possible (for minimization problems). Interestingly, it can be studied within a simple framework based on several obvious but effective comparison theorems. These theorems tell us that a better solution with smaller objective value is more likely to be a good decision if the objective function has the separation property. Therefore, it suffices to prove the separation property of the objective function for verifying BSP. Following this way, we have discussed BSP for the maximum likelihood problem. Further applications of our results can be found in Xiong [14].Recently, Big Data begins to pose significant challenges to statistics [19]. For analyzing Big Data, not only statistical methodology but also statistical theory should be considered based on computation. BSP can be viewed as a computational ability-based statistical theory, and we expect that BSP and related methodologies will be paid more attention to in the future.



---

*Source: 3485064-2019-01-16.xml*