A General Robust Bayes Pseudo-Posterior:
Exponential Convergence results with Applications

arXiv:1708.09692v1 [] 31 Aug 2017

Abhik Ghosh and Ayanendranath Basu

Abstract
Although Bayesian inference is an immensely popular paradigm among a large segment of scientists including statisticians, most of the applications consider the objective
priors and need critical investigations (Efron, 2013). And although it has several optimal properties, one major drawback of Bayesian inference is the lack of robustness
against data contamination and model misspecification, which becomes pernicious in
the use of objective priors. This paper presents the general formulation of a Bayes
pseudo-posterior distribution yielding robust inference. Exponential convergence results related to the new pseudo-posterior and the corresponding Bayes estimators are
established under the general parametric set-up and illustrations are provided for the
independent stationary models and the independent non-homogenous models. For the
first case, the discrete priors and the corresponding maximum posterior estimators are
discussed with additional details. We further apply this new pseudo-posterior to propose robust versions of the Bayes predictive density estimators and the expected Bayes
estimator for the fixed-design (normal) linear regression models; their properties are
illustrated both theoretically as well as empirically.
Keywords: Robust Bayes Estimator; Pseudo-Posterior; Density Power Divergence;
Exponential Convergence; Predictive Density Estimator; Bayesian Linear Regression.

1

Introduction

Bayesian analysis is arguably one of the most popular statistical paradigms with increasing applications across different disciplines of science and industry. It is widely preferred
by many non-statisticians due to its nice interpretability and incorporation of the prior
knowledge about experimental quantities. From a statistical point of view, it is widely
accepted even among many non-Bayesians, because of its nice optimal (asymptotic) properties. Bayesian inference is built on the famous ‘Bayes theorem’, the celebrated 1763 paper
of Thomas Bayes, which combines prior knowledge with experimental evidence to produce
the posterior conclusion. However, over these 250 years of applications, Bayesian inference
has also been subject to several criticisms and some of these debates are still ongoing; Efron
(2013) termed the Bayes’ theorem as a “controversial theorem”. Other than the controversies about its internal logic (Halpern, 1999a,b; Arnborg and Sjodin, 2001; Dupre and Tipler,
2009), a major practical drawback of Bayesian inference is its non-robust nature against
1

misspecification in models (including data contamination and outliers) and the priors, as
has been extensively observed in the literature; see Berk (1966), McCulloch (1989), Weiss
(1996), Weiss and Cho (1998), Millar and Stewart (2007), De Blasi and Walker (2012),
Walker (2013), Owhadi and Scovel (2014, 2015), Owhadi et al. (2013, 2015a,b) and the references therein. However, the optimal solution to this problem has been developed mainly
for prior misspecifications and has a long history (Berger, 1984, 1994; Berger and Berliner,
1986; Wasserman, 1990; Gelfand and Dey, 1991; Dey and Birmiwal, 1994; Delampady and
Dey, 1994; Dey et al., 1996; Gustafson and Wasserman, 1995; Ghosh et al., 2006; Martin
et al., 2009); this is because the Bayesians traditionally suggest the model to be prefixed
perfectly. Thus the possibility of model misspecification has been generally ignored.
In applying Bayesian inference to different complicated datasets of the present era,
however, we need to use many complex and sophisticated models which are highly prone
to misspecification. In reality, where “All models are wrong” (Box, 1976), the Bayesian
philosophy of refining the fixed model adaptively (Gelman et al., 1996) often fails to handle
complex modern scenarios or leads to ‘a model as complex as the data’ (Wang and Blei,
2016). These certainly led to severe consequences in posterior conclusions having erroneous
implications. The problem becomes more clear but pernicious in case of inference with
objective or reference priors. As a simple example, the Bayes estimate of the mean of a
normal model, with any objective prior and symmetric loss function, is the extremely nonrobust sample mean. What is a matter of greater concern, as noted by Efron (2013), is
that most of the recent applications of Bayesian inference hinge on objective priors and so
they always need to be scrutinized carefully, sometime even from a frequentist perspective.
The posterior non-robustness against model misspecification makes the process vulnerable
and we surely need an appropriate solution to this problem which gives due regard to the
robustness issue.
From a true Bayesian perspective, there are only few solutions to the problem of model
misspecification (Ritov, 1985, 1987; Sivaganesan, 1993; Dey et al., 1996; Shyamalkumar,
2000). However, most of them, if not all, assume that the perturbation in the model
is known beforehand, such as gross error contaminated models with known contamination
proportion . This is rarely possible in practice, specially with the modern complex datasets
and so it restricts their real-life applications. Some attempts have been made to develop
alternative solutions by linking Bayesian inference suitably with the frequentist concept of
robustness. Note that, in the frequentist sense, there are two major approaches to achieve
robustness, namely the use of heavy tailed distributions (e.g., t-distribution in place of
normal), or new (robust) inference methodologies (Hampel et al., 1986; Basu et al., 2011).
The first one has been adapted by some Bayesian scientists; see Andrade and O’Hagan
(2006, 2011) and Desgagne (2013) among others. However, the difficulty with this approach
is the availability of appropriate heavy tailed alternatives in complex scenarios and it indeed
does not solve the non-robustness of Bayesian inference for a specified model (which might
be of a lighter tail). The second approach of frequentist robustness, namely the modified
2

inference methodologies, serves the purpose but differs in the strictest probabilistic sense
from the Bayesian philosophy, since one needs to alter the posterior density appropriately
to achieve robustness against data contamination or model misspecification; the resulting
modified posteriors are generally refereed to as pseudo-posterior densities. Different such
pseudo-posteriors have been proposed by Greco et al. (2008), Agostinelli and Greco (2013),
Cabras et al. (2014), Hooker and Vidyashankar (2014), Ghosh and Basu (2016), Danesi
et al. (2016), Atkinson et al. (2017) and Nakagawa and Hashimoto (2017); all of them
have primarily considered independent stationary models and have different pros and cons.
Another recent attempt, in the borderline of these two approaches, has been proposed by
Wang and Blei (2016), who have transformed the given model to a localized model involving
hyperparameters to be estimated through the empirical Bayes approach.
Here, we consider a particular pseudo-posterior originally proposed by Ghosh and Basu
(2016) in the independently and identically distributed (IID) set-up. The choice of this
pseudo-posterior has been motivated by the several particularly nice properties that it has
been observed to possess, and its potential for extension to more general set-ups. As a brief
description, consider n IID random variables X1 , . . . , Xn taking values in a measurable
space (χ, B). Assume that there is an underlying true probability space (Ω, BΩ , P ) such
that, for i = 1, . . . , n, Xi is B/Ω measurable, independent with respect to P and it’s
induced distribution G(x) has an absolutely continuous density g(x) with respect to some
dominating σ-finite measure λ(dx). We model G by a parametric family of distributions
{Fθ : θ ∈ Θ ⊆ Rp } which is assumed to be absolutely continuous with respect to λ having
density fθ . Consider a prior density for θ over the parameter space Θ given by π(θ).
Following Ghosh and Basu (2016), the robust pseudo-posterior density, namely the R(α) posterior density of θ, given the sample observation xn = (x1 , . . . , xn )T on the random
variable X n = (X1 , . . . , Xn )T , is defined as
(α)

πn(α) (θ|xn ) = R

exp(qn (xn |θ))π(θ)
(α)

exp(qn (xn |θ 0 ))π(θ 0 )dθ 0

,

α ≥ 0,

(1)

(α)

where qn (xn |θ) is the α-likelihood of xn given by
qn(α) (xn |θ)

=
=



Z
Z
n
1+α
n
1+α
α
fθ dGn − fθ
−
1+α
α
α
Z
n
n
1X α
n
n X (α)
1+α
fθ (xi ) −
fθ − =
qθ (xi ),
α
1+α
α
i=1

(2)

i=1

with Gn being the empirical distribution based on the data and
Z
1
1
(α)
qθ (y) = (fθα (y) − 1) −
fθ1+α .
α
1+α
(0)

Note that, in a limiting sense, qn (xn |θ) = n
3

R

log(fθ )dGn − n =

(3)
Pn

i=1 (log(fθ (xi ))

− 1),

which is simply the usual log-likelihood (plus a constant); so the R(0) -posterior is just
the ordinary Bayes posterior. The idea came from a frequentist robust estimator, namely
the minimum density power divergence (DPD) estimator (MDPDE) of Basu et al. (1998),
which has proven to be a useful robust generalization of the classical maximum likelihood
estimator (MLE); see Ghosh and Basu (2016) for more details. The important similarity
of this approach (at α > 0) with the usual Bayes posterior (at α = 0) is that, it does
not require nonprametric smoothing like some other pseudo-posteriors and it is additive
in the data so that the posterior update is easily possible with new observations. Ghosh
and Basu (2016) extensively illustrated its robustness with respect to the model and prior
misspecification and proved a Bernstein-von Mises type limiting result under the IID set-up.
In this paper, we provide a generalization of the R(α) -posterior density for a completely
general model set-up beyond IID data, by imposing a suitable structure and conditions
to define the α-likelihood function. Their forms are explicitly derived for several important applications like the independent non-homogeneous data including regressions, time
series and Markov model data, diffusion processes etc. The concept of the associated R(α) marginal density of data has also been introduced. In addition to the Ghosh and Basu
(2016) interpretations, we show that the R(α) -posterior density can also be thought of as an
ordinary Bayes posterior with a suitably modified model and modified prior (see Remark
2.1), and hence we are also able to retain the conditional probability interpretation through
this proposed approach of robust Bayes analysis.
As a second major contribution of the present paper, we derive the exponential convergence results associated with the new R(α) -posterior probabilities under a completely general set-up. This, in fact, generalizes the corresponding results for the usual Bayes posterior
(Barron, 1988) and is seen to hold under appropriate assumptions based on the concepts
of merging of distributions in probability, (modified) prior negligibility and existence of
uniform exponential consistent tests. Some simpler conditions indicating the merging in
probability phenomenon are derived for the R(α) -posterior under the general set-up. The
required assumptions are explicitly studied and simplified for the two most common set-ups,
namely for the IID data and the independent non-homogeneous data including fixed-design
regressions; the results provide some significant insights about their convergence rates. Further, as an application, the exponential consistency of the Bayes estimates associated with
the R(α) -posterior is shown for a general class of “bounded” loss functions. The interesting
cases of discrete priors are discussed separately, along with the exponential consistency of
the corresponding maximum R(α) -posterior estimator under the IID set-up. These optimality results further justify the usefulness of the proposed R(α) -Bayes estimators besides their
already proved robustness advantages.
An important application of the Bayesian inference is in predictive density estimation for
future data. As a third contribution, we propose a robust version of the Bayesian predictive
density estimator based on the R(α) -posterior density and prove its exponential consistency
for the loss functions like squared error loss, absolute error loss, Hellinger loss and 0-1 loss.
4

Their robustness properties are also illustrated empirically.
The rest of the paper is organized as follows. In Section 2, we extend the form of the
(α)
R -posterior density for a completely general set-up of parametric estimation along with
several examples. Section 3 presents the main results on the exponential consistency of the
R(α) -posterior probabilities and the R(α) -Bayes estimators. In Section 4, we apply these
general results to the independent stationary models. The cases of discrete priors and the
robust Bayes predictive density estimators are discussed in Sections 4.2 and 4.3, respectively.
Section 5 presents the applications to the independent non-homogeneous models including
the fixed design regressions. The usefulness of the proposed R(α) -posterior are also examined
numerically in the linear regression models in this section. Finally the paper ends with a
brief concluding discussion. The issues of merging in probability and the associated results
are presented in Appendix A. Proofs of all theorems are given in the Online Supplement.

2

A general form of the R(α) -posterior distribution

In order to extend the R(α) -posterior density to a more general set-up, let us assume that
the random variable X n is defined on a general measurable space (χn , Bn ) for each n. Also
assume that there is an underlying true probability space (Ω, BΩ , P ) such that, for each
n ≥ 1, X n is Bn /Ω measurable and its induced distribution Gn (xn ) is absolutely continuous
with respect to some σ-finite measure λn (dxn ) having “true” probability density g n (xn ). We
model it by a parametric family of distributions Fn = {F n (·|θ) : θ ∈ Θn ⊆ Rp } where the
elements of Fn are assumed to be absolutely continuous with respect to λn having density
f n (xn |θ). Note that, we have not assumed the parameter space Θn to be independent of
the sample size n. Similarly, the prior measure πn (θ) on Θn may be n-dependent with
πn (Θn ) ≤ 1. Consider a σ-field BΘn on the parameter space Θn . Generalizing from (2), we
(α)
suitably define the α-likelihood function qn (xn |θ) in such a way that ensures
qn(0) (xn |θ) := lim qn(α) (xn |θ) = log f n (xn |θ) − n, for all xn ∈ χn .
α↓0

(4)

Our definition should guarantee that the α-likelihood, as a function of θ, is BΘn measurable
for each xn and jointly Bn × BΘn measurable when both X n and θ are random. Then, for
this general set-up, we define the corresponding R(α) -posterior probabilities following (1) as
R
πn(α) (An |xn )

= R An

(α)

exp(qn (xn |θ))πn (θ)dθ
(α)

Θn exp(qn (xn |θ))πn (θ)dθ

,

An ∈ BΘn ,

(5)

whenever the denominator is finitely defined and is positive; otherwise we may define it
(0)
(α)
arbitrarily, e.g., πn (An |xn ) = πn (An ). (4) ensures that πn is the usual Bayes posterior.
R
(α)
(α)
For an alternative representation, we define Qn (Sn |θ) := Sn exp(qn (xn |θ))dxn ,
R
(α)
(α)
(α)
(α)
(α)
Mn (Sn , An ) := An Qn (Sn |θ)πn (θ)dθ and Mn (Sn ) := Mn (Sn , Θn )/Mn (χn , Θn ) for
Sn ∈ Bn and An ∈ BΘn . Throughout this paper, we will assume that the model and priors
5

(α)

are chosen so as to satisfy 0 < Mn (χn , Θn ) < ∞. Then, the last two measures have denR
(α)
(α)
sity functions with respect to λn (dxn ) given by mn (xn , An ) = An exp(qn (xn |θ))πn (θ)dθ
(α)

(α)

(α)

(α)

and mn (xn ) = mn (xn , Θn )/Mn (χn , Θn ) respectively. Clearly, mn (xn ) is a proper
probability density function, which we refer to as the R(α) -marginal density of X n ; the cor(α)
responding R(α) -marginal distribution is Mn (·). It gives a robust version of the ordinary
(0)
Bayes marginal for α > 0; mn (xn ) corresponds to the usual marginal density of X n since
(0)
(0)
(0)
Qn (Sn |θ) = e−n F n (Sn |θ) and Mn (χn , Θn ) = Qn (χn |θ) = e−n by (4). In terms of the
R(α) -marginal density, we can re-express the R(α) -posterior probabilities from (5) as
(α)

πn(α) (An |xn ) =

mn (xn , An )
(α)

(α)

=

(α)

mn (xn , An )/Mn (χn , Θn )
(α)

mn (xn , Θn )

mn (xn )

,

An ∈ BΘn ,

(6)

(α)

whenever 0 < mn (xn ) < ∞. Then the R(α) -Bayes joint posterior distribution of the
parameter θ and the data X n is defined as
(α)
(α)
L(α)Bayes
(dθ, dxn ) = πn(α) (dθ|xn ) m(α)
n (dxn ) = mn (dxn , dθ)/Mn (χn , Θn ).
n

(7)

(α)

This gives a nice interpretation of the deduced measure Mn (Sn , An ) which is, after suitable standardization, the product measure associated with the R(α) -Bayes joint posterior
distribution of θ and X n . At α = 0, all these again simplify to the ordinary Bayes measures.
Example 2.1 [Independent Stationary Data]:
The simplest possible set-up is that of IID observations as described in Section 1. In terms
of the general notation presented above, we have X n = (X1 , . . . , Xn ) with its observed
value xn = (x1 , . . . , xn ) and the general measurable space (χn , Bn ) is the n-fold product of
(χ, B). Assuming the same true probability space (Ω, BΩ , P ), the induced data distribution
Q
Q
is Gn (xn ) = ni=1 G(xi ) which has an absolutely continuous density g n (xn ) = ni=1 g(xi )
Q
with respect to the product σ-finite measure λn (dxn ) = ni=1 λ(xi ). Similarly the model
distribution and density, in their general notation, also have the product forms: F n (xn |θ) =
Qn
Qn
n
i=1 Fθ (xi ), f (xn |θ) =
i=1 fθ (xi ) and so Fn is the n-fold product of the family of
(α)
individual distributions Fθ . Under these notations, the α-likelihood qn (xn |θ) is given by
(2) which satisfies the required measurability assumptions along with the condition in (4).
Then, under suitable assumptions on the prior distribution as before, the corresponding
(α)
R -posterior distribution is defined by (5) which is now equivalent to (1) and can be written
as a product of stationary independent terms corresponding to each xi . Also, the deduced
Q
(α)
measure becomes Qn (Sn |θ) = ni=1 Q(α) (S i |θ) for any Sn = S 1 × S 2 × · · · × S n ∈ Bn
R
(α)
(α)
with S i ∈ B for all i and Q(α) (S i |θ) = S i exp(qθ (y))dy, where qθ (y) is as defined in (3).
Other related measures can be defined from these; details are discussed in Section 4.

Example 2.2 [Independent Non-homogeneous Data]:
Suppose X1 , . . . , Xn are independently but not identically distributed random variables,
where each Xi is defined on a measurable space (χi , B i ) for i = 1, . . . , n. Considering an

6

underlying common probability space (Ω, BΩ , P ), the random variable Xi is assumed to be
B i /Ω measurable, independent with respect to P and its induced distribution Gi (x) has
an absolutely continuous density gi (x) with respect to some common dominating σ-finite
measure λ(dx), for each i = 1, . . . , n. The true distributions Gi is to be modeled by a
parametric family F i = {Fi,θ : θ ∈ Θ ⊆ Rp } which is absolutely continuous with respect
to λ having density fi,θ . Note that, although the densities are potentially different for each
i, they are assumed to share the common unknown parameter θ leaving us with enough
degrees of freedom for “good” estimation of this parameter θ.
This set-up of independent non-homogeneous observations, which we refer to as the
I-NH set-up, covers many interesting practical problems, the most common one being the
regression with fixed design. Suppose t1 , . . . , tn be n fixed, k-variate design points. For
each i = 1, . . . , n, given ti we independently observe xi which has the parametric model
density fi,θ (xi ) = f (xi ; ti , θ) depending on ti through a regression structure. For example,
E(Xi ) = ψ(ti , β),

i = 1, . . . , n,

(8)

where β ⊆ θ is the unknown regression coefficients and ψ is a suitable link function.
In general, the unknown parameter θ = (β, σ) may additionally contain some variance
parameter σ. For the subclass of generalized linear models, we take ψ(ti , β) = ψ(tTi β)
and f from the exponential family of distributions. For normal linear regression, we have
ψ(ti , β) = tTi β and f is the normal density with mean tTi β and variance σ 2 . Here, the
underlying random variables Xi s, associated with observations xi s, have the I-NH structure
with the common parameter θ = (β, σ) and the different densities fi,θ . We can further
extend this set-up to include the heterogeneous variances (by taking different σi for different
fi,θ but involving common unknown parameters) which is again a part of our I-NH set-up.
In terms of the general notation, the random variable X n = (X1 , . . . , Xn ) is defined on
the measurable space (χn , Bn ) = ⊗ni=1 (χi , B i ). The true induced distribution is Gn (xn ) =
Qn
Qn
n
i=1 Gi (xi ) and its absolutely continuous density has the form g (xn ) =
i=1 gi (xi )
Qn
n
with respect to λ (dxn ) = i=1 λ(xi ). The model distribution and density are given by
Q
Q
F n (xn |θ) = ni=1 Fi,θ (xi ) and f n (xn |θ) = ni=1 fi,θ (xi ) so that Fn = ⊗ni=1 F i .
Now, under this I-NH set-up, we can define the R(α) -posterior by suitably extending the
(α)
definition of the α-likelihood function qn (xn |θ) from its IID version in (2) keeping in mind
the general requirement (4). Borrowing ideas from Ghosh and Basu (2013), who have developed the MDPDE for the I-NH set-up, and following the intuition behind the construction
of the α-likelihood (2) of Ghosh and Basu (2016), one possible extended definition for the
α-likelihood in the I-NH case can be given by
qn(α) (xn |θ)

=

n 
X
1
i=1

(α)

with qi,θ (y) =

1
α



α

α
fi,θ
(xi )

1
−
1+α

Z

1+α
fi,θ

n


−

n X (α)
=
qi,θ (xi ),
α

(9)

i=1


R 1+α
P
(0)
α (y) − 1 − 1
fi,θ
fi,θ . Note that, qn (xn |θ) = ni=1 (log(fi,θ (xi )) − 1),
1+α
7

satisfying the required condition in (4). So, assuming a suitable prior distribution for θ,
(α)
the R(α) -posterior for the I-NH observations is defined through (5) with qn (xn |θ) being
given by (9). Note that, the resulting posterior is again a product of independent but
(α)
non-homogeneous terms. Similarly, we can also define the deduced measures Qn (Sn |θ),
(α)
(α)
(α)
Mn (·, An ) and Mn (·), along with the important R(α) -marginal density function mn (xn ).
We discuss their properties in more detail in Section 5.

Remark 2.1 In the first introduction of the R(α) -posterior under IID set-up, it has been
noted that its only drawback is the lose of the probabilistic interpretation (Ghosh and Basu,
2016). Here also we have developed the general R(α) -posterior differently from the conditional probability approach of usual Bayesian inference and referred it as a pseudo-posterior.
But, in fact, it can also be interpreted as an ordinary Bayes posterior under a suitably
modified model and prior densities, which becomes more prominent under the general setup and definition of this section. To see this, let use define an α-modified model density
(α)

(α)

qen (xn |θ) =

exp(qn (xn |θ))
(α)
Qn (χn |θ)

(α)

and the α-modified prior density π
en (θ) =

(α)

Qn (χn |θ)πn (θ)
(α)

Mn (χn ,Θn )

. Both

are proper densities and satisfy the required measurability assumptions whenever the relevant
(α)
quantities are assumed to exist finitely. Further, π
en (θ) is a function of θ only (independent
of the data) and hence may be used as a prior density in Bayesian inference; but it depends
(0)
(0)
on α and the model. In particular, at α = 0, π
en (θ) = πn (θ) and qen (xn |θ) = f n (xn |θ) so
that they indeed represent a modification of the model and the prior, respectively, in order to
achieve robustness against data contamination. Now, for any measurable event An ∈ BΘn ,
the standard Bayes (conditional)
posterioroprobability of An with respect to the (α-modified)
n
(α)
(α)
model family Fn,α = qen (·|θ) : θ ∈ Θn and the (α-modified) prior density π
en (θ) is
(α)

R

given by

An

R

(α)

qen (xn |θ)e
πn (θ)dθ

(α)
(α)
πn (θ)dθ
en (xn |θ)e
Θn q

(α)

, which simplifies to πn (An |xn ), the proposed R(α) -posterior

probability as defined in (5).
In the following we briefly present the possible forms of the α-likelihood function for
some other statistical models, but their detailed investigations are kept for the future.
Example 2.3 [Time Series Data]:
Consider the true probability space (Ω, BΩ , P ) and an index set T . A measurable time
series Xt (ω) is a function defined on T × Ω, which is a random variable on (Ω, BΩ , P ) for
each t ∈ T . Given a time series {Xt (ω) : t ∈ T }, they are assumed to be associated with
an increasing sequence of sub σ-fields {Gt } and have absolute continuous densities g(Xt |Gt )
for t ∈ T . For a stationary time series, one might take Gt = Ft−1 , the σ-field generated by
{Xt−1 , Xt−2 , . . .}, for each t ∈ T .
In parametric inference, we model g(Xt |Gt ) by a parametric density fθ (Xt |Ft−1 ) and try
to infer about the unknown parameter θ from an observed sample xn = {xt : t ∈ T = {1, 2, . . . , n}}
of size n. For example, in a Poisson autoregressive model, we assume fθ (xt |Ft−1 ) to be a
Poisson density with mean λt and λt = hθ (λt−1 , Xt − 1) for all t ∈ T = Z and some known
function hθ involving the unknown parameter θ ∈ Θ ⊆ Rp . In the Bayesian paradigm,
8

we additionally assume a prior density π(θ) and update it to get inference based on the
posterior density of θ given the observed sample data.
We can develop the robust Bayesian inference for any such time series model through the
proposed R(α) -posterior density provided a suitable α-likelihood function can be defined.
Following the construction of the MDPDE in such time series models (Kim and Lee, 2011,
2013; Kang and Lee, 2014, among others), we define the α-likelihood function under the
time series models as
qn(α) (xn |θ)

=

n 
X
1
t=1

α

fθα (xt |Ft−1 )

1
−
1+α

Z

fθ1+α (x|Ft−1 )dx

n


−

n X (α)
=
qt,θ (xt ),(10)
α
t=1

R 1+α
P
(α)
(0)
1
with qt,θ (y) = α1 (fθα (y|Ft−1 ) − 1)− 1+α
fθ (x|Ft−1 )dx. Note that, qn (xn |θ) = ni=1 (log(fθ (xt |Ft−1 )) − 1),
which clearly satisfies the required condition (4). The robust R(α) -posterior inference about
θ can then be developed based on this α-likelihood function.

Example 2.4 [Markov Precess]:
Example 2.3 can be easily generalized to Markov processes with stationary transitions.
Consider the random variables X1 , . . . , Xn defined on the underlying true probability space
(Ω, BΩ , P ) having true transition probabilities g(Xk+1 |Xk ), k = 0, 1, 2, . . . , n, with X0 being the initial value of the process. We model it by a parametric family of stationary
probabilities fθ (Xk+1 |Xk ) depending on the unknown parameter θ ∈ Θ ⊆ Rp . Then, the
required α-likelihood function
given the observed sample xni= (x1 , . . . , xn ) can be defined
R 1+α
Pn h 1 α
(α)
1
fθ (x|xk )dx − αn . Clearly it satisfies the
as qn (xn |θ) = k=1 α fθ (xk+1 |xk ) − 1+α
required condition (4) and it is possible to perform robust R(α) -posterior inference about θ
under this set-up.

Example 2.5 [Diffusion Process]:
Consider the true probability space (Ω, BΩ , P ) and an index set T . A measurable random
variable Xt defined on T follows a diffusion process if it satisfies
dXt = a(Xt , µ)dt + b(Xt , σ)dWt ,

t ≥ 0,

(11)

with X0 = x0 and two known functions a and b, where {Wt : t ≥ 0} is a standard Wiener
process and the parameter of interest is θ = (µ, σ)T ∈ Θ, a convex compact subset of
Rp × R+ . This model has important practical applications in finance, where some inference
about θ is desired based on discretized observations Xtni , i = 1, . . . , n, from the diffusion
process (11). We generally assume tni = ihn with hn → 0 and nhn → ∞ as n → ∞.
Frequintist’s robust MDPDEs of θ based on such observations are developed for two of its
special cases, a(Xt , µ) = a(Xt ) and b(Xt , σ) = σ, respectively, by Song et al. (2007) and
Lee and Song (2013).
However, whenever we have some prior knowledge about θ, quantified through a prior
density π(θ), one would like to apply the Bayesian approach of inference. A robust Bayes
inference can be developed for such models by using the proposed R(α) -posterior with a
9

suitably defined α-likelihood function. For this purpose, we note that
p
Xtni = Xtni−1 + a(Xtni−1 , µ)hn + b(Xtni−1 , σ) hn Zn,i + ∆n,i ,

i = 1, . . . , n,

i


R tn h
−1/2
where we define Zn,i = hn
Wtni − Wtni−1 and ∆n,i = tni a(Xs , µ) − a(Xtni−1 , µ) ds
i−1
i
R tn h
+ tni b(Xs , σ) − b(Xtni−1 , σ) dWs . Clearly, Zn,i are IID standard normal variables for
i−1
i = 1, . . . , n. Therefore, whenever ∆n,i can be ignored in P -probability, for large enough
n , i = 1, . . . , n, behave as I-NH variables with densities
n, the random variables Xtni |Gi−1


n ) ≡ N X n + a(X n , µ)h , b(X n , σ)2 h
n
fi,θ (·|Gi−1
ti−1
ti−1
n
ti−1
n , where Gi−1 is the σ-field generated
by {Ws : s ≤ tni }. Then, the corresponding α-likelihood function based on the observed
data xn = (xtn1 , . . . , xtnn ) can be derived as in Example 2.3 using (10). It clearly satisfies
P
(α)
(α)
the general requirement (4) and has the simplified form, qn (xn |θ) = ni=1 qi,θ (xtni ), with

(α)
qi,θ (xtni )

=












1


2πb(xtn

i−1

,σ)2 hn

α/2

1
 e
α


2
α xtn −xtn −a(xtn ,µ)hn
i
i−1
i−1
−
2b(xtn ,σ)2 hn
i−1


−

1

(1+α)3/2 



− α1 , if α > 0,



2






α xtn −xtn −a(xtn ,µ)hn

1
i−1
i−1
i

n , σ)2 hn
−
log
2πb(x
− 1,
 −
t
2
2
2b(xtn ,σ) hn
i−1

if α = 0.

i−1

The robust R(α) -posterior and the corresponding estimates of θ can be easily obtained using
this α-likelihood function. The details are left for the reader.


3
3.1

Exponential Convergence Results under General Set-up
Consistency of the R(α) -Posterior Probabilities

Barron (1988) first demonstrated the useful exponential convergence results for the usual
Bayes posterior probabilities under suitable assumptions about “merging in probability”
(see Definition A.1 and A.2 in Appendix A) and existence of uniformly consistent tests.
These results have later been refined by several authors (see Ghosal et al., 1995, 2000;
Walker, 2004; Ghosal and van der Vaart, 2007; Walker et al., 2007; Martin and Hong,
2012, among others). In this paper, we follow the approach of Barron (1988) to show
the exponential consistency of our new robust R(α) -posterior probabilities under suitable
extended assumptions. In particular, for some sequences of measurable sets An , Bn , Cn ⊆
Θn and sequences of constants bn , cn , we consider the following assumptions.
(A1) An , Bn and Cn together complete Θn , i.e., An ∪ Bn ∪ Cn = Θn .
(α)

(A2) Bn satisfies π
en (Bn ) =

(α)

Mn (χn ,Bn )
(α)
Mn (χn ,Θn )

≤ bn .

(A3) Cn are such that there exists sets Sn ∈ Bn satisfying lim Gn (Sn ) = 0 and sup
n→∞
θ∈Cn
cn .
10

(α)

c |θ)
Qn (Sn
(α)
Qn (χn |θ)

≤

(A3)∗ Cn are such that there exists sets Sn ∈ Bn satisfying P (X n ∈ Sn i.o.) = 0 and
(α)

supθ∈Cn

c |θ)
Qn (Sn
(α)
Qn (χn |θ)

≤ cn , where i.o. denotes “infinitely often”.

Here we need either of Condition (A3) or Condition (A3)∗ which, respectively, help us
to prove the convergence results in probability or with probability one. Clearly, Condition
(A3)∗ is stronger and imply (A3), but (A3) is sufficient in most practices yielding a convergence in probability type result. Also, if Condition (A3) holds with cn = e−nr for some
r > 0, then it indeed ensures the existence of a uniformly exponentially

 consistent (UEC)
test for Gn against the family of α-modified probability distributions

(α)

Qn (·|θ)

(α)
Qn (χn |θ)

: θ ∈ Cn

(α)

corresponding to the α-modified model density qen (·|θ) defined in Remark 2.1. Further, at
α = 0, these conditions simplify to those used by Barron (1988) for proving the exponential
convergence of ordinary Bayes posterior probabilities. In particular, at α = 0, Condition
(A2) simplifies to πn (Bn ) ≤ bn , i.e., the sets Bn have negligible prior probabilities if bn → 0,
and (A3) assumes the existence of a UEC test against the model distributions at θ ∈ Cn .
The following theorem gives our main result on the exponential convergence of the
(α)
R -posterior probabilities. For brevity, the proof is presented in the Online Supplement.
Theorem 3.1 (Exponential Consistency of the R(α) -posterior probabilities)
(α)

(1) Suppose that the true distribution Gn and the R(α) -marginal distribution Mn (·)
merge in probability according to Definition A.1 and let An ∈ BΘn be any sequence
(α)
of sets. Then, lim supn→∞ P πn (Acn |X n ) < e−nr = 1, for some r > 0, if and
only if there exists constants r1 , r2 > 0 and sets Bn , Cn ∈ BΘn such that Assumptions
(A1)–(A3) are satisfied with bn = e−nr1 and cn = e−nr2 respectively.
(α)

(2) Suppose the true distribution Gn and the R(α) -marginal distribution Mn (·) merge
with probabilityone according to Definition
 A.2 and let An ∈ BΘn be any sequence of
(α)
−nr
c
i.o. = 0, for some r > 0, if and only if there
sets. Then, P πn (An |X n ) ≥ e
exists constants r1 , r2 > 0 and sets Bn , Cn ∈ BΘn such that Assumptions (A1), (A2)
and (A3)∗ are satisfied with bn = e−nr1 and cn = e−nr2 respectively.
Note that, for α = 0, Theorem 3.1 coincides with the classical exponential convergence
results of ordinary Bayes posterior probabilities as proved in Barron (1988). Our theorem
generalize it for the robust R(α) -posterior probabilities under suitable conditions. Hence,
the R(α) -posterior distribution, besides yielding robust results under data contamination,
is asymptotically optimal in exactly the same exponential rate as the ordinary posterior
for all α ≥ 0. We will further illustrate the required assumptions for the IID data and the
I-NH data in Sections 4 and 5 respectively. Some useful sufficient conditions for merging
of R(α) -marginal distribution with Gn are also developed under the general case and are
presented in Appendix A.

11

3.2

Consistency of the R(α) -Bayes Estimators

Let us now examine the asymptotic properties of the R(α) -Bayes estimators associated with
the new robust R(α) -posterior distribution (5) under the general set-up of Section 2. In the
decision-theoretic framework, we consider the problem of estimation of a functional φP :=
φ(P ) of the true probability P ; for example φP could be the probability density of P , or any
summary measure (like mean) of P . For the given parametric family F n (·|θ), let us denote
φθ := φF n (·|θ) . Then, our action space is Φ = {φQ : Q is a probability measure on (Ω, BΩ )};
b on Φ × Φ denoting the loss in estimating φ
consider a non-negative loss function Ln (φ, φ)
b The general R(α) -Bayes estimator φb = φ(·;
b x ) of φ is then defined as
by φ.
n
Z
φb = arg min
φ∈Φ

Ln (φθ , φ)πn(α) (dθ|xn ) ,

(12)

provided the minimum is attained; we need the loss function Ln (φθ , φ) to be BΘn measurable
for each φ ∈ Φ. In particular, the R(α) -Bayes estimator of φθ = θ is the mean of the R(α) posterior distribution for squared error loss provided it exists finitely, or a median of the
R(α) -posterior distribution for absolute error loss. Similarly, when the loss function Ln is
the Dirac delta function (0-1 loss), the corresponding R(α) -Bayes estimator of θ is the mode
of the R(α) -posterior distribution, if it is uniquely defined.
However, if the minimum in (12) is not attained, we may define the approximate R(α) Bayes estimator φb of φ through the relation
Z
Z
b (α) (dθ|x ) ≤ inf
Ln (φθ , φ)πn(α) (dθ|xn ) + δn , with lim δn = 0. (13)
Ln (φθ , φ)π
n
n
n→∞

φ∈Φ

An useful example is the approximate mode of the R(α) -posterior for discrete parameter
space, which is an approximate R(α) -Bayes estimator under 0-1 loss. Also, note that, if the
R(α) -Bayes estimator exists, it is also an approximate R(α) -Bayes estimator.
In the following, we assume the loss function Ln to be bounded and equivalent to a
pseudo-metric dn on Φ × Φ as defined below.
Definition 3.1 A loss function Ln on Φ × Φ is said to be bounded if there exists L̄ < ∞
such that Ln (φθ , φP ) ≤ L̄ for all n and all θ ∈ Θn , where φP is the value of the target
functional φ under the true probability distribution P .
Definition 3.2 A loss Ln on Φ × Φ is said to be equivalent to a pseudo-metric dn on Φ × Φ
if there exist two strictly increasing functions h1 and h2 on [0, ∞) that are continuous at 0
with h1 (0) = h2 (0) = 0 and satisfy Ln ≤ h1 (dn ) and dn ≤ h2 (Ln ) on Φ × Φ and for all n.
Note that, Definition 3.2 is equivalent to saying lim Ln (φn , φbn ) = 0 if and only if
n→∞

lim dn (φn , φbn ) = 0. As an example, the squared Hellinger loss is bounded and equivalent
n→∞
to the L1 -distance metric. Also, the absolute error (L1 ) loss is equivalent to itself and
bounded by twice the Hellinger loss.
12

Our next theorem states the asymptotic consistency of R(α) -Bayes and approximate
R(α) -Bayes estimators of φθ to the true value φP for such loss functions. The proof follows
along the lines of the proof of Lemma 12 in Barron (1988) and is hence omitted for brevity.
Theorem 3.2 (Consistency of R(α) -Bayes Estimators) Given any sample data xn , let
b x ) be an approximate R(α) -Bayes estimator (or the R(α) -Bayes estimator) of
φbn = φ(·;
n
φP with respect to a loss function Ln that is bounded as in Definition 3.1 and is equivalent to a pseudo-metric dn as in Definition 3.2. Also, for any  > 0, define
A,n =


(α)

+L̄πn

{θ : dn (φP , φθ ) ≤ }. Then, we have dn (φP , φbn ) ≤  + h2 

Ac −1

(α)
1−πn

h1

(),n

|xn

(Ac,n |xn )

. Conse-


quently, if lim πn(α) Ac,n |X n = 0 in probability or with probability one for all  > 0, then
n→∞

lim dn (φP , φbn ) = 0 in probability or with probability one, respectively.

n→∞

In simple language, Theorem 3.2 states that whenever the target φP is close enough
to the model value φθ in the pseudo-metric dn asymptotically under the R(α) -posterior
probability, the corresponding R(α) -Bayes estimator with respect to Ln is asymptotically

consistent for φP in dn . But, Theorem 3.1 yields lim πn(α) Ac,n |X n = 0 under appropriate
n→∞

conditions and hence the corresponding R(α) -Bayes estimators are consistent in a suitable
pseudo-metric dn . In particular, Theorem 3.2 applies to the R(α) -Bayes estimators with
respect to the squared Hellinger loss and the L1 -loss to deduce their L1 consistency.

4
4.1

Application (I): Independent Stationary Models
R(α) -Posterior convergence

Consider the set-up of the independent stationary model as in Example 2.1. Let us study the
conditions required for the exponential convergence of the R(α) -posterior for this particular
(α)
n
set-up. First, to verify the
 merging
 of G and Mn , we define the individual α-modified
(α)
(α)
density as qe(α) (·|θ) = exp qθ (·) /Q(α) (χ|θ) and the α-modified prior π
en as in Remark
2.1 with πn = π. Consider the Kullback-Leibler divergence measure between two absolutely
continuous densities f1 and f2 with respect to the common σ-finite measure λ defined as


Z
KLD(f1 , f2 ) =

f1 log

f1
f2


dµ.

(14)

Then we define the information denseness of the prior π under independent stationary

models with respect to Fα = qe(α) (·|θ) : θ ∈ Θ as follows.
Definition 4.1 The prior π under the independent stationary model is said to be information dense at G with respect to Fα if there exists a finite measure π
e such that
lim inf e
n→∞

(α)
πn
nr de

de
π

(θ) ≥ 1,

for all r > 0, θ ∈ Θ,
13

(15)

and π



θ : KLD(g, qe(α) (·|θ)) < 



> 0 for all  > 0.

Note that, in view of Remark A.1 and Theorem A.2 of Appendix A, Definition 4.1 implies
(α)
that Gn and Mn merge in probability for the independent stationary models. Then, our
main Theorem 3.1 may be restated as follows.
Proposition 4.1 Consider the set-up of independent stationary models and assume that the
prior π is independent of n and is information dense at g with respect to Fα as per Definition
(α)
4.1. Take any sequence of measurable parameter sets An ⊂ Θ. Then, πn (Acn |X n ) is
exponentially small with P -probability one, if and only if there exists constants r1 , r2 > 0
and sequences of sets Bn , Cn ∈ BΘ such that (1) An ∪ Bn ∪ Cn = Θ, (2) Bn satisfies (A2)
with bn = e−nr1 and (3) Cn satisfies (A3)∗ with cn = e−nr2 .
Further, Condition 3 in Proposition 4.1 indeed holds under the assumption
of the exis- 

(α)
n (·|θ)
tence of a UEC test for Gn against the family of (α-modified) distributions Q(α)
: θ ∈ Cn ,
Qn (χn |θ)
n (α)
o
(·|θ)
or equivalently under the existence of a UEC test for G against the family QQ(α) (χ|θ)
: θ ∈ Cn
(α)

by the form of Qn (·|θ) in IID models. We can further simplify this assumption by using a
necessary and sufficient condition for the existence of UEC from Barron (1989) which states
that, “for every  > 0 there exists a sequence of UEC tests for the hypothesized distribution
P versus the family of distributions {Q : dTn (P, Q) > /2} if and only if the sequence of partitions Tn has effective cardinality of order n with respect to P ”; here, for any measurable
P
partition T , dT denotes the T -variation norm given by dT (P, Q) = A∈T |P (A) − Q(A)|.
Using this, one can show that the R(α) -posterior asymptotically concentrate on the L1 model
neighborhood of the true density g. Let us define, for any density p and any partition T ,
R
1
the “theoretical histogram” density pT as pT (x) = λ(A)
A p(y)λ(dy), for x ∈ A ∈ T, whenT
ever λ(A) 6= 0, and p = 0 otherwise. We call a sequence of partitions Tn to be “rich” if
the corresponding sequence of densities g Tn converges to the true density g in L1 -distance.


Also, define BTn = θ : d1 fθ , qe(α)Tn (·|θ) >  for any  > 0 and sequence of partition Tn ,
where d1 denotes the L1 distance, and consider the following assumption.
(α)

Assumption (B): For any  > 0, π
en (BTn ) =

(α)

Mn (χn ,BTn )
(α)

Mn (χn ,Θ)

is exponentially small for

some rich sequence of partitions Tn with effective cardinality of order n.
Note that, Assumption (B) implies Condition
2 of Proposition
for BTn , or any
n
 4.1(α)
o smaller
Q (·|θ)
Tn
subset of it. So, applying it with Bn = θ : d1 (g, fθ ) ≥ , dTn G, Q(α) (χ|θ) < /2 ⊂ B/4

o
n

(α) (·|θ)
and the existence result of UEC tests with Cn = θ : dTn G, QQ(α) (χ|θ)
> /2 , Proposition
4.1 yields the asymptotic exponential concentration of the R(α) -posterior probability in the
L1 -neighborhood An = {θ : d1 (g, fθ ) < }.
Proposition 4.2 Consider the set-up of independent stationary models and assume that
the prior π is independent of n and information dense at g with respect to Fα as per
Definition 4.1. If Assumption (B) holds then, for every  > 0, the R(α) -posterior probability
(α)
πn ({θ : d1 (g, fθ ) ≥ } |X n ) is exponentially small with P -probability one.
14

However, if Assumption (B) does not hold, we can deduce a weaker conclusion in terms
of Tn -variance distance in place of the L1 distance. The idea goes back to Barron (1988) for
a similar result in case of the ordinary posterior; an extended version for the R(α) -posterior
is presented in the following proposition.
Proposition 4.3 Consider the set-up of independent stationary models and assume that
the prior π is independent of n and information dense at g with respect to Fα as per
Definition 4.1. Then, for any sequence
Tn witheffective
of order

 of partitions
 cardinality

(α)

n, the R(α) -posterior probability πn

(α)

θ : dTn G,

Qn (·|θ)

(α)
Qn (χn |θ)

≥

Xn

is exponentially

small with P -probability one.

4.2

The cases of Discrete priors: Maximum R(α) -posterior estimator

We can derive the exponential consistency of the R(α) -posterior Bayes estimators with
respect to the bounded loss functions from Theorem 3.2 along with Proposition 4.1–4.3
of the previous subsection. Here, we consider the particular cases of discrete priors and
the most intuitive Bayes estimator under this case, namely the maximum R(α) -posterior
estimator, in more detail.
Consider the set-up of independent stationary models as before, but now with a countable parameter space Θ. On this countable parameter space, we consider a sequence of
P
discrete priors πn (θ) which are sub-probability mass functions, i.e.,
θ πn (θ) ≤ 1. The
most common loss-function to consider under this set-up is the 0-1 loss function, for which
the resulting R(α) -Bayes estimator is the (global) mode of the R(α) -posterior density; we
call this estimator of θ as the “maximum R(α) -posterior estimator (MRPE)”. However,
bα , to be referred to
when this mode is not attained, we consider an approximate version θ
(α)
as an “approximate maximum R -posterior estimator (AMRPE)”, defined by the relation
bα ) > sup π
bα )e
en(α) (θ)e
qn(α) (xn |θ)e−nδn ,
π
en(α) (θ
qn(α) (xn |θ
θ

(α)

with lim δn = 0,
n→∞

(16)

(α)

where qen (·|θ) and π
en (θ) are the α-modified model and prior densities as defined in Remark 2.1. This definition follows from the fact that the R(α) -posterior density is proportional
(α)
(α)
to π
en (θ)e
qn (xn |θ). Note that, if the MRPE exists, then it is also an AMRPE. Assume
bα = θ
bα (x ), as a function of data x , is measurable.
that this estimator θ
n
n
To derive the properties of the AMRPE, we assume that the sequence of priors satisfies
lim inf enr π
en(α) (θ) ≥ 1,
n→∞

for all r > 0, θ ∈ Θ.

(17)

This Assumption (17) indeed signifies that the (α-modified) prior probabilities are not
exponentially small anywhere in Θ. Then, we have the following theorem.
Theorem 4.4 Consider the set-up of stationary independent models with fixed countable
parameter space Θn = Θ and discrete prior sequence πn satisfying Assumption (17). Sup15

pose πn is information dense at the true probability mass function g with respect to Fα as in
(α)
Definition 4.1 and πn (Acn |X n ) is exponentially small with probability one for a sequence
of measurable subsets An ⊆ Θ. Then any approximate maximum R(α) -posterior estimator
bα ∈ An for all sufficiently large n with probability one.
θ
The next theorem gives some sufficient condition for the exponential convergence of the
in the present case of countable Θ, simplified from our main Theorem 3.1.

R(α) -posterior

Theorem 4.5 Consider the set-up of stationary independent models with fixed countable
parameter space Θn = Θ and a discrete prior sequence πn satisfying Assumption (17). Then,

for any true density g which is an information limit of the (countable) family qe(α) (·|θ) : θ ∈ Θn ,
(α)
we have πn ({θ : d1 (g, fθ ) ≥ } |X n ) is exponentially small with probability one, for each
bα satisfies: lim d1 (g, fb ) = 0, with probability one.
 > 0. Therefore, any AMRPE θ
θα
n→∞

Remark 4.1 Theorem 4.5, in a special case α = 0, yields a stronger version of Theorem 15
of Barron (1988). Our result requires fewer assumptions than required by Barron’s result.

4.3

Robust R(α) -Bayes Predictive Density Estimators

We now present a further application of the general theory to develop a robust version
of Bayes predictive density estimators that are also exponentially consistent. Considering
the set-up of IID observation as in Subsection 4.1, we are interested in estimating the
true (predictive) density g based on observed data xn = (x1 , . . . , xn ). A robust version
of the ordinary Bayes predictive density estimator of g given xn can be defined based on
(α)
the R(α) -posterior probability πn (θ|xn ) as given by (1). Let us denote by G the set of
all probability densities on (Ω, BΩ , P ) that are absolutely continuous with respect to λ.
For a suitably chosen loss function L(·, ·) on G × G, we define the R(α) -Bayes predictive
R
(α)
density estimate of g as given by gbαL (z) = arg ming∈G Θ L(fθ , g)πn (θ|xn )dθ, whenever
the minimum exists. If the minimum does not exist, we may extend this to define the
approximate R(α) -predictive density estimator gbαA,L through the relation
Z
Z
A,L (α)
L(fθ , g)πn(α) (θ|xn )dθ + δn , with lim δn = 0. (18)
L(fθ , gbα )πn (θ|xn )dθ < inf
g∈G

Θ

n→∞

Θ

The most popular and useful loss function is the the squared error loss, under which the
corresponding predictive density estimator is the mean of the model density with respect
R
(α)
to the R(α) -posterior distribution and is given by gbαE (z) = Θ fθ (z)πn (θ|xn )dθ. We refer
to it as the “expected R(α) -posterior predictive density estimator” (ERPDE) of g. The
corresponding ERPDE of the underlying true probability P , given the data xn , is
PbαE (B; xn ) =

Z

Fθ (B)πn(α) (θ|xn )dθ,

for all B ∈ B,

(19)

Θ

where we use the same notation Fθ (·) for both the distribution function and the corresponding measure. The accuracy of this estimator may be measured by a sequence of loss
16

functions Ln (P, PbαE ) on Φ × Φ, where Φ is the set of probability measures on (χ, B). Cone with Φ
e being the set of sub-probability measures
sidering the extension of Ln on Φ × Φ,
on (χ, B), we assume Ln (P, aFθ ) to be BΘ measurable for each 0 < a ≤ 1 and each P ∈ Φ
along with the following conditions.
Assumption (L1): For each n ≥ 1, a ∈ [0, 1] and for all P, Q, Q1 , Q2 ∈ Φ, Ln (·, ·) satisfies
(i) Monotonicity: Q2 ≥ Q1 ⇒ Ln (P, Q2 ) ≤ Ln (P, Q1 ).
(ii) Convexity: Ln (P, aQ1 + (1 − a)Q2 ) ≤ aLn (P, Q1 ) + (1 − a)Ln (P, Q2 ).
(iii) Scaling: Ln (P, aQ) ≤ Ln (P, Q) + ρ(a), with ρ(a) independent of n and lim ρ(a) = 0.
a→1

Assumption (L1) is satisfied by several common loss functions, including the relative
entropy loss, the L1 -distance and the Hellinger distance loss. Then, we have the following
consistency result for the ERPDE defined in (19); see the Online Supplement for its proof.
Proposition 4.6 Suppose the loss functions Ln satisfy Assumption (L1). Then, for any
(α)
 > 0, we have Ln (P, PbαE ) ≤  + ρ(πn (An, |xn )), where An, = {θ : Ln (P, Fθ ) < } and
(α)
ρ(·) is as defined in item (iii) of Assumption (L1). Therefore, whenever πn (An, |xn ) → 1
exponentially for every  > 0, Ln (P, PbαE ) is exponentially small as n → 0, i.e., the ERPDE
is exponentially consistent in Ln .
Some other loss functions used in the Bayesian analyses are the L1 -distance loss and
the (squared) Hellinger loss function. The R(α) -Bayes estimator of g with respect to the
first one is a measurable version of median, say gbαA (z), of the R(α) -posterior distribution of
fθ (z); we refer to it as the “absolute R(α) -posterior predictive density estimator” (ARPDE)
of g given the data xn . On the other hand, the R(α) -predictive density estimator of g under
hR p
i2
(α)
the (squared) Hellinger loss function is given by gbαH (z) ∝ Θ fθ (z)πn (θ|xn )dθ , which
we refer as the “Hellinger R(α) -posterior predictive density estimator” (HRPDE) of g given
xn . Note that both the underlying loss functions are bounded and equivalent to the L1 metric. Hence, Theorem 3.2 can be applied to get the following consistency result for these
predictive density estimators; here the action space is Φ = G and φP = g, the true density
under the probability P .
(α)

Proposition 4.7 If πn ({θ : d1 (g, fθ ) ≤ } |xn ) → 1 exponentially for every  > 0, then
the ARPDE and HRPDE are exponentially consistent in L1 -distance.
Combining Propositions 4.6 and 4.7 with Proposition 4.2, we get the following corollary.
Corollary 4.8 Under the assumptions of Proposition 4.2, the robust predictive density estimators ERPDE, ARPDE and HRPDE are all exponentially consistent in L1 -distance.
However, when the parameter space Θ is discrete as in Subsection 4.2, the most natural
choice of loss function is the 0-1 loss function; the resulting predictive density estimator, to
17

be referred to as the “maximum R(α) -posterior predictive density estimate” (MRPDE) of g,
bα is the (global) mode of the R(α) -posterior density. We
is given by gbαM (z) = fθbα (z), where θ
may also consider the “approximate maximum R(α) -posterior predictive density estimator”
bα is an approximate maximum R(α) (AMRPDE) of g given by gbαA,M (z) = fθbα (z), where θ
posterior estimator defined in (16). The MRPDE, if it exists, is also an AMRPDE. Their
L1 consistency at the exponential rate has already been shown in Theorem 4.5.
Example 4.1 (Robust ERPDE of the Normal Mean with Known Variance): Let
us consider the normal model with unknown mean θ and known variance σ 2 , i.e., the model
density fθ ≡ N (θ, σ 2 ) with θ ∈ Θ = R. We have an observed sample xn = (x1 , . . . , xn )T
of size n from the true density g ≡ N (θ0 , σ 2 ). Further, we have a prior density π of θ; for
simplicity, let us first assume the prior is uniform over R (improper), i.e., π(θ) = 1 for all θ ∈
R. The standard Bayes posterior distribution of θ is then N (X̄, σ 2 /n) where X̄ is the sample
mean. It is well-known that X̄ is highly non-robust against data contamination/outliers
and hence any inference based on this Bayes posterior N (X̄, σ 2 /n) is also highly sensitive to
outlying data points. In particular, the posterior estimate of θ with respect to any symmetric
loss function is X̄ having zero breakdown point and unbounded influence function.
However, we can perform robust inference based on the R(α) -posterior density with

Pn − α(θ−x2i )2
(α)
1
2σ
√
.
α > 0 which, in this case, simplifies to πn (θ|xn ) ∝ exp
α
i=1 e
α( 2πσ )
Pn − α(θ−x2i )2
(α)
2σ
√1
This is because, we now have qn (xn |θ) =
− nζα , with ζα =
α
i=1 e
α( 2πσ )
√
−α
2πσ
(1 + α)−3/2 . Ghosh and Basu (2016) illustrated that the expectation of this
R(α) -posterior (ERPE) is a very good robust estimator of θ for moderately large α ≈ 0.5.
Here we illustrate the effectiveness of the predictive density estimator, the ERPDE,

E
2
gbα (z). Note that, at α = 0, the usual Bayes predictive density estimate is N X̄, n+1
σ
n
which is again a function of the non-robust sample mean X̄. We expect to overcome this
lack of robustness through the R(α) -posterior based predictive density estimator gbαE (z) with
α > 0. We empirically compute and compare the KLD between gbαE (z) and the true density
g which measures the information loss in our estimation.
Note that, there is no simplified expression for the ERPDE with α > 0 and so we need
to compute it numerically. We use importance sampling Monte Carlo for this purpose with
the proposal distribution as N (X̄, s2n ) and 20000 steps. We simulate 1000 samples from
the N (5, 1) distribution (i.e., θ0 = 5 and σ 2 = 1). For each given sample, we compute
the ERPDE empirically using the importance sampling MC scheme and obtain the KLD
between the ERPDE gbαE and the true N (5, 1) density for different α ≥ 0. The average
KLD values over 1000 replications are reported for different sample sizes n = 20, 50, 100
(see Table 1). Clearly, in this case of pure data, the estimated predictive density gbαE is
closest to the true density in the KLD sense at α = 0, the ordinary Bayes predictive density
estimator, as expected. However, as α increases, the increase in the corresponding average
KLD values is not quite significant at small positive α and this becomes even smaller for

18

larger sample sizes (where all KLD values are much closer to zero).
Table 1: Average KLD values between gbαE , with uniform prior, and the true density N (5, 1)
for different α, sample sizes n and contamination proportions .
n
20

50

100


0
0.05
0.1
0.2
0
0.05
0.1
0.2
0
0.05
0.1
0.2

0
0.0241
0.0330
0.0655
0.1915
0.0093
0.0173
0.0544
0.1818
0.0045
0.0156
0.0471
0.1808

0.2
0.0253
0.0298
0.0460
0.1230
0.0096
0.0138
0.0304
0.1054
0.0047
0.0091
0.0219
0.1020

0.3
0.0264
0.0300
0.0424
0.0988
0.0102
0.0132
0.0224
0.0737
0.0050
0.0078
0.0153
0.0666

α
0.5
0.0299
0.0327
0.0416
0.0775
0.0115
0.0138
0.0189
0.0389
0.0054
0.0070
0.0098
0.0268

0.7
0.0359
0.0398
0.0481
0.0832
0.0133
0.0150
0.0187
0.0293
0.0062
0.0074
0.0088
0.0151

0.8
0.0417
0.0473
0.0569
0.1006
0.0142
0.0159
0.0192
0.0287
0.0065
0.0077
0.0089
0.0134

1
0.075
0.0934
0.1171
0.1904
0.0173
0.0191
0.0229
0.0341
0.0076
0.0087
0.0099
0.0129

Next, to illustrate the advantages of the proposed ERPDE at α > 0 in terms of robustness, let us repeat the above simulation exercise by replacing 100% of each sample by
a moderate outlying value of x = 8. We have considered the contamination proportions
as  = 0.05, 0.10, 0.20 and the corresponding average KLD values are reported in Table 1.
Note that, in the presence of contamination, the usual Bayes predictive density estimator
gb0E goes away from the true density yielding larger KLD values. But, the ERPDEs gbαE
with α > 0 remain much closer to the true density in the KLD sense; their KL distances
decrease as α increases up to a suitable value and then increases slightly (due to increase
in the variance part) although always being significantly smaller than that for α = 0 (b
g0E ).
The minimum KL distance shows up at α ≈ 0.5 for moderate amount of contamination and
moderate sample sizes; this optimum α-value increases with increase in both contamination
proportion and sample size. This clearly illustrate the significant robustness advantages of
the proposed ERPDE with moderately large α > 0 over the existing estimate at α = 0, at
the price of only a slight loss in case of pure data.
Table 2: Average KLD values between gbαE , with conjugate N (5, 9) prior, and the true density
N (5, 1) for different α, sample sizes n and contamination proportions .
n
20

50

100


0
0.05
0.1
0.2
0
0.05
0.1
0.2
0
0.05
0.1
0.2

0
0.0239
0.0322
0.0614
0.1908
0.0093
0.0158
0.0502
0.1826
0.0045
0.0156
0.0470
0.1800

0.2
0.0249
0.0295
0.0419
0.1217
0.0095
0.0129
0.0267
0.1060
0.0047
0.0090
0.0219
0.1007

0.3
0.0259
0.0300
0.0382
0.0970
0.0101
0.0127
0.0210
0.0740
0.0049
0.0077
0.0153
0.0659

α
0.5
0.0289
0.0329
0.0373
0.0739
0.0113
0.0135
0.0165
0.0388
0.0054
0.0070
0.0097
0.0262

0.7
0.0341
0.0387
0.0427
0.0752
0.0130
0.0152
0.0165
0.0283
0.0061
0.0073
0.0087
0.0145

0.8
0.0386
0.0439
0.0490
0.0845
0.0139
0.0164
0.0173
0.0274
0.0065
0.0076
0.0088
0.0127

1
0.0592
0.0669
0.0784
0.1276
0.0167
0.0194
0.0205
0.0312
0.0074
0.0085
0.0097
0.0121

Similar significant improvement in the robustness of the ERPDE can also be observed
19

in the cases of proper subjective priors. As an illustration, suppose we have some vague
prior belief about the true parameter value θ0 , quantified through a N (θ0 , τ 2 ) prior density,
where τ measures the strength of our belief. Note that, it is the conjugate prior to the
normal model family and the resulting R(α) -posterior density with α > 0 is given by
#
n
2
X
α(θ−xi )2
1
(θ
−
θ
)
0
√
πn(α) (θ|xn ) ∝ exp
e− 2σ2 −
.
α
2τ 2
α 2πσ i=1
"

(20)

We repeat the previous simulation exercise, but now considering the conjugate normal prior
with τ = 3 (moderately strong prior belief) and the R(α) -posterior as in (20). The resulting
values of average KLD measures are reported in Table 2; the robustness of the ERPDE
with moderate α > 0 is again clearly observed as in the previous case.

5
5.1

Application (II): Independent Non-homogeneous Models
Convergences of R(α) -Posterior and R(α) -Bayes estimators

Consider the set-up of independent but non-homogeneous (I-NH) models as described in
Example 2.2 of Section 2. We simplify the exponential convergence results for the R(α) posterior probabilities under this I-NH set-up from the general results of Section 3. Note
P
(α)
(α)
that, in this case, qn (xn |θ) = ni=1 qi,θ (xi ) for any observed data xn = (x1 , . . . , xn ), and
Q
(α)
hence Qn (Sn |θ) = ni=1 Q(i,α) (S i |θ) for any Sn = S 1 × S 2 × · · · × S n ∈ Bn with S i ∈ B i
R
(α)
for all i and Q(i,α) (S i |θ) = S i exp(qi,θ (y))dy. Assume that Θn = Θ and πn = π are


Qn
(α)
Q
i=1 exp qi,θ (xi )
(α)
independent of n. Then, we have qen (xn |θ) =
= ni=1 qe(i,α) (xi |θ) with
(α)
Qn (χn |θ)



(α)
exp qi,θ (xi )

(α)

qe(i,α) (xi |θ) = Q(i,α) (χi |θ) , so that, in the notation of Appendix A, Dn (θ) =
We define the information denseness for the I-NH models as follows:

1
n

Pn

i=1 KLD


gi , qe(i,α) (·|θ) .

Definition 5.1 The prior π under the I-NH model is said to be information dense at Gn =
(G1 , . . . , Gn ) with respect to Fn,α = ⊗ni=1 Fαi , if there exists a finite measure π
e satisfying



1 Pn
(i,α)
(15) such that π
e θ : lim supn→∞ n i=1 KLD gi , qe
(·|θ) <  > 0, for all  > 0.
When fi,θ = fθ is independent of i, then the I-NH set-up coincides with the IID setup and the information denseness in Definition 5.1 coincides with that in Definition 4.1.
Further, Definition 5.1 implies the general Definition A.3 of Appendix A and hence The(α)
orem A.2 yields Gn and Mn merge in probability for the I-NH model. Then, our main
exponential convergence results for the I-NH set-up simplifies as follows.
Proposition 5.1 Consider the set-up of I-NH models with Θn = Θ and assume that the
prior π is independent of n and information dense at Gn with respect to Fn,α as per Definition 5.1. Then, for any sequence of measurable parameter sets An ⊂ Θ, the R(α) -posterior
(α)
probabilities πn (Acn |X n ) is exponentially small with P -probability one, if and only if there
20

exists sequences of measurable parameter sets Bn , Cn ⊂ Θ such
 that An ∪ Bn ∪Cn = Θ,
(α)
(α)
Mn (χn ,Bn )
Qn (·|θ)
≤ e−nr for r > 0 and a UEC test for Gn against
: θ ∈ Cn exists.
(α)
(α)
Qn (χn |θ)

Mn (χn ,Θn )

However, the existence of the required
UEC in Proposition
5.1 is equivalent to the
n (i,α)
o
Q
(·|θ)
existence of a UEC test for Gi against Q(i,α) (χi |θ) : θ ∈ Cn uniformly over i = 1, . . . , n.
eTn =
Following the discussions of Section 4.1, this holds if Assumption (B) is satisfied for B

P
θ : n1 ni=1 d1 (fi,θ , qe(i,α) (·|θ)Tn ) >  in place of BTn . This leads to following simplification.
Proposition 5.2 Consider the set-up of the I-NH models with Θn = Θ and assume that
the prior π is independent of n and information dense at Gn with respect to Fn,α as per
eTn in place of BTn for every  > 0, the R(α) Definition 5.1. If Assumption (B) holds for B


P
(α)
posterior probability πn
θ : n1 ni=1 d1 (gi , fi,θ ) ≥  |X n is also exponentially small with
P -probability one for each  > 0.
We can then derive the consistency of any R(α) -Bayes estimator with respect to suitable
bounded loss functions from Theorem 3.2 along with the simplified Proposition 5.2.

5.2

Robust Bayes Estimation under Fixed Design Regression Models

As noted in Example 2.2 of Section 2, the most common and useful example of the general INH set-up is the regression models with fixed design. Let us consider an important example
of the
 regression
 model (8) with n fixed k-variate design points t1 , . . . , tn and fi,θ (x) =
x−ψ(ti ,β)
1
for some univariate density f . The corresponding α-likelihood is given by
σf
σ

α
Pn (α)
Mf,α
(α)
(α)
xi −ψ(ti ,β)
1
1
q
(x
)
with
q
(x
)
=
qn (xn |(β, σ)) =
f
− (1+α)σ
α
α − α,
i
i
i=1 i,(β,σ)
ασ
σ
i,(β,σ)
R 1+α
where Mf,α = f
. Consider a prior density π(β, σ) for the parameters (β, σ) over the
parameter space Θ = Rk × (0, ∞) [p = k + 1]. Note that this prior can be chosen to be
the conjugate prior or any subjective or objective priors; a common objective prior is the
Jeffrey’s prior given by π(β, σ) = σ −1 . Then, the R(α) -posterior density of (β, σ) is given
by (5) which simplifies in this case as
Qn

i=1 exp

πn(α) ((β, σ)|xn ) = R R Q
n

i=1 exp

h

h

1
ασ α f

1
ασ α f





xi −ψ(ti ,β)
σ

xi −ψ(ti ,β)
σ

α

α

−

−

Mf,α
(1+α)σ α

Mf,α
(1+α)σ α

i

i

π(β, σ)
.

(21)

π(β, σ)dβdσ

If σ is known as in the Poisson or logistic regression models (or can be assumed to be known
with properly scaled variables), we consider a prior only on β given by, say, π(β) which
is either the objective uniform prior or the conjugate prior or some other proper prior. In
such cases, we can get the simplified form for the R(α) -posterior density of β as given by

α i
h
xi −ψ(ti ,β)
1
exp
f
π(β)
α
i=1
ασ
σ
h

α i
πn(α) (β|xn ) = R Q
.
xi −ψ(ti ,β)
n
1
exp
f
π(β)dβ
α
i=1
ασ
σ
Qn

21

(22)

We can easily obtain the R(α) -Bayes estimators of β and σ under any suitable loss function.
Example 5.1 (Normal Linear Regression Model):
Here we have ψ(ti , β) = tTi β and f is the standard normal density. For simplicity, let
us assume that the data are properly scaled so that σ can be assumed to be known and
equal to 1; the unknown σ case can be considered similarly having the same robustness
implications. When σ = 1, we can simplify the R(α) -posterior from (22) and compute the
expected R(α) -posterior estimator (ERPE) of β through an importance sampling MonteCarlo. Let us denote D = [t1 , . . . , tn ]T , the fixed-design matrix, and x = (x1 , . . . , xn )T ;
b = (D T D)−1 D T x, which is also the
then the ordinary least square estimate of β is β
ordinary Bayes estimator under the uniform prior and has the variance n−1 (D T D)−1 . In our
simulation, we use 20000 steps
 in the importance sampling Monte-Carlo with the proposal
T
−1
−1
b
density Nk β, n (D D)
and empirically compute the bias and MSE of the ERPR
over 1000 replications for two different priors, different contamination proportions C =0%
(pure data), 5%, 10%, 20% and different sample sizes n = 20, 50, 100. For each cases, we
simulate n observations t11 , . . . , t1n independently from N (5, 1) to fix the predictor values
ti = (1, t1i )T and n independent error values 1 , . . . , n from N (0, 1) (note σ = 1); then the
responses are generated through the linear regression structure xi = tTi β+i for i = 1, . . . , n,
where the true value of β is taken as β 0 = (5, 2)T . For contaminated samples, [nC ] error
values are contaminated by generating them from N (5, 1) instead of N (0, 1). As the first
choice of the prior π(β), we consider the non-informative uniform prior π(β) ≡ 1; the
corresponding results are shown in Figure 1. Secondly, we consider the conjugate normal
prior for β given by π(β) ≡ Nk (β 0 , τ 2 Ik ) which signifies that the prior belief about our true
parameter value is quantified by a symmetric structure with uncertainty quantified by τ .
The empirical biases and MSEs for the latter case are presented in Figure 2.
It is clearly observed from the figures that, under pure data, the bias and the MSE
are the least for the usual Bayes estimator of β at α = 0, but their increments are not
quite significant for the ERPEs with moderate α > 0. On the other hand, in presence of
contaminations, the usual Bayes estimator (at α = 0) has severely inflated bias and MSE
and becomes highly unstable. The proposed ERPEs with α > 0 are much more stable
under contamination in terms of both bias and MSE; the maximum stability is observed for
tuning parameters α ∈ [0.4, 0.6] yielding significantly improved robust Bayes estimators.

6

Concluding Remark

This paper presents a general Bayes pseudo-posterior under general parametric set-up that
produces pseudo-Bayes estimators which incorporate prior belief in the general spirit of
Bayesian philosophy but are also robust against data contamination. The exponential
consistency of the proposed pseudo-posterior probabilities and the corresponding estimators
are proved and illustrated for the cases of independent stationary and non-homogeneous
models; separate attention is given to the case of discrete priors with stationary models.
22

(a) n = 20

(b) n = 50

(c) n = 100

Figure 1: Empirical Bias and MSE of the ERPE of β in the linear regression model with
uniform prior. [Dotted line: C = 0%, Dash-Dotted line: C = 5%, Dashed line: C = 10%,
Solid line: C = 20%]
Further applications of the proposed pseudo-Bayes estimators are described in the context
of predictive density estimation and linear regression models. All the results of Barron
(1988) turn out to be special cases of our results when the tuning parameter α is set to 0.
On the whole, we trust that this paper opens up a new and interesting area of research
on robust hybrid inference that has the flexibility to incorporate prior belief and inherits
optimal properties from the Bayesian paradigm along with the frequentists’ robustness
against data contamination and hence could be very helpful in different complex practical
problems. In this sense, all Bayesian inference methodologies can be extended with this new
pseudo-posterior in future research. In particular, a detailed study of the examples discussed
in Section 2 should be an interesting future work for different applications. Extended
versions of the Bayes testing and model selection criteria based on this new pseudo-posterior
can also be developed to achieve greater robustness against data contamination. Also it
will be practically helpful to develop a data-driven rule for the selection of the appropriate
tuning parameter α. We hope to pursue some of these extensions in the future.
23

(a) n = 20

(b) n = 50

(c) n = 100

Figure 2: Empirical Bias and MSE of the ERPE of β in the linear regression model with
Conjugate normal prior. [Dotted line: C = 0%, Dash-Dotted line: C = 5%, Dashed line:
C = 10%, Solid line: C = 20%]

A

Conditions for Merging of R(α) -marginal Distribution

n of the random variable X are
Definition A.1 Two probability distributions Gn1 and
n
2
G

n
g (X )
said to merge in probability if for every  > 0, lim P g2n (X n ) > e−n = 1, where gin is the
n
1
n→∞
density function of Gni with respect to λn for i = 1, 2.

Definition A.2 Two probability distributions Gn1 and Gn2 of the random variable Xn are
g n (X )
said to merge with probability one if for every  > 0, P g2n (X n ) > e−n for all large n = 1.
1

n

Barron (1988) described several useful conditions under which two distributions merge in
probability (or, with probability one). In particular, an application of Markov’s inequality
g n (X )
yields that Definitions A.1 and A.2 are equivalent to the conditions lim n1 log g2n (X n ) = 0
n
1
n→∞
in probability or with probability one, respectively. See Barron (1988, Section 4) for more
results and discussions.
24

We have seen that a crucial condition needed for the exponential convergence of the
(α)
probabilities and R(α) -Bayes estimators is the merging of Gn and Mn in
probability. Now we present some sufficient conditions under which this merging holds.
Consider the general set-up of Section 2. We first derive a frequentist large-deviation
approximation to the joint R(α) -Bayes distribution (7) of θ and X n that merge in (Kullback(α)
Leibler) information; this in turn implies the merging of Gn and Mn in information and
(α)
hence in probability. Let us consider the α-modified model and prior densities qen (·|θ) and
(α)
π
en (θ) as defined in Remark 2.1. Extending Barron (1988), we define the required joint
∗(α)
∗(α)
frequentist distribution of θ and X as given by Ln (dθ, dxn ) = πn (dθ) Gn (dxn ), where
R(α) -Posterior

∗(α)

(α)
−nDn (θ) (α)

∗(α)

π
en (dθ)
,
the probability distribution πn
of θ on Θn is defined as πn (dθ) = e
cn


R
(α)
(α)
(α)
(α)
with Dn (θ) = n1 KLD g n (·), qen (·|θ) and cn = e−nDn (θ) π
en (dθ). Also assume the
following condition.
Assumption
(M1): For
n
o any , r > 0, there exists a positive integer N such that
(α)
(α)
θ : Dn (θ) < 
≥ e−nr , for all n ≥ N .
π
en
1
KLD
n→∞ n

Theorem A.1 Under Assumption (M1), we have lim



∗(α)

Ln

(α)Bayes

, Ln



= 0, i.e.,

(α)Bayes

the R(α) -Bayes joint distribution Lnh
merge in informationiwith the frequentist approx

∗(α)
(α)
(α)
∗(α)
1
=
imation Ln . Further, lim n EGn KLD πn (·), πn (·|X n ) = 0, and lim n1 KLD g n , mn
n→∞

n→∞

(α)

0. Also, the second one implies that the distributions Gn and Mn

merge in probability.

Our next result simplifies Assumption (M1) further in terms of a suitable extended
notion of the information
n denseness of priors
o πn with respect to the family of α-modified
(α)
model densities Fn,α = qen (·|θ) : θ ∈ Θn .
Definition A.3 Suppose Θn = Θ is independent of n and define D̄(α) (θ) = lim sup Dn(α) (θ),
n→∞

Gn

(α)

Qn (·|θ)

the relative entropy rate of the sequence of measures
and (α)
. Then, the prior seQn (χn |θ)
n
quence πn is said to be information dense at G with respect to Fn,α if there exists a finite


measure π
e satisfying (15) such that π
e θ : D̄(α) (θ) <  > 0, for all  > 0.
Under this definition of information denseness for the general set-up, we have the re(α)
quired result on the merging of Gn and Mn as presented in the next theorem.
Theorem A.2 If the prior is information dense with respect to Fn,α as in definition A.3,
(α)
then Gn and Mn merge in information and hence they also merge in probability.
Remark A.1 Under the independent stationary models considered in Section 4, Definition
A.3 coincides with the simplified Definition 4.1.
Remark A.2 Under the countable parameter space with the independent stationary model
as considered in Subsection 4.2, Assumption (17) on the prior sequence πn simplifies Definition A.3 further, so that we can take any strictly positive probability mass function on Θ as
25


the measure π
e and hence the condition in Definition A.3 reduces to inf D̄(α) (θ) : θ ∈ Θ =

0. Thus, for any density g which is an information limit of the (countable) family qe(α) (·|θ) : θ ∈ Θn ,
the prior sequence πn will be information dense at g with respect to Fn,α .

References
Agostinelli, C. and Greco, L. (2013) A weighted strategy to handle likelihood uncertainty
in Bayesian inference Comput Stat , 28(1), 319-239.
Andrade, J. A. A. and O’Hagan, A. (2006). Bayesian robustness modeling using regularly
varying distributions. Bayesian Anal, 1, 169–188
Andrade, J. A. A. and O’Hagan, A. (2011). Bayesian robustness modelling of location and
scale parameters. Scand J Stat, 38, 691–711.
Arnborg, S. and Sjodin, G. (2001). On the foundations of Bayesianism. In: Bayesian inference and maximum entropy methods in Science and Engineering (Gif-sur-Yvette, 2000),
AIP Conf. Proc., 568, Amer. Inst. Phys., Melville, NY, 61–71.
Atkinson, A. C., Corbellini, A., and Riani, M. (2017). Robust Bayesian regression with the
forward search: theory and data analysis. TEST, 1–18.
Barron, A. R. (1988). The exponential convergence of posterior probabilities with implications for Bayes estimators of density functions. Technical Report, University of Illinois.
Barron, A. R. (1989). Uniformly powerful goodness of fit tests. Ann. Stat. 17(1), 107–124.
Basu, A., Harris, I. R., Hjort, N. L., and Jones, M. C. (1998). Robust and efficient estimation
by minimising a density power divergence. Biometrika, 85, 549–559.
Basu, A., Shioya, H. and Park, C. (2011). Statistical Inference: The Minimum Distance
Approach. Chapman & Hall/CRC, Boca Raton, FL.
Berger, J. O. (1984). The robust Bayesian viewpoint. In: Robustness of Bayesian Analyses,
Stud. Bayesian Econometrics 4, North-Holland, Amsterdam, 63–144.
Berger, J. O. (1994). An overview of robust Bayesian analysis. TEST, 3, 5–124.
Berger, J. and Berliner, L. M. (1986). Robust Bayes and empirical Bayes analysis with
-contaminated priors. Ann. Statist., 14(2), 461–486.
Berk, R. H. (1966). Limiting behavior of posterior distributions when the model is incorrect.
Ann. Math. Stat., 37, 51–58. [Corrigendum 37, 745–746].
De Blasi, P., Walker, S. G. (2012). Bayesian asymptotics with misspecified models. Statistica
Sinica, 23, 169–187.
26

Box, G. (1976). Science and Statistics. J. Amer. Stat. Assoc., 71, 791–799.
Cabras, S., Castellanos, M. E. and Ruli, E. (2014) A Quasi likelihood approximation of
posterior distributions for likelihood-intractable complex models Metron, 72(2), 153–167.
Danesi, I. L., Piacenza, F., Ruli, E., and Ventura, L. (2016). Optimal B-robust posterior
distributions for operational risk. pre-print.
Desgagne, A. (2013). Full Robustness in Bayesian Modelling of a Scale Parameter. Bayesian
Anal., 8(1), 187–220
Delampady, M. and Dey, D. K. (1994) Bayesian robustness for multiparameter problems J
Stat. Plann. Inf., 375–382.
Dey, D. K. and Birmiwal, L. (1994). Robust Bayesian analysis using divergence measures.
Stat. Prob. Lett., 20, 287–294.
Dey, D.K., Ghosh, S.H. and Lou, K. (1996). On local sensitivity measures in Bayesian
analysis (with discussion). In: Bayesian Robustness, IMS Lecture Notes, 29.
Dupre, M. J. and Tipler, F. J. (2009). New axioms for rigorous Bayesian probability.
Bayesian Anal., 4, 599–606.
Efron, B. (2013). Bayes’ theorem in the 21st century. Science, 340, 1177–1178.
Gelfand, A. E. and Dey, D. K. (1991). On Bayesian robustness of contaminated classes of
priors. Statist. Decisions, 9, 63–80.
Gelman, A., Meng, X. L., and Stern, H. (1996). Posterior predictive assessment of model
fitness via realized discrepancies. Statistica sinica, 733-760.
Ghosal, S., Ghosh, J. K., and Samanta, T. (1995). On convergence of posterior distributions.
Ann. Stat., 23, 2145–2152.
Ghosal, S., Ghosh, J. K., and van der Vart, A. W. (2000). Convergence rates of posterior
distributions. Ann. Stat., 28, 500–531.
Ghosal, S., and van der Vaart, A. W. (2007). Convergence rates of posterior distributions
for non-IID observations. Ann. Stat., 35(1), 192–223.
Ghosh, A., and Basu, A. (2013). Robust estimation for independent non-homogeneous observations using density power divergence with applications to linear regression. Electron.
J. Stat., 7, 2420—2456.
Ghosh, A., and Basu, A. (2016). Robust Bayes Estimation using the Density Power Divergence. Ann. Inst. Stat. Math., 68(2), 413–437.

27

Ghosh, J. K., Delampady, M. and Samanta, T. (2006). An Introduction to Bayesian Analysis: Theory and Methods. Springer.
Greco, L., Racugno, W. and Ventura, L. (2008). Robust likelihood functions in bayesian
analysis. J. Stat. Plann. Inf., 138, 1258–1270
Gustafson, P. and Wasserman, L. (1995). Local sensitivity diagnostics for Bayesian inference. Ann. Stat., 23, 2153–2167.
Halpern, J. Y. (1999a). A counterexample to theorems of Cox and Fine. J. Artificial Intelligence Res., 10, 67–85.
Halpern, J. Y. (1999b). Coxs theorem revisited. Technical addendum to: “A counterexample
to theorems of Cox and Fine”. J. Artificial Intelligence Res., 11, 429–435.
Hampel, F. R. (1974). The influence curve and its role in robust estimation. J. Amer. Stat.
Assoc. 69, 383–393.
Hampel, F. R., Ronchetti, E., Rousseeuw, P. J., and Stahel, W. (1986). Robust Statistics:
The Approach Based on Influence Functions. John Wiley & Sons, New York.
Hooker, G., and Vidyashankar, A. N. (2014). Bayesian Model Robustness via Disparities.
TEST 23(3), 556–584.
Insua, D. R., and Ruggeri, F. (2000). Robust Bayesian Analysis. Springer.
Kang, J., and Lee, S. (2014). Minimum density power divergence estimator for Poisson
autoregressive models. Comput. Stat. Data Anal., 80, 44–56.
Kim, B., and Lee, S. (2011). Robust estimation for the covariance matrix of multi-variate
time series. J. Time Ser. Anal., 32(5), 469–481.
Kim, B., and Lee, S. (2013). Robust estimation for the covariance matrix of multivariate
time series based on normal mixtures. Comput. Stat. Data Anal., 57, 125–140.
Lee, S., and Song, J. (2013). Minimum density power divergence estimator for diffusion
processes. Ann Inst Stat Math, 65, 213–236.
Martin, R., and Hong, L. (2012). On convergence rates of Bayesian predictive densities and
posterior distributions. ArXiv preprint, arXiv:1210.0103 [].
Martin, J., Perez, C. J., and Muller, P. (2009). Bayesian robustness for decision making
problems: Applications in medical contexts. Int. J. Approx. Reas., 50(2), 315–323.
McCulloch, R. E. (1989). Local model influence. J. Amer. Stat. Assoc., 84, 473–478.
Millar, R. B., and Stewart, W. S. (2007). Assessment of Locally Influential Observations in
Bayesian Models. Bayesian Anal., 2(2), 365–384.
28

Nakagawa, T., and Hashimoto, S. (2017). Robust Bayesian inference based on quasiposterior under heavy contamination. Technical Report.
Owhadi, H. and Scovel, C. (2015). Brittleness of Bayesian inference and new Selberg formulas. Commun. Math. Sci., to appear.
Owhadi, H. and Scovel, C. (2014). Qualitative Robustness in Bayesian Inference. preprint,
arXiv:1411.3984.
Owhadi, H., Scovel, C. and Sullivan, T. J. (2015a). Brittleness of Bayesian inference under
finite information in a continuous world. Electron. J. Stat., 9, 1–79.
Owhadi, H., Scovel, C. and Sullivan, T. J. (2015b). On the Brittleness of Bayesian inference.
SIAM Rev., 57, 566–582.
Owhadi, H., Scovel, C., Sullivan, T. J., McKerns, M., and Ortiz, M. (2013). Optimal
uncertainty quantification. SIAM Rev., 55, 271–345.
Ritov, Y. A. (1985). Robust Bayes Decision Procedures - Gross Error in the Data Distribution Ann. Stat., 13(2) 626–637
Ritov, Y. A. (1987). Asymptotic results in robust quasi-bayesian estimation. J. Mult. Anal.,
23(2), 290-302.
Shalizi, C. R. (2009). Dynamics of Bayesian updating with dependent data and misspecified
models, Electron. J. Stat., 3, 1039–1074.
Shyamalkumar, N. D. (2000). Likelihood Robustness. In: Robust Bayesian Analysis.
Springer, NewYork.
Sivaganesan, S. (1993). Robust Bayesian diagnostic. J. Stat. Plann. Inf., 35, 171–188.
Song, J., Lee, S., Na, O., and Kim, H. (2007). Minimum density power divergence estimator
for diffusion parameter in discretely observed diffusion processes. Korean Comm. Stat.,
14(2), 267–280.
Walker, S. G. (2004). New approaches to Bayesian consistency. Ann. Stat., 32(5), 2028–2043.
Walker, S. G. (2013). Bayesian inference with misspecified models J. Stat. Plann. Inf., 143,
1621–1633.
Walker, S. G., Lijoi, A. and Prunster, I. (2007). On rates of convergence for posterior
distributions in infinite-dimensional models. Ann. Stat., 35, 738–746.
Wang, C., and Blei, D. M. (2016). A General Method for Robust Bayesian Modeling . ArXiv
Preprint, arXiv:1510.05078.
Wasserman, L. (1990). Prior envelopes based on belief functions. Ann. Statist., 18, 454–464.
29

Weiss, R. (1996). An approach to Bayesian sensitivity analysis. J. Royal Stat. Soc. B, 58,
739–750.
Weiss, R. E., and Cho, M. (1998). Bayesian marginal influence assessment. J. Stat. Plann.
Inf., 71, 163–177.

30

B
B.1

Online Supplement: Proofs of the Results
Proof of Theorem 3.1

We use an argument similar to that used by Barron (1988). Let us consider the following
two assumptions in addition to Assumptions (A1)–(A3) and (A3)∗ .
(α)

satisfy

(α)

satisfy

(A4) The true distribution Gn and the R(α) -marginal distribution Mn
(α)

lim P

n→∞

mn (X n )
≥ an
g n (X n )

!
= 1.

(A4)∗ The true distribution Gn and the R(α) -marginal distribution Mn
P

!
(α)
mn (X n )
< an i.o. = 0.
g n (X n )

Note that, if Conditions (A4) and (A4)∗ hold with an = e−n for every  > 0, they
(α)
indicate that the true distribution Gn and the R(α) -marginal distribution Mn merge in
probability or with probability one respectively.
Now, we start with two primary results on the convergence of the R(α) -posterior probabilities.
Lemma B.1 Suppose Assumptions (A1)–(A3) and (A4) hold with lim bn = lim cn = 0 such
that rn := (bn + cn )/an is finitely defined. Then, for all δ > 0, we have

rn 
≤ δ.
lim sup P πn(α) (Acn |X n ) >
δ
n→∞

(23)

Further, if additionally Assumptions (A3)∗ and (A4)∗ are satisfied, then for any summable
sequence δn > 0 we have


rn
(α)
c
P πn (An |X n ) >
i.o. = 0.
δn

(24)

Proof: Note that, with G∞ probability one, the R(α) -posterior probability can be reexpressed
(α)
(α)
mn (X n , Acn )/Mn (χn , Θn )g n (X n )
πn(α) (Acn |X n ) =
,
(25)
(α)
mn (X n )/g n (X n )
since g n (X n ) is non-zero for each n with G∞ probability one. Let us first consider the
numerator in (25) and define En to be the event that the numerator is greater than (bn +
cn )/δ. Note that, Gn (En ) ≤ Gn (En ∩ Snc ) + Gn (Sn ) for any sequence of measurable sets

31

Sn ∈ Bn . So, taking Sn to be the critical sets of Assumption (A3), we get
n

≤

G (En ∩

Snc )

δ
(bn + cn )

Z

(bn +

≤
≤
≤

Z

(α)
cn )Mn (χn , Θn )

δ

=

≤

Gn (dxn )
c
E∩Sn
(α)
mn (xn , Acn )
Gn (dxn )
(α)
n
Mn (χn , Θn )g (xn )

=

δ

=

=

c
Sn

Z

c
Sn

c
m(α)
n (xn , An )dxn

Z

Z

(α)

(bn + cn )Mn (χn , Θn ) Snc
Z
δ
(α)

(bn + cn )Mn (χn , Θn ) Acn
Z
δ

[by Markov’s inequality and definition of En ]

Acn

exp(qn(α) (xn |θ))πn (θ)dθdxn

c
Q(α)
n (Sn |θ)πn (θ)dθ
c
Q(α)
n (Sn |θ)πn (θ)dθ

[by Fubini Theorem]
Z

c
Q(α)
n (Sn |θ)πn (θ)dθ



+
(α)
Cn
(bn + cn )Mn (χn , Θn ) Bn
[for the sets Bn and Cn from Assumptions (A1)–(A3)]
"Z
#
Z
(α)
c |θ)
δ
Q
(S
n
n
Q(α)
Q(α)
n (χn |θ)πn (θ)dθ +
n (χn |θ)πn (θ)dθ
(α)
(α)
Cn Qn (χn |θ)
(bn + cn )Mn (χn , Θn ) Bn
"
#
(α)
c |θ)
Q
(S
δ
n
Mn(α) (χn , Bn ) + sup (α) n Mn(α) (χn , Cn )
(α)
θ∈Cn Qn (χn |θ)
(bn + cn )Mn (χn , Θn )
"
#
(α)
δ
Mn (χ , Cn )
bn + cn (α) n
[by Assumptions (A2) and (A3)]
(bn + cn )
Mn (χ , Θn )
n

≤ δ.
Hence, Gn (En ) ≤ δ + Gn (Sn ) and using Assumption (A3) we get lim sup Gn (En ) ≤ δ.
n→∞

Further, by Assumption (A4) the denominator in (25) is less than an has probability tending
to zero. Combining the numerator and denominator probabilities (using the bound by the
union of events related to numerator and denominator), we get the desired result (23).
To prove the second part (24), we proceed as before by noting that P (X n ∈ En i.o.) ≤
P (X n ∈ En ∩ Snc i.o.) + P (X n ∈ Sn i.o.). Then, defining En with any summable sequence
δn and proceeding as before, we get P (X n ∈ En ∩ Snc i.o.) = 0 by Borel-Cantelli Lemma.
Next, by Assumption (A3)∗ , we have P (X n ∈ Sn i.o.) = 0 and hence P (X n ∈ En i.o.) = 0.
Then, the desired result (24) follows by noting that the denominator in (25) is less than an
infinitely often with probability zero by Assumption (A4)∗ .

Lemma B.2 Suppose, for some sequence of constants rn , we have


lim P πn(α) (Acn |X n ) ≤ rn = 1.

n→∞

(26)

Then, for any sequences bn and cn satisfying bn cn ≥ rn , there exists parameter sets Bn , Cn ⊂
Θn such that Conditions (A1)–(A3) hold.

32

Moreover, if additionally we have


P πn(α) (Acn |X n ) > rn i.o. = 0,

(27)

then Conditions (A1), (A2) and (A3)∗ hold.
n
o
(α)
Proof: Let us define Sn = xn : πn (Acn |xn ) > rn so that lim Gn (Sn ) = 0 by Assumpn→∞
tion (26). Next, for any sequence cn , we construct the parameter sets
(
Cn =

)

(α)

θ:

Qn (Snc |θ)
(α)

Qn (χn |θ)

≤ cn

(
, Bn =

θ ∈ Acn :

)

(α)

Qn (Snc |θ)
(α)

Qn (χn |θ)

> cn

.

Then, Conditions (A2) and (A3) hold by constructions of Cn and Bn . Finally, to show
(α)
(α)
(α)
Condition (A2), note that mn (xn , Acn ) ≤ rn Mn (χn , Θn )mn (xn ) for all xn ∈ Snc by its
definition. Then,
(α)

Mn (χn , Bn )
(α)

=

Mn (χn , Θn )
≤

≤
≤
≤
≤
≤
≤

1
(α)

Mn (χn , Θn )
1

Z

Q(α)
n (χn |θ)πn (θ)dθ

Bn

Z

c
Q(α)
n (Sn |θ)πn (θ)dθ
(α)
Mn (χn , Θn )cn Acn
[by Definition of Bn and Markov’s inequality]
Z Z
1
exp(qn(α) (xn |θ))dxn πn (θ)dθ
(α)
c
Mn (χn , Θn )cn An Sn
Z
1
c
m(α)
[by Fubini Theorem]
n (xn , An )dxn
(α)
Mn (χn , Θn )cn Sn
Z
1
rn Mn(α) (χn , Θn )m(α)
n (xn )dxn [by the construction of Sn ]
(α)
Mn (χn , Θn )cn Sn
Z
rn
m(α) (xn )dxn
cn Sn n
R
R
rn
(α)
(α)
[ Sn mn (xn )dxn ≤ χ mn (xn )dxn = 1]
n
cn
bn
[for any sequence bn satisfying bn cn ≥ rn ]

For the second part of the Lemma , we use the same definitions of sets as above. Then,
by Assumption (27), we have P (X n ∈ Sn i.o.) = 0 and hence Condition (A3)∗ holds by the
construction of Cn . Other two conditions then hold similarly as before.

Proof of Theorem 3.1:
Theorem 3.1 now follows directly from the above two lemmas.
The sufficiency part of the theorem follows from Lemma B.1 by taking bn = e−nr1 ,
cn = e−nr2 , an = e−n and δn = e−n∆ (for Part 2) with , ∆ > 0 and  + ∆ < min{r1 , r2 }
Then, rn and rn0 = rn /δn tend to zero exponentially fast.
The Necessity part of the theorem follows from Lemma B.2 with rn = e−nr and then
33

letting bn = e−nr1 , cn = e−nr2 for any r1 , r2 > 0 with r1 + r2 ≤ r.

B.2



Proof of Theorem 4.4

bα , it is sufficient to show that
Note that, by the definition of θ
sup π
en(α) (θ)e
qn(α) (X n |θ) < sup π
en(α) (θ)e
qn(α) (X n |θ)e−nδn a.s.[G],

θ∈Acn

for all large n.

(28)

θ

Now, by the information denseness assumption, Remark A.1 and Theorem A.2 of Appendix
(α)
A imply that Gn and Mn merge in probability. Therefore, the exponential convergence of
(α)
πn (Acn |X n ) is equivalent to
X

−nr1
π
en(α) (θ)e
qn(α) (X n |θ) ≤ m(α)
< gn (X n )e−nr a.s.[G],
n (X n )e

for all large n, for some r1 , r > 0.

θ∈Acn

Let us now choose a θ ∗ ∈ Θ such that KLD(g, qe(α) (·|θ ∗ )) < r/4. Then, using SLLN along
with Assumption (17), we get
gn (X n ) < π
en(α) (θ ∗ )e
q (α) (X n |θ ∗ )enr/2 a.s.[G],

for all large n.

Therefore, for all large n, we have with a.s.[G],
X

sup π
en(α) (θ)e
qn(α) (X n |θ) ≤

θ∈Acn

π
en(α) (θ)e
qn(α) (X n |θ)

θ∈Acn

< gn (X n )e−nr
< π
en(α) (θ ∗ )e
q (α) (X n |θ ∗ )e−nr/2
< sup π
en(α) (θ)e
qn(α) (X n |θ)e−nδn .
θ

bα ∈ An a.s.[G], for all sufficiently large n.
This completes the proof that θ

B.3



Proof of Theorem 4.5
(α)

Using the equivalence of d1 and dH , it is enough to show that πn (Ac |X n ) is exponentially
small with probability one, with A = {θ : dH (g, fθ ) ≥ } for each fixed  > 0. Note that,
(α)
Gn and Mn merge in probability by applying Theorem A.2 and Remark A.2 of Appendix
A. So, we will use Theorem 3.1 by constructing suitable parameter sets Bn and Cn with
A ∪ Bn ∪ Cn = Θ.


Put Bn = θ : πn (θ) < e−n/4 and Cn = θ ∈ Acn : πn (θ) ≥ e−n/4 . Then, clearly
A ∪ Bn ∪ Cn = Θ. Further, for some τ ∈ (0, 1),
(α)

(α)

Mn (χn , Bn )
(α)

Mn (χn , Θn )

=

X

Qn (χn |θ)

θ∈Bn

Mn (χn , Θn )

(α)

πn (θ) ≤

34

e−
(α)

n(1−τ )
4

X

Mn (χn , Θn ) θ∈Bn

τ
Q(α)
n (χn |θ)πn (θ)

But, since the prior sequence πn satisfies Assumption (17), we get, for all sufficiently large
n, (assuming all the relevant quantities exists finitely)
X

Mn(α) (χn , Θn ) =

Q(α)
n (χn |θ)πn (θ)

θ∈Θ

≥ e−n(1−τ )/8

X

τ
Q(α)
n (χn |θ)πn (θ)

θ∈Θ
−n(1−τ )/8

≥ e

X

τ
Q(α)
n (χn |θ)πn (θ) ,

θ∈Bn

and hence

(α)

Mn (χn , Bn )
(α)

≤ e−n(1−τ )/8 .

Mn (χn , Θn )
Thus, the first two conditions of Theorem 3.1 hold. For the third condition related to Cn ,
P
note that θ∈Cn πn (θ) ≤ 1 and so the number of points in Cn is less than en/4 . Then,


(α)
exp(qn (·|θ))
: θ ∈ Cn having the critical
consider the likelihood ratio test for gn against
(α)
Qn (χn |θ)

sets

(
Sn =

)

(α)

xn : max

exp(qn (xn |θ))

θ∈Cn

(α)

Qn (χn |θ)

> gn (xn ) .

We will show that this Sn serves as the desired
the required condition (A3) )
on Cn .
( set in

1/2
(α)
exp(qn (xn |θ))
For note that, Sn = ∪θ∈Cn Sn,θ , where Sn,θ = xn :
> gn (xn )1/2 . But
(α)
Qn (χn |θ)

for each of these sets, we get from Markov inequality that,
n
1
Gn (Sn,θ ) ≤ 1 − dH (g, fθ ) < e−n/2 ,
2


and hence Gn (Sn ) < e−n/4 . Similarly, we can also show that
(α)

(α)

Qn (Snc |θ)
(α)
Qn (χn |θ)

≤

c |θ)
Qn (Sn,θ
(α)
Qn (χn |θ)

< e−n/2 ,

uniformly over θ ∈ Cn . Hence, all the required conditions of Theorem 3.1 hold and we get
the first part of the present theorem.
The second part follows from Theorem 4.4 and Remark A.2.


B.4

Proof of Proposition 4.6
(α)

Fix  > 0 and xn . Let a = πn (An, |xn ). If a = 0, the result is trivial. So, assume a > 0
(α)
(α)
and consider the distribution πn (dθ|xn , An, ) obtained from πn (An, |xn ) by conditioning

35

on θ ∈ An, . Then, we have


Ln P, PbαE



!

Z
≤ Ln

Fθ πn(α) (dθ|xn )

P,

[By Monotonicity]

An,

!

Z

(aFθ )πn(α) (dθ|xn , An, )

P,

= Ln

An,

Z

Ln (P, aFθ )πn(α) (dθ|xn )

≤

[By Convexity]

An,

Z

[Ln (P, Fθ ) + ρ(a)] πn(α) (dθ|xn )

≤

[By Scaling]

An,

≤  + ρ(a).


B.5

Proof of Theorem A.1

First note that, by help of Equation (6), we can rewrite the R(α) -Bayes joint distribution as
(α)

L(α)Bayes
(dθ, dxn ) = πn(α) (dθ|xn ) m(α)
n
n (dxn ) =

mn (dxn , dθ)
(α)
Mn (χn , Θn )

= qen(α) (dxn |θ)e
πn(α) (dθ), (29)

(α)

(α)

en × λn . On the other hand, the frequnetist
which has a density qen (xn |θ) with respect to π
(α)
∗(α)
approximation Ln has the density function e−nDn (θ) gn (xn ) /cn and hence we get
KLD



(α)Bayes
L∗(α)
n , Ln



(α)

"
= EL∗(α) log

e−nDn

n

(θ) g (X ) /c
n
n
n
(α)
qen (X n |θ)

"
= Eπ∗(α) EGn
n

−nDn(α) (θ)

+ log

"
= Eπ∗(α) −nDn(α) (θ) + EGn log
n

#

gn (X n )
(α)

qen (X n |θ)
gn (X n )
(α)

qen (X n |θ)
h
i
= Eπ∗(α) −nDn(α) (θ) + nDn(α) (θ) − log cn
n
Z

(α)
−nDn (θ) (α)
= − log cn = − log
e
π
en (dθ)

#
− log cn
#
− log cn

Therefore, for any  > 0, we get
Z



(α)
1
1
∗(α)
(α)Bayes
−nDn (θ) (α)
KLD Ln , Ln
= − log
e
π
en (dθ)
n
n
n
o

1
≤
− log π
en(α) θ : Dn(α) (θ) < 
2 n


≤
+ = ,
for all but finitely many n,
2 2

36

(30)

by applying Assumption (M1) with r = 2 . Since  > 0 is arbitrary, this completes the proof
of the first part of Theorem.
The second part of the theorem follows by the relation
i 1

 1
h



1
(α)Bayes
KLD L∗(α)
= EGn KLD πn∗(α) (·), πn(α) (·|X n ) + KLD g n , m(α)
.
n , Ln
n
n
n
n
To proof the the last part of theorem, note that the Kullback-Leibler divergence satisfies
the relation

 2
gn (X )
E log (α) n
≤ KLD g n , m(α)
+ .
n
e
mn (X )
n

Therefore, by the second part of theorem, we get lim E log
n→∞

(α)

and Mn

B.6

merge in probability by using Markov inequality.

gn (X n )
(α)

mn (X n )

= 0 and hence Gn


Proof of Theorem A.2

We will show that Assumption (M1) holds and then the present theorem will follow by
n
Theorem A.1. To show Assumption (M1), let us fix , r > 0 and define ρn (θ) = enr dπ
de
π (θ).
Then, using Fatou’s Lemma, we get
Z n
n
o
o
lim inf enr πn θ : Dn(α) (θ) < 
= lim inf enr I
θ : Dn(α) (θ) <  πn (dθ)
n→∞
n→∞
Z n
o
= lim inf I
θ : Dn(α) (θ) <  ρn (θ)e
π (dθ)
n→∞
Z
n
o
=
lim inf I
θ : Dn(α) (θ) <  ρn (θ)e
π (dθ)
n→∞
Z n
o
=
I
θ : D̄(α) (θ) <  π
e(dθ)
n
o
= π
e θ : Dn(α) (θ) <  ,
(31)
which is strictly positive by the information denseness with respect to Fn,α (Definition A.3).
This implies Assumption (M1) and we are done.


37

