A discussion of prior-based Bayesian information criterion (PBIC)

Professor Bayarri and coauthors’ paper (hereafter, PBIC) offers a stimulating and welcomed addition to the already extensive and yet still rapid expanding literature on model selection and related topics. In a 2013 review on model selection in linear mixed models by Müller, Scealy, and Welsh (2013), the authors classified main approaches in mixed model selection into three categories, the information criteria, the shrinkage methods and the fence methods. The current paper is not specifically regarding mixed model selection problems; however, as one shall see, there is a connection in various ways. The paper focuses on a special case of the information criteria, namely, the Bayesian information criterion (BIC) and its extensions. In this regard, two other references may be mentioned, in addition to those cited by the authors. One is the δ-BIC method of Broman and Speed (2002), in which a tuning constant, δ, is multiplied to the logarithm penalty to improve finitesample performance; the other is an extended BIC proposed by Chen and Chen (2008), which allows the number of covariates to increase with the sample size. The current paper has noted a number of problems with general use of BIC. Some similar notes were made regarding not just BIC but the information criteria in general by Jiang, Rao, Gu, and Nguyen (2008) in the context ofmixedmodel selection. Among the problems mentioned in both papers is the so-called effective sample size (ESS). The issue was naturally raised in Jiang et al. (2008) because the latter authors were concerned with correlated observations. Intuitively, when the data are correlated, the ESS is smaller than the total number of observations due to the ‘redundancy’ in the data that each data point does not bring as much new information as an independent data point. Take a look at an extreme case where n data points are so correlated that they are identical; obviously, in this case the ESS should be 1, rather than n. Another example, given in Jiang et al. (2008) (also see Jiang & Nguyen, 2015), is a linear mixedmodel, whichmay be viewed as a two-way extension of the group mean model discussed extensively in PBIC. In the linear mixed model, the observations, yij, satisfy yij = x′ ijβ + ui + vj + eij, i = 1, . ,m1, j = 1, . ,m2, where xij is a vector of known covariates, β is a vector of unknown regression coefficients (the fixed effects), ui, vj are random effects, and eij is an additional error. It is assumed that ui’s, vj’s and eij’s are independent such that ui ∼ N(0, σ 2 u ), vj ∼ N(0, σ 2 v ), eij ∼ N(0, σ 2 e ). It is well-known (e.g., Hartley & Rao, 1967; Harville, 1977; Miller, 1977) that, in this case, the ESS for estimating σ 2 u and σ 2 v is not the total sample size, n = m1m2, but m1 and m2, respectively. Now suppose that one wishes to select the fixed covariates, which are components of xij, under the assumed model structure using BIC. It is not clear what should be in place of n in the log(n) penalty (it does not make sense to let n = m1m2). Note that them1,m2 as the ESS for estimating σ 2 u , σ 2 v , respectively, can be interpreted intuitively – they are the numbers of appearance of the uis and vjs, respectively, in the model. In general, the ESS for correlated data is somewhere between 1 and n, the sample size (this is also noted in PBIC), but exact quantification of ESS is difficult. In PBIC, the authors consider independent, rather than dependent data; still, they show that the ESS issue arises, when it comes to estimating different parameters. More importantly, the authors are able to quantify the ESS, in a certain way. I wonder if the quantification has some general, intuitive explanation, as in the special examples discussed above. By the way, the notation ni , used to denote the ESS for estimating the ith group mean in the group mean model, might cause some confusion as being the eth power of ni; perhaps, ne,i is a better notation? Another problem, noted both in PBIC and in Jiang et al. (2008), is how to reasonably count the number of (free) parameters, or the degrees of freedom associated with the parameters. In this regard, Ye (1998) introduced the generalised degrees of freedom, which, in particular, is not necessarily an integer. This is similar to ESS, which can also be a non-integer. As noted, there is an extensive literature in model selection, even if one focuses attention on BIC extensions. Furthermore, even though most of these extensions are proven to be consistent, finite-sample