DOI: 10.5061/dryad.j9kd51cfq
title: Identifying the best approximating model in Bayesian phylogenetics: Bayes factors, cross-validation or wAIC?

description: There is still no consensus as to how to select models in Bayesian
 phylogenetics, and more generally in applied Bayesian statistics. Bayes
 factors are often presented as the method of choice, yet other approaches
 have been proposed, such as cross-validation or information criteria. Each
 of these paradigms raises specific computational challenges, but they also
 differ in their statistical meaning, being motivated by different
 objectives: either testing hypotheses or finding the best-approximating
 model. These alternative goals entail different compromises, and as a
 result, Bayes factors, cross-validation and information criteria may be
 valid for addressing different questions. Here, the question of Bayesian
 model selection is revisited, with a focus on the problem of finding the
 best-approximating model. Several model selection approaches were
 re-implemented, numerically assessed and compared: Bayes factors,
 cross-validation (CV), in its different forms (k-fold or leave-one-out),
 and the widely applicable information criterion (wAIC), which is
 asymptotically equivalent to leave-one-out cross validation (LOO-CV).
 Using a combination of analytical results and empirical and simulation
 analyses, it is shown that Bayes factors are unduly conservative. In
 contrast, cross-validation represents a more adequate formalism for
 selecting the model returning the best approximation of the
 data-generating process and the most accurate estimates of the parameters
 of interest. Among alternative CV schemes, LOO-CV and its asymptotic
 equivalent represented by the wAIC, stand out as the best choices,
 conceptually and computationally, given that both can be simultaneously
 computed based on standard MCMC runs under the posterior distribution.
description: ef2.tar.gz: a
 multiple sequence alignment of elongation factor 2 in 30 eukaryotic
 species (627 aligned positions), taken from Lartillot and Philippe, 2006.
 Syst Biol 55:195. 
 metazoa.tar.gz: a
 concatenation of genes (35371 aligned positions) across 35 metazoans + 2
 choanoflagellates and 12 fungi, originally published in Philippe et al,
 2005. Mol Biol Evol 22:1246 
 table1_simu_replicates.tar.gz:
 simulation replicates produced for table 1 (normal
 model). 
 table2_cv_replicates.tar.gz: 
 cross-validation replicates of the elongation factor data set
 used for table 2. 
 figure2_jackknife_replicates.tar.gz: 
 jackknife replicates of the metazoa dataset used for figure
 2. 
 figure3_jackknife_replicates.tar.gz: 
 simulation replicates obtained by  posterior predictive
 simulations used for figure 3. 
 figure4_jackknife_replicates.tar.gz:
 jackknife replicates of the metazoa dataset used for
 figure 4. 
 analyses.tar.gz: scripts for
 reproducing the analyses of table 2, figures 2-4. 
 normcv.tar.gz: source code and
 scripts for running the multivariate normal model of table 1 and figure 1.
 also available at https://github.com/bayesiancook/normcv.git.
 pbmpi.tar.gz: 
 source code of phylobayes mpi, version 1.9, used for the
 phylogenetic analyses presented in the article. also available at
 https://github.com/bayesiancook/pbmpi.git. 
   The alignments are standard amino-acid
 multiple-sequence alignments. The script for
 reproducing the analyses requires the following programs:
 - phylobayes mpi, a Bayesian phylogenetic analysis program,
 version 1.9: https://github.com/bayesiancook/pbmpi.git
 - norm cv: a program for simulating and computing model fit under
 a special multivariate model described in the
 article: https://github.com/bayesiancook/normcv.git