Correcting Approximate Bayesian Computation

In their review of approximate Bayesian computation (ABC), Csillery et al. [pg. 411, 1] stated that my [2] “main” objections to ABC are that inference is limited to a finite set of models, and that these models are generally complex, although they failed to state the reasons for my objections. Csillery et al. further state that my criticisms were “general criticisms of model-based approaches and are not specific to ABC” [pg. 411, 1]. However my main objection to ABC was that it can produce posterior “probabilities” that are not true probabilities. The source for Csillery et al. ‘s claim that my objections were not specific to ABC is Beaumont et al. [3], who did acknowledge my main objection but without addressing the underlying reasons. Instead, Beaumont et al. [pg. 438, 3] state “… Templeton is in effect claiming that standard Bayesian inferences are invalid, and that Bayesian posterior probabilities are mathematically incapable of being probabilities.” The words “in effect” indicate that I actually never made this statement, and indeed I do not believe it. Contrary to the statement in Csillery et al. [1], my objections were very specific to ABC. By misrepresenting my views, both Csillery et al. [1] and Beaumont et al. [3] avoid addressing my specific criticisms and instead mount a general, but irrelevant, defense of Bayesian statistics and model-based inference. Rather than reiterate these published objections [2, 4], I will give a specific numerical example to illustrate my main objection. Fagundes et al. [5], a paper repeatedly cited as an exemplar of how to apply ABC to phylogeographic inference, assigned a posterior “probability” of 0.781 to the out-of-Africa replacement model of human evolution, and a posterior “probability” of 0.001 to the assimilation model that allows admixture between the human population expanding out of Africa with pre-existing Eurasian populations. This degree of admixture is measured by the parameter M, which can vary from 0 to 1 inclusively. The out-of-Africa replacement hypothesis is a special case of the assimilation model in which M=0 [5]. One elementary property of probability measures is that the probability of a general model must be greater than or equal to the probability of a special-case model nested within in it [4]. Obviously, the “probability” 0.001 is not greater than or equal to the “probability” 0.781. These numbers patently show that: 1) ABC can and does produce results that are mathematically impossible; 2) the “posterior probabilities” of ABC cannot possibly be true probability measures; and 3) ABC is statistically incoherent (incoherent methods can violate the constraints of formal logic). There are multiple contributors to the incoherence of ABC [4], but of fundamental importance is that the equation used in ABC to assign posterior probabilities to models is mathematically incorrect in every case in which one or more pairs of the models are logically overlapping. Even when the ABC posterior “probabilities” are not overtly incoherent, they are still mathematical errors in such cases. Given that some logical overlap is common when dealing with complex models, this means that much of the literature using ABC is invalid. Although I have been misrepresented as being anti-Bayesian, I am not even anti-ABC. I therefore offered a coherent correction to ABC [4]: only the general model is simulated to obtain the posterior distribution of the parameter that determines the special case followed by the construction of the 95% credible interval for this parameter. If the special case parameter value falls outside this interval, the special case model is rejected at the 5% level of significance; otherwise it is accepted. When this coherent ABC test was applied to the data of Fagundes et al. [5], the replacement hypothesis was rejected relative to the general assimilation model [4] –exactly the opposite conclusion that was reached using the incoherent ABC test. This demonstrates that coherence vs. incoherence is not some trivial statistical property, but can have a profound impact on the biological conclusions reached using ABC. Note that my suggested coherent use of ABC is one of falsification of a null hypothesis. Beaumont et al. [3] argue that statistics that assign probabilities of truth to a finite, non-exhaustive set of models are to be preferred over statistics that attempt to falsify null hypotheses. Beaumont et al. [3] also emphasize the importance of minimizing false positives for other statistical techniques, but turn a blind eye to the fact that testing among a finite set of non-exhaustive alternatives has an unknowable and uncorrectable false positive error rate. This was the reason for my objection concerning limiting inference to the relative truth of a finite set. Moreover, given that falsification of hypotheses is often used as the defining attribute that distinguishes science from other methods of knowledge, restricting ABC to assigning probabilities of truth to a finite set [3] is inexplicable. As the example of the coherent test given above illustrates, this restriction is also unnecessary. ABC can indeed be used to falsify null hypotheses in a coherent fashion, and ABC will be greatly strengthened when this unnecessary restriction is dropped. One of my main reasons for objecting to ABC in terms of model complexity was that adjustments for model dimensionality are needed. I [2] had suggested the Bayesian Information Criterion (BIC). Csillery et al. [1] agree with me, but suggested instead the related measures of AIC or DIC. Any of these three measures would be appropriate and all require the explicit calculation of dimensionality. In contrast, Beaumont et al. [3] vigorously deny the need for measures like BIC or any model dimensionality correction. ABC is a potentially powerful tool for statistical analysis of complex models in evolutionary biology and ecology. The numerical example given in this letter clearly vindicates my recent criticisms of ABC [2, 4, 6], and demonstrates the irrelevance of the counter-arguments [1, 3]. Hence, the potential of ABC is currently not realized because of serious statistical and mathematical flaws. As shown in this letter, I have suggested specific corrections for some of these flaws, and elsewhere I will give corrections for several additional serious flaws. Once fully corrected, the potential of ABC can be realized.