Li et al. Genetics Selection Evolution 2010, 42:32 http://www.gsejournal.org/content/42/1/32 Page 5 of 14 Table 1 Summary of the microsatellites and basic population genetic estimates for the microsatellites (Continued) HAUT24 BM1818 HAUT27 22 23 26 45733839 35634770 26396836 45733962 35635033 26396987 7.09 4.03 8.85 0.70 0.63 0.61 0.025 0.019 0.126 0.143 0.102 0.103 0.861 0.458 0.453 0.406 0.538 0.376 0.424 0.486 0.396 0.004** 0.144 0.083 0.027* 0.013* 0.003** BTA, Bos taurus autosome; AR, allelic richness; HE, expected heterozygosity, FIS, inbreeding coefficient, observed homozygosity, FOBS, and expected homozygosity, FEXP, NA, not available; the probabilities for the Ewens-Watterson test were calculated based on homozygosity (PH) or Fishers’s exact test (PE); *, the significance level of P < 0.05, **, the significance level of P < 0.01; the genomic positions for the loci are BLASTed against STS or primer sequence in ENSEMBL cow genome Btau4.0 http://www.ensembl.org/Bos_taurus/Info/Index updated until 11/02/2010 FST(i, j) = exp( i +  i +  i) 1 + exp( i +  i +  i) where ai, bj and gij are locus, population and locus-by- population parameters, respectively [12]. In this study, the interpretations of the potential outliers are based on the locus effect (ai). Outliers from our data set were identified on the basis of the distribution following [12]. Rather than a fixed FST as assumed in the above fre- quentist method of [9], this BAYESFST test uses more information from the raw data and does not assume the same FST for each population [5,12]. Tests to detect loci under selection for pairwise populations To test for additional evidence of selection, we used the combination of statistics lnRH, lnRV and lnRθ’ in the population pairwise comparisons. The principle behind these tests is that variability at a neutral microsatellite locus is given by θ = 4 Neμ, where Ne is the effective population size and μ is the mutation rate. A locus linked to a beneficial mutation will have a smaller effec- tive population size and consequently a reduction in variability below neutral expectations. The relative var- iance in variability, lnRθ, can be assessed instead by esti- mating the relative variance in repeat number, lnRV, or heterozygosity, lnRH, for loci between populations. The lnRV was calculated using the equation lnRV = ln (Vpop1/Vpop2) where Vpop1 and Vpop2 are the variance in repeat number for population 1 and population 2, respectively [10]. The lnRH test is based on the calcula- tion of the logarithm of the ratio of H for each locus for a pair of populations as follows ⎛ 1 ⎞ ⎜ ⎟ − 1 ln RH = ln ⎝⎜ 1 − H pop1 ⎠⎟ 2 2 ⎛ 1 ⎞ ⎜ ⎟ − 1 ⎝⎜ 1 − H pop2 ⎠⎟ where H denotes expected heterozygosity (see equa- tion 2 in [6]). In addition, we attempted to calculate ln Rθ by estimating θ directly using a coalescence-based Bayesian Markov chain Monte Carlo simulation approach employing the MSVAR program [47]. The tests have been shown to be relatively insensitive to mutation rate, deviation from the stepwise mutation model, demographic history of population and sample size [16]. As suggested by [48], to detect the most recent and strong selective sweeps, the combination of lnRH and lnRV statistics is as powerful as lnRV alone, but using both statistics together lowers the rate of false positives by a factor of 3 because the variance in repeat number and the heterozygosity of a population measure different aspects of the variation at a locus. Thus, com- binations of any two of the three tests were implemen- ted here and significance of lnRH, lnRV and lnRθ’ for each comparison was calculated according to standard methods [6,10,48]. These statistics are generally nor- mally distributed, and simulations have confirmed that outliers (e.g. more than 1.96/2.58 standard deviations from the mean for 95%/99% confidence intervals, respectively) are likely to be caused by selection [48]. The tests were implemented for every pairwise compari- son involving native populations from different trait categories (Eastern Finncattle, Western Finncattle and Northern Finncattle vs. Yakutian, Istoben, Kholmogory and Ukrianian Grey), i.e. 12 population pairs for the horn (polled/horned) trait. Tests to detect loci under selection within a population The coalescence simulation approach using the DetSel 1.0 program [49] was used to detect outlier loci within the Finnish native populations (Eastern Finncattle, Western Finncattle and Northern Finncattle). It has the advantage of being able to take into account a wide range of potential parameters simultaneously and giving results that are robust regarding the starting assump- tions. For each pair of populations (i, j), and for all loci, we calculated Fi and Fj (Fi and Fj are the population- specific divergence; for details see [7,49]) and generated the expected joint distribution of Fi and Fj by perform- ing 10,000 coalescent simulations. Thus, every locus fall- ing outside the resulting confidence envelope can be seen as potentially under selection. The following nui- sance parameters were used to generate distribu- tions with similar numbers of allelic stages as in the