Variation in recombination rate affects detection of FST outliers under neutrality

Genome scans have become a standard analysis in population genetics used to identify genomic regions involved in important evolutionary processes. With a reference genome and DNA re-sequencing data, the genome can be divided into analysis windows of a fixed physical size and the distribution of summary statistics across those windows can be examined. Outliers to the genome-wide distribution of a particular summary statistic may then be investigated for their roles in processes such as adaptation and speciation. A particularly important summary statistic used in genome scans is Wright’s FST . Wright (1) examined the behaviour of allele frequencies in metapopulations and referred to the correlation of frequencies among demes as FST . The evolutionary history of a population can be thought of as a series of genealogies describing the relationships between every individual at a certain point in the genome, and Slatkin (2) showed that for a neutrally evolving metapopulation FST can be expressed in terms of the coalescence times of those genealogies. In particular, FST = T̄ −T̄0 T̄ , where T̄0 is the average coalescence times for a pair of alleles drawn from the same deme and T̄ is the average coalescence time for a pair of alleles drawn from the metapopulation as a whole. FST varies across the genome; the mutations that happened to occur in the population and the individuals that happened to be sampled will give rise to variation in FST at different points in the genome. There are numerous methods to average across linked sites in an attempt to account for these sources of variation (reviewed in (3)). Furthermore, the underlying demographic process is also stochastic and will generate different histories across the genome generating heterogeneity in FST . In the context of genome scans, ˆ FST is calculated in analysis windows, and the genome-wide distribution is examined under the implicit assumption that all windows share the same sampling distribution. However, recombination breaks down associations across genetically linked sites allowing coalescence times to vary across the genome causing some windows to reflect more or fewer coalescent histories than others. Thus, the sampling distributions of summary statistics that capture aspects of coalescent history likely covary with the rate of recombination. Consider a 10,000bp analysis window in a neutrally evolving genomic region that experiences virtually no recombination (for example, the mitochondrial genome or non-pseudoautosomal regions of the Y-chromosome in humans). Every nucleotide within this window would share a single genealogy, and every polymorphism present would reflect the exact same coalescent history. If there were a large number of polymorphisms in the window, the combined ˆ FST across all of them may provide a very precise estimate of their shared history. If one had a large number of these analysis windows and examined the distribution of their averages, it should resemble the coalescent distribution of FST , which, for a population conforming to the island model, can be approximated using the χ2 distribution (4), which has a long upper tail. Now consider the alternate extreme, 10,000bp of freely recombining nucleotides. In this case, all sites would have a distinct evolutionary history, each representing a quasiindependent instantiation of the coalescent process. As a result, every polymorphism in the window would carry information on a different genealogy, and each of these may have a distinct FST . The observed ˆ FST across all polymorphisms in this set provides an estimate of the mean of the coalescent distribution of FST . If one had a large number of these sets of loci, the values of ˆ FST should be tightly distributed about the mean, and central limit theorem predicts that the distribution should be approximately Gaussian. Obviously 10,000 contiguous nucleotides do not experience free recombination, but eukaryotes do exhibit wide variation in recombination rates across their genomes. For example, humans (5), collared flycatchers (6) and Drosophila melanogaster (7) all exhibit recombination rate variation over at least an order of magnitude across substantial proportions of their genomes. If the sampling distribution of ˆ FST had a longer tail in regions of the genome with low recombination and was narrower in more highly recombining ones, heterogeneous landscapes of FST may arise under neutrality. Determining outliers from the genome-wide distribution of summary statistics may, therefore, result in false positives and negatives depending on