abstract: Demographic models built from genetic data play important roles in illuminating prehistorical events and serving as null models in genome scans for selection. We introduce an inference method based on the joint frequency spectrum of genetic variants within and between populations. For candidate models we numerically compute the expected spectrum using a diffusion approximation to the one-locus two-allele Wright-Fisher process, involving up to three simultaneous populations. Our approach is a composite likelihood scheme, since linkage between neutral loci alters the variance but not the expectation of the frequency spectrum. We thus use bootstraps incorporating linkage to estimate uncertainties for parameters and significance values for hypothesis tests. Our method can also incorporate selection on single sites, predicting the joint distribution of selected alleles among populations experiencing a bevy of evolutionary forces, including expansions, contractions, migrations, and admixture.
  .
  We model human expansion out of Africa and the settlement of the New World, using 5 Mb of noncoding DNA resequenced in 68 individuals from 4 populations (YRI, CHB, CEU, and MXL) by the Environmental Genome Project. We infer divergence between West African and Eurasian populations 140 thousand years ago (95% confidence interval: 40 – 270 kya). This is earlier than other genetic studies, in part because we incorporate migration. We estimate the European (CEU) and East Asian (CHB) divergence time to be 23 kya (95% c.i.: 17 – 43 kya), long after archeological evidence places modern humans in Europe. Finally, we estimate divergence between East Asians (CHB) and Mexican-Americans (MXL) of 22 kya (95% c.i.: 16.3 – 26.9 kya), and our analysis yields no evidence for subsequent migration. Furthermore, combining our demographic model with a previously estimated distribution of selective effects among newly arising amino acid mutations accurately predicts the frequency spectrum of nonsynonymous variants across three continental populations (YRI, CHB, CEU).
author: Ryan N. Gutenkunst; Ryan D. Hernandez; Scott H. Williamson; Carlos D. Bustamante
bibliography: popgen.bib
title: Inferring the joint demographic history of multiple populations from multidimensional SNP frequency data

# Introduction

Demographic models inferred from genetic data play several important roles in population genetics. First, they complement archeological evidence in understanding prehistorical events (such as the number and timing of major continental migrations) which have left no written record . Second, they facilitate the search for genetic regions that have been targets of non-neutral forces, such as recent natural selection, by guiding our expectations as to how much sequence and haplotype variation one expects to see in a given genomic region (and, more importantly, the variance around these expectations) . Finally, existing demographic models can guide sampling design for subsequent population or medical genetic studies.

Given their many uses, it is not surprising that many studies have inferred demographic models for populations of humans and other species .

The process of inferring a demographic model consistent with a particular data set typically involves exploring a large parameter space by simulating the model many times, often using coalescent-theory based Monte Carlo approaches. For computational reasons, many of the demographic inference procedures developed thus far have focused on single population models or models with multiple populations but no subsequent migration after subpopulations split (i.e., , but also see ). Methods that do consider multiple populations with migration often assume independent non-recombining regions  and do not often scale to genomic size data sets. Approaches for jointly considering recombination and migration often use a restricted set of summary statistics  of the data, which limits their statistical power. Finally, complex demographic inferences that make use of many summary statistics are often very computationally intensive , which precludes thorough investigation of their statistical properties.

Here, we develop and apply a computationally efficient diffusion-based approach to the problem of demographic inference, based on the multi-population allele frequency spectrum (AFS) (i.e., the joint distribution of allele frequencies across SNPs) . Given a genetic region sequenced in multiple individuals from each of $P$ populations, the resulting AFS is a $P$-dimensional matrix. Each entry of this matrix records the number of diallelic genetic polymorphisms in which the derived allele was found in the corresponding number of samples from each population. For example, if diploid individuals from two populations were sequenced, with 10 individuals from population 1 and 5 from population 2, the AFS would be a 21-by-11 matrix (indexed from 0). The \[2,0\] entry would record the number of polymorphisms for which the derived allele was seen twice in population 1 but never seen in population 2, while the \[20,5\] entry would record polymorphisms for which the derived allele was homozygous in all individuals from population 1 and seen 5 times in population 2. If all polymorphic sites possess only two alleles and can be considered independent, the AFS is a complete summary of the data. Many of the statistics commonly used for population genetic inference, such as $F_{ST}$ and Tajima's $D$, are summaries of the AFS (see ).

Efficient techniques exist for simulating the AFS of a single population . The joint AFS between two populations has been used by several recent studies , but these have all relied upon very computationally intensive coalescent simulations. Here we approximate the joint multi-population AFS by numerical solution of a diffusion equation, and our implementation supports up to three simultaneous populations. Because the diffusion approach neglects linkage, our comparison with the data is through a composite likelihood function. Such likelihoods are consistent estimators under a wide range of population genetic scenarios for selectively-neutral data, but do not correctly capture variances . (Lower recombination induces higher linkage and higher variance among the expected entries of the AFS.) As we demonstrate below, the efficiency of our diffusion approach enables both conventional and parametric bootstrap resampling of the data, allowing us to accurately estimate confidence intervals for parameter values and critical values for hypothesis tests , accounting for any degree of linkage found in the data. This bootstrap procedure overcomes the traditional concerns with composite likelihood as a philosophy for inference in population genetics

To demonstrate the utility of our approach, we apply our method to two epochs in human history, using single nucleotide polymorphism (SNP) data from the Environmental Genome Project (EGP) , the largest public database of human resequencing data. We first study the expansion of humans out of Africa, jointly modeling the history of African, European, and East Asian populations. We then study the settlement of the New World, jointly modeling European, East Asian, and admixed Mexican populations. In both cases, we quantify the uncertainty of our parameter inferences and test hypotheses about migration (bootstrapping to account for linkage). In particular, we infer an earlier divergence between African and Eurasian populations than previous studies, because our inferences account for the substantial migration between these populations. Our methods also find no evidence for multiple migrations between East Asia and the New World. While similarly complex models for human continental populations have been studied , to our knowledge, our analysis is the first in which the full joint AFS is used for inference and in which uncertainty and goodness-of-fit have been quantified.

An important advantage of the diffusion approach is the ease with which selection can be incorporated. As an illustrative application, we also predict the distribution of protein-coding variation between populations. In agreement with the data, we find that less nonsynonymous variation is shared between populations than might be expected based only on patterns of shared noncoding variation.

While no model can capture the full complexity of any species' genetic history, the models presented refine our understanding of the expansion of humanity across the globe. None of the methodology is specific to humans, and we expect our method will find wide application to demographic inference of other species.

# Methods

## Diffusion approximation

To efficiently simulate the AFS, we adopt a diffusion approach. Such approaches have a long and distinguished history in population genetics, dating back to R. A. Fischer . The diffusion approach is a continuous approximation to the population genetics of a discrete number of individuals evolving in discrete generations. An important underlying assumption is that per-generation changes in allele frequency are small. Consequently, the diffusion approximation applies when the effective population size $N$ is large and migration rates and selection coefficients are of order $1/N$.

If we have samples from $P$ populations, the numbers of sampled sequences from each population are $n_1, n_2, \dots, n_P$. (For diploids, $n_1$ is twice the number of individuals sampled from population 1.) Entry $d_1, d_2, \dots, d_P$ of the AFS records the number of diallelic polymorphic sites at which the derived allele was found in $d_1$ samples from population 1, $d_2$ from population 2, and so forth. (If ancestral alleles cannot be determined, then the "folded" AFS can be considered, in which entries correspond to the frequency of the minor allele.)

We model the evolution of $\phi(x_1, x_2, \dots, x_P, t)$, the density of derived mutations at relative frequencies $x_1, x_2, \dots, x_P$ in populations $1, 2, \dots, P$ at time $t$. (All $x$ run from 0 to 1.) Given an infinitely-many-sites mutational model  and Wright-Fisher reproduction in each generation, the dynamics of $\phi$ for an arbitrary finite number of populations $P$ are governed by a linear diffusion equation: $$\begin{aligned}
\label{eqn:main}
\frac{\partial }{\partial \tau} \phi =
\frac{1}{2} \sum_{i = 1,2,\dots,P} \frac{\partial^2}{\partial^2 x_i} \, \frac{x_i (1-x_i)}{\nu_i} \, \phi
 -\sum_{i = 1,2,\dots,P} \frac{\partial}{\partial x_i} \bigg( \gamma_i x_i (1-x_i) + \sum_{j=1,2,\dots,P} \, M_{i \leftarrow j} (x_j - x_i)\bigg) \phi.
\end{aligned}$$ The first term models genetic drift, and the second term models selection and migration. Fig A illustrates the effects of different evolutionary forces on components of $\phi$. Time is in units of $\tau = t/(2 N_{ref})$, where $t$ is the time in generations and $N_{ref}$ is a reference effective population size. The relative effective size of population 1 is $\nu_1 = N_1/N_{ref}$. The scaled migration rate is $M_{1 \leftarrow 2} = 2 N_{ref} m_{1 \leftarrow 2}$, where $m_{1 \leftarrow 2}$ is the proportion of chromosomes per generation in population 1 that are new migrants from population 2. (Thus migration is assumed to be conservative ). Finally, the scaled selection coefficient is $\gamma_1 = 2 N_{ref} s_1$, where $s_1$ is the relative selective advantage or disadvantage of variants in population 1. Boundary conditions are no-flux except at two corners of the domain, where all population frequencies are 0 or 1; these are absorbing points corresponding to allele loss or fixation. Because the diffusion equation is linear, we can solve simultaneously for the evolution of all polymorphism by continually injecting $\phi$ density at low frequency in each population (at a rate proportional to the total mutation flux $\theta$), corresponding to novel mutations.

Changes in population size and migration alter the parameters in Eqn , while population splits and mergers alter the dimensionality of $\phi$. For example, if new population 3 is admixed with a proportion $f$ from population 1 and $1-f$ from population 2 then $$\label{eqn:split}
\phi(x_1, x_2, x_3, t) = \phi(x_1, x_2, t) \, \delta\big(x_3 - [f x_1 + (1-f) x_2]\big),$$ where $\delta$ denotes the Dirac delta function. To remove population 2, $\phi$ is integrated over $x_2$: $\phi(x_1, x_3, t) = \int_0^1 \phi(x_1, x_2, x_3, t) \,\textrm{d} x_2$.

Given $\phi$, the expected value of each entry of the AFS, $M[d_i, d_j, \dots,d_P]$, is found via a $P$-dimensional integral over all possible population allele frequencies of the probability of sampling $d_i, d_j\dots, d_P$ derived alleles times the density $\phi$ of sites with those population allele frequencies. For SNP data obtained by resequencing, these probabilities are binomial, so $$M[d_i, d_j, \dots, d_P] =
 \int_0^1 \cdots \int_0^1 \prod_{i = 1,2,\dots,P}  {n_i\choose d_i} x_i^{d_i} (1-x_i)^{n_i - d_i} \, \phi(x_1,x_2,\dots,x_P)\, \textrm{d}x_i.\label{eqn:to_M}$$ In some cases of ascertained data , the resulting bias can be corrected by modifying the above equation .

## Likelihood-based inference

Let $\Theta$ correspond to the parameters of a demographic model we wish to estimate from the observed multi-population allele frequency spectrum, which we denote $S[d_i, d_j, \dots]$. Assuming no linkage between polymorphisms, each entry in the AFS is an independent Poisson variable , with mean $M[d_i, d_j, \dots]$ (which depends on $\Theta$). We can, therefore, construct a likelihood function $\mathcal{L}(\Theta \mid S)$ using standard statistical theory: $$\mathcal{L}(\Theta \mid S) = \prod_{i = 0 \dots P} \prod_{d_i = 0\dots n_i} \frac{e^{-M[d_i, d_j,\dots,d_P]} M[d_i, d_j,\dots,d_P]^{S[d_i, d_j,\dots,d_P]}}{S[d_i, d_j,\dots,d_P]!}.\label{eqn:likelihood}$$

In words, our approach consists of calculating the expected allele frequency spectrum $M$ using a particular demographic model (and set of parameter values for that demographic model) using our diffusion approach. We then maximize the similarity between $M$ and the observed AFS, $S$, over the parameter values that $\Theta$ can take on. Competing demographic models can be chosen from using standard statistical theory such as the nested likelihood ratio test or Information Criteria such as the Akaike or Bayesian Information Criteria.

For linked polymorphisms, $\mathcal{L}$ is a composite likelihood. Such likelihoods are consistent estimators under a wide range of neutral population genetic scenarios , but simulations incorporating linkage are necessary to estimate variances and define critical values for hypothesis testing and model selection. In our applications, we estimate variances using simulations from the coalescent simulator *ms* .

## Numerics

Solving the multi-population diffusion equation is substantially more demanding than the single-population case . This is primarily because the boundary conditions are more complex, and the numerical grid of population frequencies $x$ must be much coarser to be computationally tractable, because it is of $P$ dimensions. For example, a previous single-population study  used a uniform $x$ grid of order $10^4$ values between 0 and 1. Extending this grid to a three-population simulation would require an infeasible array of size $10^{12}$. Instead, we use a nonuniform grid and extrapolation to enable accurate computation using of order 100 values along each dimension, for a final array size of order $10^6$.

We solve the diffusion equation on a regular nonuniform grid, using a finite difference scheme  inspired by the method of Chang and Cooper  (supporting information). Mutations in population 1 arise at frequency $1/(2 N_1) = 1/(2 N_{ref} \nu_1)$. The diffusion approximation applies when $N_{ref} \rightarrow \infty$, but the minimum frequency in our numerical simulation is that of the first grid point, denoted $\Delta$. To overcome this, we extrapolate our results to an infinitely fine grid. We use a quadratic extrapolation on the logarithm of the AFS entry, modeling the bias introduced by the finite initial grid point $\Delta$ as $$\log M_\text{calc}(\Delta) = \log M_\infty + a  \Delta + b \Delta^2.\label{eqn:extrap}$$ Here $M_\text{calc}(\Delta)$ is an AFS element calculated at grid size $\Delta$ and $M_\infty$ is the extrapolated value. Given three evaluations at different grid sizes $\Delta$, we solve for $M_\infty$ and use this value when calculating likelihoods. This vastly increases both the speed and accuracy of our calculation (supporting information). While higher-order extrapolations may improve accuracy in some cases, they may also be more sensitive to numerical noise. Our empirical experience is that a quadratic approximation provides a good compromise between accuracy, efficiency, and robustness.

The computational cost for a single likelihood evaluation scales as $G^{P+1}$ where $G$ is the number of grid points used. In our experience, for stability and accuracy $G$ should somewhat larger than the largest population sample size. Although our theoretical framework extends to an arbitrary number of populations, the exponential scaling of computation with $P$ limits our current applications to three simultaneous populations. Importantly, our likelihood calculation is deterministic and numerically smooth, so numerical derivatives can be used in optimization. We use the the quasi-Newton BFGS method , which converges in order $N_p^2$ steps, where $N_p$ is the number of free parameters.

Our implementation of these methods, $\partial$a$\partial$i, is written in cross-platform Python and C, making use of the NumPy , Scipy , and Matplotlib libraries . It is distributed under the open-source BSD license. All calculations herein were performed with $\partial$a$\partial$i version 1.1.0.

We estimated parameter uncertainties by both conventional bootstrap (fitting data sets resampled over loci) and parametric bootstrap (fitting simulated data sets). To generate simulated data we used the coalescent program *ms* , a region-specific recombination rate, and the detailed EGP sequencing strategy (supporting information).

The confidence intervals reported in Tables  and  derive from a normal approximation to the bootstrap results. For the conventional bootstrap, confidence intervals were calculated as: $\overline{\theta^*}  \pm 1.96  \, \sigma(\theta^*)$. For the parametric bootstrap, biased-corrected intervals were calculated as: $\hat\theta - (\overline{\theta^*} - \hat\theta) \pm 1.96  \, \sigma(\theta^*)$. The maximum-likelihood value is denoted $\hat \theta$, while $\overline{\theta^*}$ and $\sigma(\theta^*)$ denote the mean and standard deviation of the bootstrap results. Aside from the growth rates $r$, all our model parameters are positive by definition, so in those cases we used their logarithms when calculating confidence intervals.

Pearson's $\chi^2$ goodness-of-fit test was performed using all $21^3 - 2$ = 9259 bins in the AFS. Results are similar if we restrict our analysis to entries in which the expected value is $>1$ or $>5$.

## Data

We used the National Institute of Environmental Health Science's Environmental Genome Project SNPs database , which results from direct Sanger resequencing of environmental response genes in several populations. We considered all diallelic SNPs in 5.01 Mb of sequence from noncoding regions of 219 autosomal genes (supporting information). These data have been the subject of many publications, including . As an assessment of quality, additional high-coverage short-read sequencing has recently been performed across 8 samples in this data set. Over 26,000 sites, the SNP concordance between this next-generation sequencing and the original Sanger sequencing averages 99.5% (D. Nickerson, personal communication). Given the high quality of this data set, we do not incorporate sequencing error into our modeling. We believe such correction will be essential in future applications to less accurate short-read sequencing data, as inference based on the frequency spectrum is sensitive to rare alleles.

To estimate the ancestral allele, we aligned to the panTro2 build of the chimp genome . Like other methods based on the unfolded AFS, our analysis is sensitive to errors in identifying the ancestral allele. We statistically corrected the AFS for ancestral misidentification , using a context-dependent substitution model . This procedure has been shown to perform better than aligning to multiple species . To account for missing data and ease qualitative comparisons between populations, we projected all spectra down to 20 samples per population  (supporting information).

The human-chimp divergence in the data is 1.13%. We assumed a divergence time of 6 My  and a generation time of 25 years. This yielded an estimated neutral mutation rate of $\mu = 2.35 \times 10^{-8}$ per site per generation, which is comparable to direct estimates . There is some controversy as to the appropriate generation time to assume in human population genetic studies . In particular, the human generation time may differ between cultures and may have changed during our biological and cultural evolution. The bootstrap uncertainties reported in Tables  and  do not include systematic uncertainties in the human-chimp divergence or generation times. The generation time, however, formally cancels when converting between genetic and chronological times.

## Nonsynonymous polymorphism

In our prediction of the distribution of nonsynonymous polymorphism, the distribution of selective effects assumed was a negative-gamma distribution with shape parameter $\alpha = 0.184$ and scale $\beta = 8200$ . The AFS was calculated by trapezoid-rule integration over this distribution, using 201 evaluations logarithmically spaced over $\gamma = [-300, -10^{-6}]$. All demographic parameters, including the scaled mutation rate $\theta$, were set to the maximum-likelihood values from our Out of Africa analysis.

# Results

First, we explored how various demographic forces affect the AFS, building intuition for our subsequent applications to real data. We then compared the performance of diffusion versus coalescent methods for evaluating the AFS, finding that the diffusion approach is substantially faster. We then applied our diffusion approach to infer parameters for plausible demographic models for the history of continental human populations. We first considered the expansion of humans out of Africa and then the settlement of the New World. In these applications, we inferred the maximum composite-likelihood parameters of our models using diffusion fits to the real data. To account for linkage in estimating variances and critical values for hypothesis tests, we then repeatedly fit both conventional and parametric bootstrap data sets. Finally, in an application incorporating selection, we predicted the distribution of nonsynonymous variation between populations in our Out of Africa model, finding good agreement with the available data.

## Demographic effects on the AFS

In Fig , we provide examples of the AFS under different demographic scenarios. Fig B illustrates the isolation-with-migration model for which the spectra are calculated. The expected spectrum at zero divergence time is shown in Fig C. Fig D shows the expected spectrum at various divergence times under various demographic scenarios. Qualitatively, correlation between population allele frequencies declines with increasing divergence time, depopulating the diagonal of the AFS. On the other hand, migration prolongs and sustains correlation. Less obviously, AFS entries corresponding to shared low-frequency alleles distinguish between increased migration and reduced divergence time (supporting information). Additionally, differences in genetic drift between populations with different effective sizes result in asymmetries in the AFS. These qualitative features of the AFS are also evident in human data; detailed modeling allows us to quantify our inference regarding the type, timing, and strength of demographic events that are consistent with the data.

## Computational performance

The computer program implementing our method is named $\partial$a$\partial$i (Diffusion Approximations for Demographic Inference). It is open-source and freely available at `http://dadi.googlecode.com`.

Fig E compares $\partial$a$\partial$i with a coalescent approach to evaluating the likelihood of frequency spectrum data. The coalescent simulator *ms*  was used to generate a simulated data set from the model in Fig B, with parameters $\nu_1=0.9$, $\nu_2 = 0.1$, $M=2$, $\tau = 2$, $\theta = 1000$, scaled total recombination rate $\rho=1000$, and 20 samples per population. Coalescent-based estimates of the expected AFS were generated by averaging $10^5$ *ms* simulations, each run with $\theta = 1$ and $\rho = 0$. These estimates were scaled to $\theta = 1000$ for comparison with the simulated data set. (This procedure is substantially faster than simulating with larger $\theta$ and $\rho$.) Each estimate took approximately 7.2 seconds of computation. The histogram in Fig E shows the resulting distribution of estimated likelihoods of the data. Shown by the red line in Fig E is the result from our diffusion approach (with grid sizes $G = \{40,50,60\}$), which took approximately 2.0 seconds of computation. The yellow line is the likelihood from $10^8$ coalescent simulations, illustrating the high accuracy of our diffusion approach. (Note that the coalescent approach we consider here is not necessarily optimal. We are, however, unaware of any such approach that is competitive in computational speed with the diffusion method.)

The computational advantage of the diffusion method is even larger when placed in the context of parameter optimization. Unlike the coalescent approach, there is no simulation variance, so efficient derivative-based optimization methods can be used. As examples, consider our applications to human data, which involve 20 samples per population. On a modern workstation, fitting a single-population three-parameter model took roughly a minute, while fitting a two-population six-parameter model took roughly 10 minutes. The fits of three-population models with roughly a dozen parameters typically took a few hours to converge from a reasonable initial parameter set. This speed allows us to use extensive bootstrapping to estimate variances, overcoming the limitations of composite likelihood.

## Expansion out of Africa

Our analysis of human expansion out of Africa used data from three HapMap populations: 12 Yoruba individuals from Ibadan, Nigeria (YRI); 22 CEPH Utah residents with ancestry from northern and western Europe (CEU); and 12 Han Chinese individuals sampled in Beijing, China (CHB). Because approaches based on the frequency spectrum are sensitive to miscalling of the ancestral state, we statistically corrected for ancestral misidentification using an approach that accounts for a myriad of mutation and context-dependent biases (such as CpG effects) . To ease qualitative comparison among populations and account for missing data, we projected the data down to 20 sampled chromosomes per population . Because this data set is of very high quality ($>$`<!-- -->`{=html}99% concordance of sequenced SNPs with next-generation sequencing of the same individuals to high coverage; see Materials and Methods), we do not explicitly correct for sequencing errors here. We were left with 17,446 segregating diallelic single nucleotide polymorphisms (SNPs) from effectively 4.04 Mb of sequence. Fig A shows the resulting AFS. For ease of visualization, the top row of Fig C shows the two-population marginal spectra.

There are many possible three-population demographic models one could consider for these populations. To develop a parsimonious yet realistic model, we first considered the marginal AFS for each population and each pair of populations. Previous analyses found that the YRI spectrum is well-fit by a two-epoch model with ancient population growth , and we found this as well (supporting information). Previous analyses of the CEU and CHB populations found that both populations went through bottlenecks  concurrent with divergence . Such models qualitatively fit the marginal CEU-CHB spectrum (supporting information).

```latex
\begin{table*}
\caption{
{\bf Out of Africa inferred parameters}}
\begin{minipage}{\textwidth}
\begin{tabular*}{\hsize}{@{\extracolsep{\fill}}cccc}
& &  conventional & parametric bootstrap\\
& maximum &  bootstrap 95\% & bias-corrected 95\%\\
parameter\footnote{See Fig~\ref{fig:YRIfit}B for model schematic. Growth rates $r$ and migration rates $m$ are per generation.} & likelihood  & confidence interval & confidence interval\\
\hline
$N_A$ &  7,300 & 4,400 -- 10,100 & 6,300 -- 9,200\\
$N_{AF}$ & 12,300 & 11,500 -- 13,900 & 11,100 -- 13,100\\
$N_B$ &  2,100 & 1,400 -- 2,900 & 1,700 -- 2,600\\
$N_{EU0}$ &  1,000 & 500 -- 1,900 & 500 -- 1,500\\
$r_{EU}$ (\%) &  0.40 & 0.15 -- 0.66 & 0.26 -- 0.57\\
$N_{AS0}$ &  510 & 310 -- 910 & 320 -- 750\\
$r_{AS}$ (\%) &  0.55 & 0.23 -- 0.88 & 0.32 -- 0.79\\
$m_{AF-B}$ ($\times 10^{-5}$) & 25 & 15 -- 34 & 19 -- 36\\
$m_{AF-EU}$ ($\times 10^{-5}$) & 3.0  & 2.0 -- 6.0 & 1.6 -- 7.6\\
$m_{AF-AS}$  ($\times 10^{-5}$) & 1.9 & 0.3 -- 10.4 & 0.7 -- 6.9\footnote{One low-migration outlier was removed for each of these estimations.}\\
$m_{EU-AS}$  ($\times 10^{-5}$) & 9.6 & 2.3 -- 17.4$^b$ & 5.7 -- 20.2\\
$T_{AF}$ (kya) & 220 &  100 -- 510 & 90 -- 410\\
$T_{B}$ (kya) & 140 & 40 -- 270 & 60 -- 310\\
$T_{EU-AS}$ (kya) & 21.2 & 17.2 -- 26.5 & 17.6 -- 23.9\\
\end{tabular*}
\end{minipage}
\label{tbl:YRIfit}
\end{table*}
```

Combining these demographic features yields the model illustrated in Fig B. The maximum likelihood values for the 14 free parameters are reported in Table . Qualitatively, the resulting model reproduces the observed spectra well, as seen in the second and third rows of Fig C. (The correlation between adjacent residuals is due in part to our projection of the data down from a larger sample size (supporting information).) Allowing for asymmetric gene flow yielded very little improvement in fit, as did allowing for growth in the Eurasian ancestral population or allowing the CEU and CHB bottleneck and divergence times to differ (data not shown).

Our composite likelihood function assumes that polymorphic sites are independent. Because it thus overestimates the number of effective independent data points, confidence intervals calculated directly from the composite likelihood function will be too liberal. To control for linkage, we performed both conventional and parametric bootstraps. Because our sequenced genes are typically well separated, they can be treated as independent, and our conventional bootstrap resampled from the 219 sequenced loci. For the parametric bootstrap, simulated data sets that incorporate linkage and the EGP's sequencing strategy were generated with *ms* .

Table  reports parameter 95% confidence intervals from both the conventional and bias-corrected parametric bootstraps. The parametric bootstraps yield slightly smaller confidence intervals than the conventional bootstrap, suggesting that some variability in the data has not been accounted for by our simulations. This variability may involve small varied selective forces on the sequenced regions, or slight relatedness between sampled individuals. The parametric bootstrap results additionally show that our method possesses very little bias in parameter inference (supporting information).

As seen in Table , the times for growth in the African ancestral population and divergence of the Eurasian ancestral population ($T_{AF}$ and $T_{B}$) have particularly wide confidence intervals, likely a consequence of the high inferred migration rate $m_{AF-B}$ between the African and Eurasian ancestral populations. $T_{AF}$ shows high correlation with the ancestral population size $N_A$, while $T_B$ shows no strong linear correlation with any other single parameter (supporting information). We found that 92 out of our 100 conventional bootstrap fits yield $N_{AS0} < N_{EU0}$, supporting the contention that the CHB population suffered a more severe bottleneck than the CEU population .

We used several metrics to assess our model's goodness-of-fit, in additional to visual inspection of the residuals seen in Fig C. Fig D compares the decay of linkage disequilibrium (LD) in the data and in the parametric bootstrap simulations. The agreement seen is notable because our demographic inference used no LD information in building and fitting the model. This LD comparison thus serves as independent validation of both our model and bootstrap simulations We also asked whether the likelihood $\mathcal{L}$ found in the real data fit is atypical of fits to simulated data. Out of fits to 100 simulated data sets, 2 produced a smaller likelihood (worse fit) than the real data fit (Fig E), yielding a p-value of $\approx$`<!-- -->`{=html}0.02. One can craft examples in which a likelihood-based goodness-of-fit test fails to exclude very poor models . Thus we also applied Pearson's $\chi^2$ goodness-of-fit test, a more robust and standard method for data that is in Poisson-distributed bins, such as the AFS . In our case, we must use our parametric bootstraps to assess the significance of the sum-of-squared-residuals test statistic $X^2$, because many entries in the AFS are small and because they are not strictly independent. Fig E shows the bootstrap-derived empirical distribution of $X^2$. Two of the bootstraps yielded a larger $X^2$ (worse fit) than the real data fit, giving a p-value of $\approx$`<!-- -->`{=html}0.02, identical to that from the likelihood-based test. (The two simulations that yield a higher $X^2$ than the real fit are not the same two that yield a lower $\mathcal{L}$, suggesting that these tests are somewhat independent.) In some cases specific frequency classes of SNPs, such as rare alleles, may be of particular interest. In the supporting information, we provide comparisons of the joint distribution of rare alleles seen in the data with that from our simulations. These comparisons indicate that our model also reproduces well this interesting region of the frequency spectrum. Finally, in Fig  we compare the model and data using larger bins of SNPs specific to specific populations or segregating at high or low frequency. In all cases the model agrees within the uncertainty of the bootstrapped data. Taken together, these tests suggest that our model provides a reasonable, though not complete, explanation of the data, lending credence to our demographic estimates.

The inferred contemporary migration parameters ($m_{AF-EU}$, $m_{AF-AS}$ and $m_{EU-AS}$) are small, raising the question as to whether they are statistically distinguishable from zero. Figure F shows that the improvement in fit to the real data upon adding contemporary migration to the model is much larger than would be expected if there were no such migration, implying that the contemporary migration we infer is highly statistically significant. Omitting ancient migration ($m_{AF-B}$) reduced fit quality even more, indicating that the data also demand substantial ancient migration.

## Settling the New World

To study the settlement of the Americas, we used the previously considered 22 CEU and 12 CHB individuals plus an additional 22 individuals of Mexican descent sampled in Los Angeles (MXL). Data were processed as in our Out of Africa analysis, yielding 13,290 segregating SNPs from effectively 4.22 Mb of sequence. Fig A shows the resulting AFS, while Fig C shows the marginal spectra.

A model in which the CEU and CHB diverge from an equilibrium population did not reproduce the AFS well (supporting information). Interestingly, a model allowing a prior size change in the ancestral population better fit the AFS but very poorly fit the observed LD decay (supporting information). Thus, reproducing the AFS does not guarantee reproduction of LD, at least given a historically unrealistic model. To develop a more realistic model, we endeavored to include the effects of Eurasian divergence from and migration with the African population. Computational limits precluded us from considering all 4 populations simultaneously, so we dropped the African population from the simulation upon MXL divergence (Fig B).

```latex
\begin{table*}
\caption{
{\bf Settlement of New World inferred parameters}}
\begin{minipage}{\textwidth}
\begin{tabular*}{\hsize}{@{\extracolsep{\fill}}cccc}
& &  conventional & parametric bootstrap\\
& maximum &  bootstrap 95\% & bias-corrected 95\%\\
parameter\footnote{See Fig~\ref{fig:MXLfit}B for model schematic. Growth rates $r$ and migration rates $m$ are per generation. $f_{MX}$ is the average European admixture proportion of the Mexican-Americans sampled.} & likelihood  & confidence interval & confidence interval\\
\hline
$N_{EU0}$ &  1,500 & 700 -- 2,100 & 900 -- 2,200\\
$r_{EU}$ (\%) &  0.23 & 0.08 -- 0.45 & 0.16 -- 0.34\\
$N_{AS0}$  &  590 & 320 -- 800 & 410 -- 790\\
$r_{AS}$ (\%) &  0.37 & 0.16 -- 0.60 & 0.24 -- 0.51\\
$N_{MX0}$ &  800 & 160 -- 1,800 & 140 -- 1,600\\
$r_{MX}$ (\%) &  0.50 & 0.14 -- 1.17 & 0.41 -- 0.98\\
$m_{EU-AS}$ ($\times 10^{-5}$) & 13.5 & 7.5 -- 32.2 & 9.9 -- 20.8\\
$T_{EU-AS}$ (kya) & 26.4 & 18.1 -- 43.1 & 21.7 -- 30.7\\
$T_{Mx}$ (kya) & 21.6 & 16.3 -- 26.9 & 18.6 -- 24.7\\
$f_{MX}$ (\%) & 48 & 42 -- 60 & 41 -- 55\\
\end{tabular*}
\end{minipage}
\label{tbl:MXLfit}
\end{table*}
```

Table  records the maximum-likelihood parameter values inferred for this model. Because this fit did not include African data, we could not reliably infer demographic parameters involving the African population. Thus, for this point estimate we fixed the Africa-related parameters $N_A$, $N_{AF}$, $N_B$, $m_{AF-B}$, $m_{AF-EU}$, $m_{AF-AS}$, $T_{AF}$ and $T_{B}$ to their maximum-likelihood values from Table . Fig C compares the model and data spectra. The residuals show little correlation, with the possible exception that the model may underestimate the number of high-frequency segregating alleles.

Parameter confidence intervals are reported in Table . To account for our uncertainty in those parameters derived from the Out of Africa fit, for each conventional bootstrap fit we used a set of Africa-related parameters randomly chosen from the sets yielded by our Out of Africa conventional bootstrap. For the parametric bootstrap, we used the maximum-likelihood point estimates. Again, we see that the conventional bootstrap confidence intervals are comparable to, although slightly wider than, the parametric bootstrap intervals. Several parameters in this analysis have direct correspondence with our Out of Africa analysis. Of particular note, the confidence intervals for the CEU-CHB divergence time $T_{EU-AS}$ overlap.

In assessing goodness of fit, Fig D shows that this model does indeed reproduce the observed pattern of LD decay. Unlike in our Out of Africa analysis, however, here the LD decay was used to choose the form of the model (although not its parameter values), so this is not a completely independent assessment of fit. Of our 100 parametric bootstrap fits, 13 yielded a worse likelihood than the real fit (Fig E), for a p-value of $\approx0.13$. Applying Pearson's $\chi^2$ test, we find that 23 of 100 bootstrap fits yield a higher (worse) $X^2$ than the fit to the real data, for a p-value of $\approx0.23$, similar to that of the likelihood analysis. Comparing distributions of rare alleles, our model typically reproduces the observed distribution well, although it may be somewhat overestimating the proportion of alleles that are rare or absent in the CHB population (supporting information). In sum, our model appears to be a reasonable explanation of this data, somewhat better than in our Out of Africa analysis.

An essential feature of the Mexican-American individuals considered here is that they are typically admixed from Native American and European ancestors. The $\approx$`<!-- -->`{=html}50% average European admixture proportion we inferred for the MXL population is consistent with previous estimates for Los Angeles Latinos . We have no direct data from the Native American populations ancestral to MXL, but our model does account for their divergence from East Asia. A model neglecting this divergence (by setting $T_{MX}$ to zero) fit the data substantially worse and yields an unrealistically high average European admixture proportion into MXL of 0.68.

Not only are Mexican-American individuals admixed, their admixture proportions also vary, and this subtlety is not directly accounted for in our analysis. To assess its effect on our results, we first roughly estimated the ancestry proportion of each individual, using essentially a maximum-likelihood version  of the algorithm used in *structure*  (supporting information). (Methods based on "admixture LD", which identify breakpoints between regions of Native American and European ancestry, may be more powerful . However, the strategy used by the EGP of sequencing widely spaced genes will resolve few of these breakpoints, limiting the applicability of these methods.) We then performed additional parametric bootstrap analyses, using simulations with a distribution of individual ancestry chosen to mimic that seen in the data and, to further test the method, with an extremely wide distribution. These simulations showed that variation in individual ancestry does not bias our parameter inferences (supporting information). Remarkably, it does not even change our statistical power. This is evidenced by the fact that these bootstrap simulations yielded confidence intervals identical to our original simulations without variation in ancestry proportion (supporting information). Nevertheless, future studies may profit by incorporating individual ancestry information , perhaps inferred from admixture LD.

Finally, our model allowed us to assess the role recurrent migration from Asia played in the settlement of the New World . When we added CHB-MXL migration to our model, we found that the maximum likelihood migration rate was $1.7 \times 10^{-5}$ per generation. As shown in Fig F, the resulting improvement in likelihood is typical (p-value $\approx$`<!-- -->`{=html}0.45) of fits including CHB-MXL migration to data simulated without it. Our data and analysis thus yielded no evidence of recurrent migration in the settlement of the New World. Note, however, that this simple test does not necessarily rule out more complex scenarios, in which migration may vary over time.

## Nonsynomymous polymorphism

Polymorphisms that change protein amino acid sequence are of medical interest because they are particularly likely to affect gene function . Correspondingly, they are often subject to natural selection. Diffusion approaches are particularly useful for studying such nonsynonymous polymorphism, because they easily incorporate selection. Although the diffusion approximation assumes that sites are unlinked, nonsynonymous segregating sites are rare enough that this is often a reasonable approximation .

As an illustration, we used our Out of Africa demographic model to predict the distribution of such variation between continental populations. To do so, we must specify a distribution for the selective effects of nonsynonymous mutations that enter the population. For this we adopted a negative gamma distribution whose parameters were recently inferred . The resulting distribution of segregating variation is shown in Figure A. (To ease comparison, we have assumed the same scaled mutation rate as in the neutral case of Fig C.) As expected, selection sharply reduces the amount of segregating polymorphism. Figure B shows the proportion of variants within various classes. Also as expected, selection shifts nonsynonymous variation toward lower frequencies, raising the proportion of singletons and lowering the proportion at frequency greater than 10%. Less obviously, it also reduces the proportion of variation that is shared between populations. In the neutral case, 43% of polymorphism is predicted to be present in more than one population, while in the selected case only 35% is. Thus genetic inferences from coding polymorphism may be less transferable between populations than might be expected from neutral patterns of allele sharing.

In the data considered here, there are about 400 nonsynomymous polymorphisms segregating in the three populations considered. This is too few for a detailed goodness-of-fit test of our predicted distribution. (Although see supporting information for a direct AFS comparison.) Nevertheless, we observe that our predictions shown in Figure B all lie within the bootstrap 95% confidence intervals from the data.

# Discussion

Our diffusion approximation to the joint allele frequency spectrum is a powerful tool for population genetic inference. Although the diffusion approximation neglects linkage between sites, our method's computational efficiency allows us to use extensive bootstrap simulations to account for the effects of linkage. (Let us reiterate that linkage does not affect the expected site-frequency spectrum of neutral sites, so our diffusion-based approach is estimating the same AFS that coalescent simulations are estimating, but in a small fraction of the time). We applied our method to human expansion out of Africa and settlement of the New World, using public resequencing data from the Environment Genome Project. The flexibility of the diffusion approach also allowed us to consider the distribution of non-neutral variation, which is difficult to address with other approaches. Although no model can capture in detail the complete history of any population, the models presented here help refine our understanding of human expansion across the globe.

Our demographic results are broadly consistent with previous analyses of human populations. In particular, single-population analyses have also inferred African population growth and European and Asian bottlenecks . Also, the migration rates we infer are similar to those inferred by Schaffner et al.  but somewhat smaller than those of Cox et al. . On the other hand, Keinan et al.  inferred no significant migration between CEU and CHB. Finally, our estimate of a New World founding effective population size in the hundreds is compatible other inferences .

Perhaps our most interesting demographic results are the inferred divergence times. Other studies  have estimated divergence times between Europeans and East Asians similar to the $\approx$`<!-- -->`{=html}23 kya we infer. Interestingly, archeological evidence places humans in Europe much earlier ($\approx$`<!-- -->`{=html}40 kya) . Our inferred divergence time of $\approx$`<!-- -->`{=html}22 kya between East Asians and Mexican-Americans is somewhat older than the oldest well-accepted New World archeological evidence . The divergence we infer may reflect the settlement of Beringia, rather than the expansion into the New World proper . Finally, the divergence time of $\approx$`<!-- -->`{=html}140 kya we infer between African and Eurasian populations is consistent with archeological evidence for modern humans in the Middle East $\approx$`<!-- -->`{=html}100 kya , but it is much older than other inferences of $\approx$`<!-- -->`{=html}50 kya divergence from mitochondrial DNA . This discrepancy may be explained by our inclusion of migration in the model. Migration preserves correlation between population allele frequencies, so an observed correlation across the genome can be explained by either recent divergence without migration or ancient divergence with migration. In fact, the African-Eurasian migration rate we infer of $\approx$$25\times 10^{-5}$ per generation is comparable to the $\approx$$100\times10^{-5}$ inferred from census records between modern continental Europe and Britain .

One difficulty in interpreting our divergence times is that the sampled populations may not best represent those in which historically important divergences occurred. For example, the Yoruba are a West African population, so the divergence time we infer between Yoruba and Eurasian ancestral populations may correspond to divergence within Africa itself. Future studies of more populations  will help alleviate this difficulty.

Another difficulty is that the genic loci we study here may not be ideal for demographic inference. Although we consider only noncoding sequence in fitting our historical model, selection on regulatory or linked coding sites may skew the AFS . In fact, the EGP data have been shown to differ in some ways (e.g. Tajima's $D$) from intergenic regions . Nevertheless, we use the EGP data because it is currently the largest public resource of noncoding human genetic variation, and we fit a neutral model because disentangling the small expected effects of selection on these sites from demographic effects will require additional data. The rapidly declining cost of sequencing will give future studies access to many more loci that are likely to be less influenced by selection. Importantly, the computational burden of our method is independent of the amount of sequence used to construct the AFS. Additional loci will also increase power to discriminate between models and incorporate more detail.

The AFS encodes substantial demographic information. It is has been shown, however, that an isolated population's AFS does not uniquely and unambiguously identify its demographic history ; we expect a similar result to hold for multiple interacting populations. Moreover, the AFS does not capture all the information in the data. As illustrated by the alternative New World models we considered, patterns of linkage disequilibrium encode additional information. Future studies may profit from coupling our efficient AFS simulation with methods that address other aspects of the data.

We have developed a powerful diffusion-based method for demographic inference from the joint allele frequency spectrum. We applied our method to human expansion out of African and the settlement of the New World, developing models of human history that refine our knowledge and raise intriguing questions. We also applied our method to predict the distribution of nonsynonymous variation across populations, and this prediction is consistent with the available data. Our methods and the models inferred from it offer a foundation for studying the history and evolution of both our own species and others.


abstract: The next-generation astronomy archives will cover most of the universe at fine resolution in many wavelengths. One of the first of these projects, the Sloan Digital Sky Survey (SDSS) will create a 5-wavelength catalog over 10,000 square degrees of the sky. The 200 million objects in the multi-terabyte database will have mostly numerical attributes, defining a space of 100+ dimensions. Points in this space have highly correlated distributions. The archive will enable astronomers to explore the data interactively. Data access will be aided by multidimensional spatial indices. The data will be partitioned in many ways. Small tag objects consisting of the most popular attributes speed up frequent searches. Splitting the data among multiple servers enables parallel, scalable I/O. Hashing techniques allow efficient clustering and pairwise comparison algorithms. Randomly sampled subsets allow debugging otherwise large queries at the desktop. Central servers will operate a data pump that supports sweeping searches that touch most of the data.
author: Alexander S. Szalay, Peter Kunszt, Anirudha Thakar; Jim Gray and Don Slutz
title: The Sloan Digital Sky Survey$^1$ and its Archive

# Introduction

Astronomy is undergoing a major paradigm shift. Data gathering technology is riding Moore's law: data volumes are doubling quickly, and becoming more homogeneous. For the first time data acquisition and archival is being designed for online interactive analysis. Shortly, it will be much easier to download a detailed sky map or object class catalog, than wait several months to access a telescope that is often quite small. Several multi-wavelength projects are under way: SDSS, GALEX, 2MASS, GSC-2, POSS2, ROSAT, FIRST and DENIS, each surveying a large fraction of the sky. Together they will yield a Digital Sky, of interoperating multi-terabyte databases. In time, more catalogs will be added and linked to the existing ones. Query engines will become more sophisticated, providing a uniform interface to all these datasets. In this era, astronomers will have to be just as familiar with mining data as with observing on telescopes.

# The Sloan Digital Sky Survey

The Sloan Digital Sky Survey (SDSS) will digitally map about half of the Northern sky in five spectral bands from ultraviolet to the near infrared. It is expected to detect over 200 million objects. Simultaneously, it will measure redshifts for the brightest million galaxies (see http://www.sdss.org/). The SDSS is the successor to the Palomar Observatory Sky Survey (POSS), which provided a standard reference data set to all of astronomy for the last 40 years. Subsequent archives will augment the SDSS and will interoperate with it. The SDSS project thus consists of not only of building the hardware, and reducing and calibrating the data, but also includes software to classify, index, and archive the data so that many scientists can use it. The SDSS will revolutionize astronomy, increasing the amount of information available to researchers by several orders of magnitude. The SDSS archive will be large and complex: including textual information, derived parameters, multi-band images, spectra, and temporal data. The catalog will allow astronomers to study the evolution of the universe in great detail. It is intended to serve as the standard reference for the next several decades. After only a month of operation, SDSS found the two most distant known quasars. With more data, other exotic properties will be easy to mine from the datasets. The potential scientific impact of the survey is stunning. To realize this potential, data must be turned into knowledge. This is not easy - the information content of the survey will be larger than the entire text contained in the Library of Congress.

The SDSS is a collaboration between the University of Chicago, Princeton University, the Johns Hopkins University, the University of Washington, Fermi National Accelerator Laboratory, the Japanese Participation Group, the United States Naval Observatory, and the Institute for Advanced Study, Princeton, with additional funding provided by the Alfred P. Sloan Foundation, NSF and NASA. The SDSS project is a collaboration between scientists working in diverse areas of astronomy, physics and computer science.

The survey will be carried out with a suite of tools developed and built especially for this project - telescopes, cameras, fiber spectrographic systems, and computer software. SDSS constructed a dedicated 2.5-meter telescope at Apache Point, New Mexico, USA. The telescope has a large, flat focal plane that provides a 3-degree field of view. This design balances the areal coverage of the instrument against the detector's pixel resolution. The survey has two main components: a photometric survey, and a spectroscopic survey. The photometric survey is produced by drift scan imaging of 10,000 square degrees centered on the North Galactic Cap using five broadband filters that range from the ultraviolet to the infrared. The effective exposure is 55 sec. The photometric imaging uses an array of 30x2Kx2K Imaging CCDs, 22 Astrometric CCDs, and 2 Focus CCDs. Its 0.4 arcsec pixel size provides a full sampling of the sky. The data rate from the 120 million pixels of this camera is 8 Megabytes per second. The cameras can only be used under ideal conditions, but during the 5 years of the survey SDSS will collect more than 40 Terabytes of image data. The spectroscopic survey will target over a million objects chosen from the photometric survey in an attempt to produce a statistically uniform sample. The result of the spectroscopic survey will be a three-dimensional map of the galaxy distribution, in a volume several orders of magnitude larger than earlier maps. The primary targets will be galaxies, selected by a magnitude and surface brightness limit in the r band. This sample of 900,000 galaxies will be complemented with 100,000 very red galaxies, selected to include the brightest galaxies at the cores of clusters. An automated algorithm will select 100,000 quasar candidates for spectroscopic follow-up, creating the largest uniform quasar survey to date. Selected objects from other catalogs will also be targeted.

The spectroscopic observations will be done in overlapping 3$^\circ$ circular tiles. The tile centers are determined by an optimization algorithm, which maximizes overlaps at areas of highest target density. The spectroscopic survey will utilize two multi-fiber medium resolution spectrographs, with a total of 640 optical fibers. Each fiber is 3 seconds of arc in diameter, that provide spectral coverage from 3900 - 9200 Å. The system can measure 5000 galaxy spectra per night. The total number of galaxy spectra known to astronomers today is about 100,000 - only 20 nights of SDSS data! Whenever the Northern Galactic cap is not accessible, SDSS repeatedly images several areas in the Southern Galactic cap to study fainter objects and identify variable sources. SDSS has also been developing the software necessary to process and analyze the data. With construction of both hardware and software largely finished, the project has now entered a year of integration and testing. The survey itself will take about 5 years to complete.

## The SDSS Archives

The SDSS will create four main data sets: a photometric catalog, a spectroscopic catalog, images, and spectra. The photometric catalog is expected to contain about 500 distinct attributes for each of one hundred million galaxies, one hundred million stars, and one million quasars. These include positions, fluxes, radial profiles, their errors, and information related to the observations. Each object will have an associated image cutout ("atlas image") for each of the five filters. The spectroscopic catalog will contain identified emission and absorption lines, and one-dimensional spectra for 1 million galaxies, 100,000 stars, and 100,000 quasars. Derived custom catalogs may be included, such as a photometric cluster catalog, or quasar absorption line catalog. In addition there will be a compressed 1TB Sky Map. These products add up to about 3TB.

The collaboration will release this data to the public after a period of thorough verification. This public archive is expected to remain the standard reference catalog for the next several decades. This long-lifetime presents design and legacy problems. The design of the SDSS archival system must allow the archive to grow beyond the actual completion of the survey. As the reference astronomical data set, each subsequent astronomical survey will want to cross-identify its objects with the SDSS catalog, requiring that the archive, or at least a part of it, be dynamic with a carefully defined schema and metadata.

Observational data from the telescopes is shipped on tapes to Fermi National Laboratory (FNAL) where it is reduced and stored in the Operational Archive (OA), protected by a firewall, accessible only to personnel working on the data processing. Data in the operational archive is reduced and calibrated via method functions. Within two weeks the calibrated data is published to the Science Archive (SA). The Science Archive contains calibrated data organized for efficient science use. The SA provides a custom query engine that uses multidimensional indices. Given the amount of data, most queries will be I/O limited, thus the SA design is based on a scalable architecture, ready to use large numbers of cheap commodity servers, running in parallel. Science archive data is replicated to Local Archives (LA) within another two weeks. The data gets into the public archives (MPA, PA) after approximately 1-2 years of science verification, and recalibration. A WWW server will provide public access.

The Science Archive and public archives employ a three-tiered architecture: the user interface, an intelligent query engine, and the data warehouse. This distributed approach provides maximum flexibility, while maintaining portability, by isolating hardware specific features. Both the Science Archive and the Operational Archive are built on top of Objectivity/DB, a commercial OODBMS.

Querying these archives requires a parallel and distributed query system. We have implemented a prototype query system. Each query received from the User Interface is parsed into a Query Execution Tree (QET) that is then executed by the Query Engine. Each node of the QET is either a query or a set-operation node, and returns a bag of object-pointers upon execution. The multi-threaded Query Engine executes in parallel at all the nodes at a given level of the QET. Results from child nodes are passed up the tree as soon as they are generated. In the case of aggregation, sort, intersection and difference nodes, at least one of the child nodes must be complete before results can be sent further up the tree. In addition to speeding up the query processing, this data push strategy ensures that even in the case of a query that takes a very long time to complete, the user starts seeing results almost immediately, or at least as soon as the first selected object percolates up the tree (Thakar etal 1999).

## Typical Queries

The astronomy community will be the primary SDSS user. They will need specialized services. At the simplest level these include the on-demand creation of (color) finding charts, with position information. These searches can be fairly complex queries on position, colors, and other parts of the attribute space. As astronomers learn more about the detailed properties of the stars and galaxies in the SDSS archive, we expect they will define more sophisticated classifications. Interesting objects with unique properties will be found in one area of the sky. They will want to generalize these properties, and search the entire sky for similar objects.

A common query will be to distinguish between rare and typical objects. Other types of queries will be non-local, like "find all the quasars brighter than r=22, which have a faint blue galaxy within 5 arcsec on the sky". Yet another type of a query is a search for gravitational lenses: "find objects within 10 arcsec of each other which have identical colors, but may have a different brightness". This latter query is a typical high-dimensional query, since it involves a metric distance not only on the sky, but also in color space. Special operators are required to perform these queries efficiently. Preprocessing, like creating regions of attraction is not practical, given the number of objects, and that the sets of objects these operators work on are dynamically created by other predicates.

# Data Organization

Given the huge data sets, the traditional Fortran access to flat files is not a feasible approach for SDSS. Rather non-procedural query languages, query optimizers, database execution engines, and database indexing schemes must replace traditional "flat" file processing. This "database approach" is mandated both by computer efficiency, and by the desire to give astronomers better analysis tools.

The data organization must support concurrent complex queries. Moreover, the organization must efficiently use processing, memory, and bandwidth. It must also support the addition of new data to the SDSS as a background task that does not disrupt online access.

It would be wonderful if we could use an off-the-shelf SQL, OR, or OO database system for our tasks, but we are not optimistic that this will work. As explained presently, we believe that SDSS requires novel spatial indices and novel operators. It also requires a dataflow architecture that executes queries concurrently using multiple disks and processors. As we understand it, current systems provide few of these features. But, it is quite possible that by the end of the survey, some commercial system will provide these features. We hope to work with DBMS vendors towards this end.

## Spatial Data Structures

The large-scale astronomy data sets consist primarily of vectors of numeric data fields, maps, time-series sensor logs and images: the vast majority of the data is essentially geometric. The success of the archive depends on capturing the spatial nature of this large-scale scientific data.

The SDSS data has high dimensionality – each item has thousands of attributes. Categorizing objects involves defining complex domains (classifications) in this N-dimensional space, corresponding to decision surfaces.

The SDSS teams are investigating algorithms and data structures to quickly compute spatial relations, such as finding nearest neighbors, or other objects satisfying a given criterion within a metric distance. The answer set cardinality can be so large that intermediate files simply cannot be created. The only way to analyze such data sets is to pipeline the answers directly into analysis tools. This data flow analysis has worked well for parallel relational database systems (DeWitt 92). We expect these data river ideas will link the archive directly to the analysis and visualization tools.

The typical search of these multi-Terabyte archives evaluates a complex predicate in k-dimensional space, with the added difficulty that constraints are not necessarily parallel to the axes. This means that the traditional indexing techniques, well established with relational databases, will not work, since one cannot build an index on all conceivable linear combinations of attributes. On the other hand, one can use the fact that the data are geometric and every object is a point in this k-dimensional space (Samet 1990a,b). Data can be quantized into containers. Each container has objects of similar properties, e.g. colors, from the same region of the sky. If the containers are stored as clusters, data locality will be very high - if an object satisfies a query, it is likely that some of the object's "friends" will as well. There are non-trivial aspects of how to subdivide, when the data has large density contrasts (Csabai etal 96).

These containers represent a coarse-grained density map of the data. They define the base of an index tree that tells us whether containers are fully inside, outside or bisected by our query. Only the bisected container category is searched, as the other two are wholly accepted or rejected. A prediction of the output data volume and search time can be computed from the intersection.

The SDSS data is too large to fit on one disk or even one server. The base-data objects will be spatially partitioned among the servers. As new servers are added, the data will repartition. Some of the high-traffic data will be replicated among servers. It is up to the database software to manage this partitioning and replication. In the near term, designers will specify the partitioning and index schemes, but we hope that in the long term, the DBMS will automate this design task as access patterns change.

There is great interest in a common reference frame the sky that can be universally used by different astronomical databases. The need for such a system is indicated by the widespread use of the ancient constellations - the first spatial index of the celestial sphere. The existence of such an index, in a more computer friendly form will ease cross-referencing among catalogs. A common scheme, that provides a balanced partitioning for all catalogs, may seem to be impossible; but, there is an elegant solution, a 'shoe that fits all': that subdivides the sky in a hierarchical fashion. Our approach is described in detail by Kunszt etal (1999).

## Broader Metadata Issues

There are several issues related to metadata for astronomy datasets. One is the database schema within the data warehouse, another is the description of the data extracted from the archive and the third is a standard representation to allow queries and data to be interchanged among several archives. The SDSS project uses Platinum Technology's Paradigm Plus, a commercially available UML tool, to develop and maintain the database schema. The schema is defined in a high level format, and a script generator creates the .h files for the C++ classes, and the .ddl files for Objectivity/DB. This approach enables us to easily create new data model representations in the future (SQL, IDL, XML, etc).

About 20 years ago, astronomers agreed on exchanging most of their data in self-descriptive data format. This format, FITS, standing for the Flexible Image Transport System (Wells 81) was primarily designed to handle images. Over the years, various extensions supported more complex data types, both in ASCII and binary form. FITS format is well supported by all astronomical software systems. The SDSS pipelines exchange most of their data as binary FITS files. Unfortunately, FITS files do not support streaming data, although data could be blocked into separate FITS packets. We are currently implementing both an ASCII and a binary FITS output stream, using such a blocked approach. We expect large archives to communicate with one another via a standard, easily parseable interchange format. We plan to define the interchange formats in XML, XSL, and XQL.

The Operational Archive exports calibrated data to the Science Archive as soon as possible. Datasets are sent in coherent chunks. A chunk consists of several segments of the sky that were scanned in a single night, with all the fields and all objects detected in the fields. Loading data into the Science Archive could take a long time if the data were not clustered properly. Efficiency is important, since about 20 GB will be arriving daily. The incoming data are organized by how the observations were taken. In the Science Archive they will be inserted into the hierarchy of containers as defined by the multi-dimensional spatial index, according to their colors and positions.

Data loading might bottleneck on creating the clustering units - databases and containers - that hold the objects. Our load design minimizes disk accesses, touching each clustering unit at most once during a load. The chunk data is first examined to construct an index. This determines where each object will be located and creates a list of databases and containers that are needed. Then data is inserted into the containers in a single pass over the data objects.

## Scalable Server Architectures

Accessing large data sets is primarily I/O limited. Even with the best indexing schemes, some queries must scan the entire data set. Acceptable I/O performance can be achieved with expensive, ultra-fast storage systems, or with many of commodity servers operating in parallel. We are exploring the use of commodity servers and storage to allow inexpensive interactive data analysis. We are still exploring what constitutes a balanced system design: the appropriate ratio between processor, memory, network bandwidth, and disk bandwidth.

Using the multi-dimensional indexing techniques described in the previous section, many queries will be able to select exactly the data they need after doing an index lookup. Such simple queries will just pipeline the data and images off of disk as quickly as the network can transport it to the astronomer's system for analysis or visualization. When the queries are more complex, it will be necessary to scan the entire dataset or to repartition it for categorization, clustering, and cross comparisons. Experience will teach us the ratio between processor power, memory size, IO bandwidth, and system-area-network bandwidth.

Our simplest approach is to run a scan machine that continuously scans the dataset evaluating user-supplied predicates on each object (Acharya 95). Consider building an array of 20 nodes, each with 4 Intel Xeon 450 Mhz processors, 256MB of RAM, and 12x18GB disks (4TB of storage in all). Experiments show that one such node is capable of reading data at 150 MBps while using almost no processor time (Hartman 99). If the data is spread among the 20 nodes, they can scan the data at an aggregate rate of 3 GBps. This half-million dollar system could scan the complete (year 2004) SDSS catalog every 2 minutes. By then these machines should be 10x faster. This should give near-interactive response to most complex queries that involve single-object predicates.

Many queries involve comparing, classifying or clustering objects. We expect to provide a second class of machine, called a hash machine that performs comparisons within data clusters. Hash machines redistribute a subset of the data among all the nodes of the cluster. Then each node processes each hash bucket at that node. This parallel-clustering approach has worked extremely well for relational databases in joining and aggregating data. We believe it will work equally well for scientific spatial data.

The hash phase scans the entire dataset, selects a subset of the objects based on some predicate, and "hashes" each object to the appropriate buckets - a single object may go to several buckets (to allow objects near the edges of a region to go to all the neighboring regions as well). In a second phase all the objects in a bucket are compared to one another. The output is a stream of objects with corresponding attributes.

These operations are analogous to relational hash-join, hence the name (DeWitt 92). Like hash joins, the hash machine can be highly parallel, processing the entire database in a few minutes. The application of the hash-machine to tasks like finding gravitational lenses or clustering by spectral type or by redshift-distance vector should be obvious: each bucket represents a neighborhood in these high-dimensional spaces. We envision a non-procedural programming interface to define the bucket partition and analysis functions.

The hash machine is a simple form of the more general data-flow programming model in which data flows from storage through various processing steps. Each step is amenable to partition parallelism. The underlying system manages the creation and processing of the flows. This programming style has evolved both in the database community (DeWitt 92, Graefe 93, Barclay 95) and in the scientific programming community with PVM and MPI (Gropp 98). This has evolved to a general programming model as typified by a river system (Arpaci-Dusseau 99).

We propose to let astronomers construct dataflow graphs where the nodes consume one or more data streams, filter and combine the data, and then produce one or more result streams. The outputs of these rivers either go back to the database or to visualization programs. These dataflow graphs will be executed on a river-machine similar to the scan and hash machine. The simplest river systems are sorting networks. Current systems have demonstrated that they can sort at about 100 MBps using commodity hardware and 5 GBps if using thousands of nodes and disks (Sort benchmark).

With time, each astronomy department will be able to afford local copies of these machines and the databases, but for now, they will be a network service. The scan machine will be interactively scheduled: when an astronomer has a query, it is added to the query mix immediately. All data that qualifies is sent back to the astronomer, and the query completes within the scan time. The hash and river machines will be batch scheduled.

## Desktop Data Analysis

Most astronomers will not be interested in all of the hundreds of attributes of each object. Indeed, most will be interested in only 10% of the entire dataset - but different communities and individuals will be interested in a different 10%. We plan to isolate the 10 most popular attributes (3 Cartesian positions on the sky, 5 colors, 1 size, 1 classification parameter) into small 'tag' objects, which point to the rest of the attributes. Then we will build a spatial index on these attributes. These will occupy much less space, thus can be searched more than 10 times faster, if no other attributes are involved in the query.

Large disks are available today, and within a few years 100GB disks will be common. This means that all astronomers can have a vertical partition of the 10% of the SDSS on their desktops. This will be convenient for targeted searches and for developing algorithms. But, full searchers will still be much faster on the server machines because the servers will have much more IO bandwidth and processing power. Vertical partitioning can also be applied by the scan, hash, and river machines to reduce data movement and to allow faster scans of popular subsets. We also plan to offer a 1% sample (about 10 GB) of the whole database that can be used to quickly test and debug programs. Combining partitioning and sampling converts a 2 TB data set into 2 gigabytes, which can fit comfortably on desktop workstations for program development.

It is obvious, that with multi-terabyte databases, not even the intermediate data sets can be stored locally. The only way this data can be analyzed is for the analysis software to directly communicate with the Data Warehouse, implemented on a server cluster, as discussed above. Such an Analysis Engine can then process the bulk of the raw data extracted from the archive, and the user needs only to receive a drastically reduced result set.

Given all these efforts to make the server parallel and distributed, it would be inefficient to ignore IO or network bottlenecks at the analysis level. Thus it is obvious that we need to think of the analysis engine as part of the distributed, scalable computing environment, closely integrated with the database server itself. Even the division of functions between the server and the analysis engine will become fuzzy - the analysis is just part of the river-flow described earlier. The pool of available CPU's will be allocated to each task.

The analysis software itself must be able to run in parallel. Since it is expected that scientists with relatively little experience in distributed and parallel programming will work in this environment, we need to create a carefully crafted application development environment, to aid the construction of customized analysis engines. Data extraction needs to be considered also carefully. If our server is distributed and the analysis is on a distributed system, the extracted data should also go directly from one of the servers to one of the many Analysis Engines. Such an approach will also distribute the network load better.

# Summary

Astronomy is about to be revolutionized by having a detailed atlas of the sky available to all astronomers. With the SDSS archive it will be easy for astronomers to pose complex queries to the catalog and get answers within seconds, and within minutes if the query requires a complete search of the database. The SDSS datasets pose interesting challenges for automatically placing and managing the data, for executing complex queries against a high-dimensional data space, and for supporting complex user-defined distance and classification metrics. The SDSS project is "riding Moore's law": the data set we started to collect today - at a linear rate - will be much more manageable tomorrow, with the exponential growth of CPU speed and storage capacity. The scalable archive design presented here will be able to adapt to such changes.

We would like to acknowledge support from the Astrophysical Research Consortium, the HSF, NASA and Intel's Technology for Education 2000 program, in particular George Bourianoff (Intel).