I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_1D_model_synthesis.ipynb".
The first task is:
parallelus
There was no convergence on a set of optimal parameter values.
three epoch model
The built-in three_epoch model specifies a piecewise history (with only instantaneous population size changes instead of gradual changes). At time $TF+TB$ the ancient population underwent a size change, stayed at this size ($\nu_B \times N_{ref}$) for $TB \times 2N_{ref}$ generations and then underwent a second size size change $TF \times 2N_{ref}$ generations in the past to the contemporary population size ($\nu_F \times N_{ref}$). The model has therefore two population size parameters, $\nu_B$ and $\nu_F$ as well as two time parameters, $TB$ and $TF$.
With 4 parameters, this model is already so complex that a sweep through the parameter space (i. e. starting optimisations from many different positions) is prohibitively time-consuming. I have therefore started optimisations with initial parameter values that were generated by randomly perturbing a neutral parameter combination, $\nu_B = \nu_F = TB = TF = 1$, by up to 4-fold. I can therefore not be certain whether the optimal parameter combinations thus found correspond to the global or a just a local optimum of the likelihood function.
erythropus
Can you write Python code for it?

# load optimisation results from file

ar_ery = []

import glob

for filename in glob.glob("OUT_three_epoch/ERY_perturb_ar_ery*.dill"):
    ar_ery.extend(dill.load(open(filename)))

# extract successful (i. e. converged) optimisation
successfull_popt = [flatten(out)[:9] for out in ar_ery if out[1][4] == 0]

# create data frame
df = pd.DataFrame(data=successfull_popt, \
                  columns=['nuB_0', 'nuF_0', 'TB_0', 'TF_0', 'nuB_opt', 'nuF_opt', 'TB_opt', 'TF_opt', '-logL'])

# sort data frame by negative log likelihood
df.sort_values(by='-logL', ascending=True) # smaller is better


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D.ipynb".
The first task is:
I wonder which population dadi assumes to be pop1. In the sfs2d spectrum object, ery is pop1 and par is pop2.
Can you write Python code for it?

?dadi.PhiManip.phi_1D_to_2D

# create link to function that specifies the model
func = split_1grow_2decline_1decline_nomig

# create extrapolating version of the model function
func_ex = dadi.Numerics.make_extrap_log_func(func)

?split_1grow_2decline_1decline_nomig

nu1s = 0.5
nu2s = 0.5
nu2f = 0.05
nu1b = 40
nu1f = 0.15
Ts = 0.1
T2 = 0.1
Tb = 0.1
Tf = 0.1

sfs2d_folded.sample_sizes

model_spectrum = func_ex((nu1s, nu2s, nu2f, nu1b, nu1f, Ts, T2, Tb, Tf), sfs2d_folded.sample_sizes, pts_l)

theta = dadi.Inference.optimal_sfs_scaling(model_spectrum.fold(), sfs2d_folded)

theta

dadi.Plotting.plot_2d_comp_multinom(model_spectrum.fold(), sfs2d_folded, vmin=1)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D.ipynb".
The first task is:
More colormaps
Can you write Python code for it?

sfs2d_folded = sfs2d_unfolded.fold()

# plot the folded GLOBAL minor allele frequency spectrum

dadi.Plotting.plot_single_2d_sfs(sfs2d_folded, vmin=1, cmap=pylab.cm.jet)

# setting the smallest grid size slightly larger than the largest population sample size (36)
pts_l = [40, 50, 60]


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D.ipynb".
The first task is:
The fitting of parameters for various 1D models to the SFS's of par and ery has indicated the following:
- ery has undergone a population size increase by >20 fold (between about 1-2 $\times2N_{ref}$ generations ago) and later (<1 $\times2N_{ref}$ generations ago) a decrease to about 15% of the ancient populations size
- par has undergone only one size change to <10% of the ancient population size, this is inferred to have happened in the distant past, about 2-4 ($\times 2N_{ref}$) generations ago
I think it would be good to incorporate this information in the specification of a more complex 2D model.
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D.ipynb".
The first task is:
There are a couple of built-in models that I could use, but I think I need a custom model here that includes the information from the 1D model fitting.
I would like to write a model function that specifies an ancient split between ery and par, then a population decline in par that lasts until the present and later an exponential growth in ery that is more recently followed by a population decline.
An alternative model to test would be a population decline in the ancestral population, followed by the split between, later population increase in ery which is more recently followed by a population decline.
Can you write Python code for it?

def split_1grow_2decline_1decline_nomig((nu1s, nu2s, nu2f, nu1b, nu1f, Ts, T2, Tb, Tf), (n1, n2), pts):
    """
    model function: specifies an ancient split, followed by growth in pop1 and
                    decline in pop2, later also decline in pop1

    nu1s: rel. size of pop1 after split
    nu2s: rel. size of pop2 after split
    nu2f: final rel. size for pop2
    nu1b: rel. size of pop1 after first size change
    nu1f: final rel. size of pop1
    Ts: time betweem population split and size change in pop2
    T2: time between size change in pop2 and first size change in pop1
    Tb: time between first and second size change in pop1
    Tf: time between second size change in pop1 and present

    The population split happend Tf+Tb+T2+Ts (x2N) generations in the past.

    n1,n2: sample sizes
    pts: number of grid points to use in extrapolation
    """

    # define grid
    xx = yy = dadi.Numerics.default_grid(pts)

    # phi for the equilibrium ancestral pop
    phi = dadi.PhiManip.phi_1D(xx)

    # population split into pop1 and pop2
    phi = dadi.PhiManip.phi_1D_to_2D(xx, phi)

    # stepwise change in size for pop1 and pop2 after split
    phi = dadi.Integration.two_pops(phi, xx, Ts, nu2=nu2s, nu1=nu1s, m12=0, m21=0)
    # stepwise change in size for pop2 only
    phi = dadi.Integration.two_pops(phi, xx, T2, nu2=nu2f, nu1=nu1s, m12=0, m21=0)
    # stepwise change in size for pop1 only
    phi = dadi.Integration.two_pops(phi, xx, Tb, nu2=nu2f, nu1=nu1b, m12=0, m21=0)
    # stepwise change in size for pop1 only
    phi = dadi.Integration.two_pops(phi, xx, Tf, nu2=nu2f, nu1=nu1f, m12=0, m21=0)

    # calculate spectrum
    sfs = dadi.Spectrum.from_phi(phi, (n1, n2), (xx, yy))
    return sfs


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
Apparently the continuous migration model predicts many more variants at frequency 1 in erythropus that are shared with parallelus than the ancient migration model.
ancient migration, isolation and secondary contact
The two populations are relatively close to the hybrid zone centre at the Col de la Quillane in the Pyrenees. The clines for some characters have been shown to be many kilometers wide. It can therefore be assumed that there is some recent gene flow between the two populations. Can this gene flow be detected from this spectrum?
Can you write Python code for it?

def split_mig_iso_mig(params, ns, pts):
    """
    params = (nu1,nu2,Tc,m1,m2,Ti,Tsc)
    ns = (n1,n2)

    Split into two populations of specifed size, with potentially asymmetric migration
    for a time Tc followed by a period of complete isolation Ti which is followed by a restart of
    migration until present. Migration rates in the two epochs are assumed to equal (and constant).

    nu1: Size of population 1 after split.
    nu2: Size of population 2 after split.
    Tc: Time of gene flow after split (in units of 2*Na generations)
    m1: Migration rate from ery into par (in units of 2*Na ind per generation)
    m2: Migration rate from par into ery (in units of 2*Na ind per generation)
    Ti: Time of isolation after cessation of gene flow
    Tsc: Time of gene flow after period of isolation until present

    The split lies Tc+Ti+Tsc * 2Na generations in the past.

    n1,n2: Sample sizes of resulting Spectrum
    pts: Number of grid points to use in integration.
    """
    nu1,nu2,Tc,m1,m2,Ti,Tsc = params

    xx = dadi.Numerics.default_grid(pts)

    phi = dadi.PhiManip.phi_1D(xx)

    # split
    phi = dadi.PhiManip.phi_1D_to_2D(xx, phi)

    # divergence with potentially asymmetric migration
    phi = dadi.Integration.two_pops(phi, xx, Tc, nu1, nu2, m12=m2, m21=m1)

    # divergence without gene flow
    phi = dadi.Integration.two_pops(phi, xx, Ti, nu1, nu2, m12=0, m21=0)

    # divergence with potentially asymmetric migration
    phi = dadi.Integration.two_pops(phi, xx, Tsc, nu1, nu2, m12=m2, m21=m1)

    fs = dadi.Spectrum.from_phi(phi, ns, (xx,xx))
    return fs


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
Cells with counts below 1 are masked and appear white in the upper plot.
Can you write Python code for it?

sfs2d = sfs2d_unfolded.fold()

# plot the folded GLOBAL minor allele frequency spectrum

dadi.Plotting.plot_single_2d_sfs(sfs2d, vmin=1, cmap=pylab.cm.jet)

ns = sfs2d.sample_sizes

# setting the smallest grid size slightly larger than the largest population sample size (36)
pts_l = [40, 50, 60]


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
Compare best fit IM with RSC model spectrum
I would like to compare expected spectra from a recent secondary contact model with expected spectra from an isolation with continuous migration model (previous section).
Can you write Python code for it?

success = [flatten(out)[:9] for out in ar_split_mig if out[1][4] == 0]

df_split_mig = pd.DataFrame(data=success, \
                  columns=['nu1_0','nu2_0', 'T_0', 'm_0', 'nu1_opt', 'nu2_opt', 'T_opt', 'm_opt', '-logL'])

popt_split_mig = np.array(df_split_mig.sort_values(by='-logL', ascending=True).iloc[0, 4:8])
popt_split_mig

# get split_mig model function (built-in)
func_ex_split_mig = dadi.Numerics.make_extrap_log_func(dadi.Demographics2D.split_mig)

# get optimally scaled split-migration model spectrum
model_spectrum_split_mig = dadi.Inference.optimally_scaled_sfs(func_ex_split_mig(popt_split_mig, ns, pts_l), sfs2d)

# parameter combination that specifies a recent secondary contact model
p_sec_contact = [0.67858642,  1.29659005,  0.35015044,  0.00483189,  1.92454244]


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
From the dadi example script YRI_CEY.py:

Because this is a test of a parameter on the boundary of the parameter space
(m cannot be less than zero), our null distribution is an even proportion
of chi^2 distributions with 0 and 1 d.o.f. To evaluate the p-value, we use the
point percent function for a weighted sum of chi^2 dists.

See Self2007 for more complex cases.
Can you write Python code for it?

# calculate p-value for Chi-square dist. with 1 degree of freedom
p = dadi.Godambe.sum_chi2_ppf(D, weights=(0.5, 0.5))
p


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
Gene flow significantly improves the fit to the observed spectrum. I doubt that there could be enough linkage in the data to compromise this result.
Recent secondary contact model
Phylogeographic studies as well as paleoclimatic data strongly suggest that the two subspecies, erythropus and parallelus, diverged in isolation during at least the last ice age and only recently (~10,000 YBP) came into secondary contact. Clines of several morphological as well as molecular markers have been shown to be very wide, which indicates substantial gene flow across the hybrid zone into adjacent pure populations. The two population samples come from such "pure" populations in close proximity to the hybrid zone. They may therefore show evidence of recent gene flow between the two subspecies.
I am now defining a model function that is a mixture of the split_nomig and split_mig models. It specifies a split, divergence in isolation for a time $T_i$, then secondary contact of the two diverged populations since $T_c$ times $2N_{ref}$ generations in the past.
Can you write Python code for it?

def secondary_contact(params, ns, pts):
    """
    params = (nu1,nu2,Ti,Tc,m)
    ns = (n1,n2)

    Split into two populations of specified size,
    then diverge in isolation,
    finally come into secondary contact.

    nu1: Size of population 1 after split.
    nu2: Size of population 2 after split.
    Ti: Time of divergence in isolation (in units of 2*Na generations)
    Tc: Time in the past when secondary contact began (and continues till present)
    m: symmetrical migration rate (in units of 2*Na per generation)

    n1,n2: Sample sizes of resulting Spectrum
    pts: Number of grid points to use in integration.
    """
    nu1,nu2,Ti,Tc,m = params

    xx = dadi.Numerics.default_grid(pts)

    phi = dadi.PhiManip.phi_1D(xx)
    phi = dadi.PhiManip.phi_1D_to_2D(xx, phi) # split

    # divergence in isolation
    phi = dadi.Integration.two_pops(phi, xx, Ti, nu1, nu2, m12=0, m21=0)

    # secondary contact
    phi = dadi.Integration.two_pops(phi, xx, Tc, nu1, nu2, m12=m, m21=m)

    fs = dadi.Spectrum.from_phi(phi, ns, (xx,xx))
    return fs

ar_sec_contact = []

for filename in glob("OUT_2D_models/sec_contact*dill"):
    ar_sec_contact.append(dill.load(open(filename)))

# get parameters from all runs into a table

returned = ([flatten(out)[:11] for out in ar_sec_contact])

df = pd.DataFrame(data=returned, \
                  columns=['nu1_0','nu2_0', 'Ti_0', 'Tc_0', 'm_0', 'nu1_opt', 'nu2_opt', 'Ti_opt', 'Tc_opt', 'm_opt', '-logL'])

df.sort_values(by='-logL', ascending=True).head(10)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
I am assuming that $\nu_1$ refers to ery and $\nu_2$ refers to par. The split_nomig model with its optimal parameter values suggests the following:
Can you write Python code for it?

print "The ancestral population of ery and par split apart {0:,} generations ago.".format(int(0.290465*2*N_ref)),
print "Since then until present the ery population had a size of {0:,} and the par population of {1:,}.".format(int(0.636945*N_ref)
, int(1.229606*N_ref))


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
Interpretation of parameters
Can you write Python code for it?

mu = 3e-9
L = sfs2d.data.sum() # this sums over all entries in the spectrum, including masked ones, i. e. also contains invariable sites
print "The optimal value of theta per site for the ancestral population is {0:.5f}.".format(theta/L)
N_ref = theta/L/mu/4
print "The effective size of the ancestral population of ery and par (in number of diploid individuals) implied by this theta is: {0:,}.".format(int(N_ref))

print "The ancestral population split apart {0:,} generations ago.".format(int(popt[2]*2*N_ref)),
print "Immediately after the split the ERY population changed to a size of {0:,} and the PAR population to {1:,}.".format(int(popt[0]*N_ref), int(popt[1]*N_ref)),
print "Since the split of the ancestral population, PAR received 1 individual from ERY every {0:.2f} generations,".format(1.0/(popt[3]/2*popt[1])),
print "while ERY received 1 PAR individual every {0:.2f} generations.".format(1.0/(popt[4]/2*popt[0])),
print "Put another way: The PAR population contained a constant proportion of {0:.2e} of new immigrant alleles each generation".format(popt[3]/2/N_ref/popt[1]),
print "and the ERY population contained a constant proportion of {0:.2e} of new immigrant alleles each generation.".format(popt[4]/2/N_ref/popt[0])


mu = 3e-9
L = sfs2d.data.sum() # this sums over all entries in the spectrum, including masked ones, i. e. also contains invariable sites
print "The total sequence length for the 2D spectrum is {0:,}.".format(int(L))
N_ref = theta/L/mu/4
print "The effective size of the ancestral population Nref of ery and par (in number of diploid individuals) implied by this theta is: {0:,}.".format(int(N_ref))

print "After the split of the ancestral population {0:,} generations ago, the daughter populations increased in size (instantaneously).".format(int((popt[2]+popt[3])*2*N_ref)),
print "ERY increased to about the same size as the ancestral population (Nref), while PAR increased to {0:.2f} times of Nref.".format(popt[1]),
print "After the split ERY and PAR diverged in isolation for {0:,} generations.".format(int(popt[2]*2*N_ref)),
print "The two populations came into secondary contact {0:,} generations ago.".format(int(popt[3]*2*N_ref)),
print "ERY and PAR stayed in continuous secondary contact until the present and",
print "during that time exchanged one individual from par to ery every {0:.1f} generations".format(1.0/(popt[4]/2*popt[0])),
print "and one individual from ery into par every {0:.1f} generations.".format(1.0/(popt[4]/2*popt[1]))


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
It looks as though allowing for asymmetrical gene flow significantly improves the fit to the data. Linkage is unlikely to push the p-value above a significance value of 1e-2, for instance.
Is the migration rate from ery into par (m1) statistically significant?
Can you write Python code for it?

ar_split_asym_mig_nom1 = []

for filename in glob("OUT_2D_models/split_asym_mig_nom1*dill"):
    ar_split_asym_mig_nom1.append(dill.load(open(filename)))

l = 2*5+1

returned = [flatten(out)[:l] for out in ar_split_asym_mig_nom1]

df = pd.DataFrame(data=returned, \
                  columns=['nu1_0','nu2_0', 'T_0', 'm1_0', 'm2_0', 'nu1_opt', 'nu2_opt', 'T_opt', 'm1_opt', 'm2_opt', '-logL'])

df.sort_values(by='-logL', ascending=True).head(10)

ll_s = -18340.9

D = 2*(ll_c - ll_s)
D

# calculate p-value for Chi-square dist.
# the weights specify a weighted sum of a chi^2 distributions with 1 and 2 d.o.f
#
p = dadi.Godambe.sum_chi2_ppf(D, weights=(0.5, 0.5))
p


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
It was difficult to achieve convergence. Parameter combinations with quite different time of primary contact ($T_C$) have very similar likelihood.
The log likelihood of the split_asym_mig model with best-fit parameters was: -18104.96.
Can you write Python code for it?

D = 2*(-17582.88 - -18104.96)
D

# calculate p-value for Chi-square dist.
# the weights specify a weighted sum of chi^2 distributions with 1 and 2 d.o.f
# this is because Ti is 0 in the split_asym_mig model and at the boundary of the parameter space
p = dadi.Godambe.sum_chi2_ppf(D, weights=(0.5, 0.5))
p


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
Note, that the fitting of 1D models to the spectra of each population had indicated a population size reduction for both populations and more so for parallelus, which seems to contradict what this 2D model says about the two populations.
compare marginal spectra
As mentioned above, the population size history inferred from the 2D spectrum contradicts what was inferred from the individual 1D spectra of each population. The 2D spectrum is not computed from the same number of sites as the two 1D spectra. While inclusion of a site in each 1D spectrum required read data from at least 9 individuals, the 2D spectrum required read data from at least 9 individuals in both populations. That is because each population's SAF files were created with minInd 9 (see lines 1426-1435 in assembly.sh). When estimating the 2D SFS with realSFS only sites that occurred in each population's SAF files were included (see lines 1661-1664 in assembly.sh).
Can you write Python code for it?

# get marginal 1D spectra from the 2D spectrum

fsm_ery = sfs2d.marginalize([1])
fsm_par = sfs2d.marginalize([0])

# get original 1D spectra

# import 1D unfolded spectrum of ery
fs_ery_unfolded = dadi.Spectrum.from_file('ERY.unfolded.sfs')
fs_ery = fs_ery_unfolded.fold()

# import 1D unfolded spectrum of par
fs_par_unfolded = dadi.Spectrum.from_file('PAR.unfolded.sfs')
fs_par = fs_par_unfolded.fold()

pylab.plot(fs_par, 'go-', label='individual', linewidth=2.5, markersize=12)
pylab.plot(fsm_par, 'g>--', label='marginal', linewidth=2.5, markersize=12)
pylab.grid()
pylab.title('PAR')
pylab.legend()


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
The 2D spectrum contains counts from 60k sites that are variable in par or ery or both.
For the estimation of the 2D SFS, realSFS has only taken sites that had data from at least 9 individuals in each population (see assembly.sh, lines 1430 onwards).
plot the data
Can you write Python code for it?

import pylab

%matplotlib inline

pylab.rcParams['font.size'] = 14.0
pylab.rcParams['figure.figsize'] = [14.0, 12.0]

dadi.Plotting.plot_single_2d_sfs(sfs2d_unfolded, vmin=1, cmap=pylab.cm.jet)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
The best parameter combination from the ancient migration model had a likelihood of 17582.88. So the addition of $T_{SC}$ to the model had not improved the fit significantly. A time of secondary contact cannot be inferred from this spectrum.
Can you write Python code for it?

popt = np.array(df.sort_values(by='-logL', ascending=True).iloc[1,7:14])
popt

func_ex = dadi.Numerics.make_extrap_log_func(split_mig_iso_mig)

model = func_ex(popt, ns, pts_l)

theta = dadi.Inference.optimal_sfs_scaling(model, sfs2d)

mu = 3e-9
L = sfs2d.data.sum() # this sums over all entries in the spectrum, including masked ones, i. e. also contains invariable sites

N_ref = theta/L/mu/4

print "The effective size of the ancestral population of ery and par (in number of diploid individuals) implied by this theta is: {0:,}.".format(int(N_ref))
print "A Tsc of {0:.6f} corresponds to {1:,} generations.".format(popt[-1], int(popt[-1]*2*N_ref))


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
The gene flow from ery into par is highly significant. It is unlikely that this result can be compromised by linkage between the SNP's.
compare symmetric with asymmetric gene flow
Can you write Python code for it?

# get maximum log likelihood of best fit parameter combination for the model
# with symmetrical migration rates

func = dadi.Demographics2D.split_mig

func_ex = dadi.Numerics.make_extrap_log_func(func)

# best-fit paramter combination for the split_mig model (see section 2 above)
popt_split_mig = [0.994185, 1.766127, 0.922632, 0.250688]

model_spectrum_sym_mig = func_ex(popt_split_mig, ns, pts_l)

model_spectrum_sym_mig_scaled = dadi.Inference.optimally_scaled_sfs(model_spectrum_sym_mig, sfs2d)

dadi.Plotting.plot_2d_comp_multinom(model=model_spectrum_asym_mig_scaled.fold(), \
                                    data=model_spectrum_sym_mig_scaled.fold(), vmin=1,\
                                   title=['sym mig', 'asym mig'], pop_ids=['ery', 'par'])


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
The marginal 1D spectrum for ERY looks quite different. It mostly looks shifted down. This makes sense, when a smaller number of sites (those that are present in the SAF files of both ERY and PAR) go into the 2D spectrum than into the individual 1D spectrum.
Since the individual and marginal 1D spectra have similar shape, they should lead to similar demographic inference. The difference in the inference of population sizes for the two populations from 1D and 2D models can therefore not be explained by sytematic differences between 1D and 2D spectra.
plot residuals
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
The model spectrum for asymmetric gene flow has an excess (brown) of low to medium MAF SNP's in PAR that occur at low frequency in ERY. It has fewer medium to high frequency SNP's that are private to PAR and more medium to high frequency SNP's that are private to ERY. It has far fewer low frequency SNP's that are private to ERY.
Can you write Python code for it?

popt_split_mig = popt_split_mig + [popt_split_mig[-1]]
popt_split_mig

popt

pd.DataFrame(data=np.array([popt_split_mig, popt]), index=['sym_mig', 'asym_mig'], columns=['nu1', 'nu2', 'T', 'm1', 'm2'])


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
The optimal parameter values look very robust: very different starting values converge to almost identical optimal parameter values.
Interpretation of parameters
Can you write Python code for it?

# create link
func = split_nomig

# create extrapolating version of the model function
func_ex = dadi.Numerics.make_extrap_log_func(func)

# calculate best-fit model spectrum

model_spectrum = func_ex((0.636945, 1.229606, 0.290465), ns, pts_l)

theta = dadi.Inference.optimal_sfs_scaling(model_spectrum, sfs2d)

L = sfs2d.data.sum() # this sums over all entries in the spectrum, including masked ones, i. e. also contains invariable sites
print "The optimal value of theta per site for the ancestral population is {0:.4f}.".format(theta/L)

mu = 3e-9
print "The total sequence length for the 2D spectrum is {0:,}.".format(int(L))
N_ref = theta/L/mu/4
print "The effective size of the ancestral population of ery and par (in number of diploid individuals) implied by this theta is: {0:,}.".format(int(N_ref))


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
These parameter combinations consistently indicate a much smaller gene flow from erythropus into parallelus than in the other direction. This seems plausible when considering the observed expansion of the range of parallelus towards the south of the Pyrenees. Shapes of neutral clines across the hybrid zone should also be less steep on the erythropus side than on the parallelus side. A more trivial explanation is that the sampling location for ERY (Greixer) is closer to the cline centre than the sampling location for PAR (Aunat). A quick look at the map does not seem to support this, but migration barriers, also those that do not exist anymore, like forests, will have had an effect on the past gene flow.
plot residuals
Can you write Python code for it?

# get optimal parameter combination
popt = np.array(df.sort_values(by='-logL', ascending=True).iloc[0, 5:10])
popt

func = split_asym_mig

func_ex = dadi.Numerics.make_extrap_log_func(func)

# calculate best-fit model spectrum
model_spectrum = func_ex(popt, ns, pts_l)

# optimally scale model spectrum
model_spectrum_asym_mig_scaled = dadi.Inference.optimally_scaled_sfs(model_spectrum, sfs2d)

theta = dadi.Inference.optimal_sfs_scaling(model_spectrum, sfs2d)
theta

dadi.Plotting.plot_2d_comp_multinom(model_spectrum_asym_mig_scaled, sfs2d, vmin=1)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
This RSC parameterisation reduces the fit to the data by 1137 log likelihood units cmpared to the best-fit parameters which suggest practically no time of complete isolation.
Can you write Python code for it?

# I have edited the functions `dadi.Plotting.plot_2d_comp_multinom` and `dadi.Plotting.plot_2d_comp_Poisson`
# in `Plotting.py` in the dadi repo to add the argument `title`
dadi.Plotting.plot_2d_comp_multinom(data=SC_model_spectrum.fold(), model=model_spectrum_split_mig.fold(), \
                                    vmin=1, title=['RSC', 'continuous migration'], pop_ids = ['ery', 'par'])


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
This ancient migration model fits the data significantly better than the split asym mig model. That also means that the recent period of complete isolation ($T_i$) is significant. Assuming that I had found the best-fit parameters for that model, this means that a model that allows for a very ancient split (>1 million generations ago) and that specifies a recent cessation of gene flow fits the data better than a model with a less ancient split and constant migration until the present.
plot residuals
Can you write Python code for it?

popt_mig_iso = np.array(df.sort_values(by='-logL', ascending=True).iloc[0,6:12])
popt_mig_iso

# create unscaled model spectrum

func_ex = dadi.Numerics.make_extrap_log_func(split_asym_mig_iso)

model_mig_iso = func_ex(popt_mig_iso, ns, pts_l)

dadi.Plotting.plot_2d_comp_multinom(model=model_mig_iso, data=sfs2d, vmin=1)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
This has reduced the residuals for low frequency shared polymorphisms along the diagonal, but the model still cannot fit divergently fixed SNP's well.
Interpretation of parameters
Translation of m
The dadi manual says about the migration rate parameter (on page 11):

The migration parameter m12 specifies the rate of migration from pop 2 into pop 1. It is equal to the fraction of individuals each generation in pop 1 that are new migrants from pop 2, times the 2Nref.

So the fraction of new immigrant alleles each generation from $pop_2$ in $pop_1$ is:
$$
p_{m12} = \frac{m_{12}}{2N_{ref}}
$$
Since dadi's migration rate is scaled by 2Nref, the migration rate parameter as reported by dadi has units of immigrant gametes per generation. However, whenever pop 1 does not have the same size as the ancestral population ($N_{ref}$), dadi's $m_{12}$ needs to be rescaled by multiplication with $\nu_1$ to get the number of immigrant gametes per generation ($M_{12}$):
$$
\begin{align}
p_{m12} &= \frac{m_{12}}{2N_{ref}\nu_1} \[5pt]
M_{12} &= p_{m12} \times 2N_{ref}\nu_1 \[5pt]
M_{12} &= \nu_1 m_{12}
\end{align}
$$
See this confirm units of dadi thread on the dadi forum.
What this model fit says in plain English:
Can you write Python code for it?

print "The ancestral population split apart {0:,} generation ago.".format(int(popt[2]*2*N_ref)),
print "Since then, ery and par had a constant population size of {0:,} and {1:,}, respectively.".format(int(popt[0]*N_ref), int(popt[1]*N_ref)),
print "Since the split, a fraction of {0:.2e} of the population size of ERY were made up each generation of new immigrant individuals from PAR".format(popt[3]/2/N_ref/popt[0]),
print " and a fraction {0:.2e} of the population size of PAR were made up each generation of new immigrant individuals from ERY.".format(popt[3]/2/N_ref/popt[1]),
print "Put another way:",
print "Since the split ery received a constant number of {0:.2f} new immigrant alleles per generation, while par received a constant number of {1:.2f} per generation.".format(popt[3]*popt[0], popt[3]*popt[1])


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
This is a piecewise model. I could make gene flow fade out gradually, but that would require four more parameters that specify the initial and final levels of gene flow. Also, it would make the model less comparable to the split_asym_mig model, which is also piecewise.
Can you write Python code for it?

ar_split_asym_mig_iso = []

for filename in glob("OUT_2D_models/split_asym_mig_iso*dill"):
    ar_split_asym_mig_iso.append(dill.load(open(filename)))

l = 2*6+1

# get all optimisation results
returned = [flatten(out)[:l] for out in ar_split_asym_mig_iso]

df = pd.DataFrame(data=returned, \
                  columns=['nu1_0','nu2_0', 'Tc_0', 'm1_0', 'm2_0', 'Ti_0', 'nu1_opt', 'nu2_opt', 'Tc_opt', 'm1_opt', 'm2_opt', 'Ti_opt', '-logL'])

df.sort_values(by='-logL', ascending=True).head(10)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
This model adds a single migration rate parameter $m$ . It is assumed that migration is of equal size $m$ in each direction.
Can you write Python code for it?

# load optimisation results from file

ar_split_mig = []

for filename in glob("OUT_2D_models/split_mig_[0-9]*dill"):
    ar_split_mig.append(dill.load(open(filename)))

success = [flatten(out)[:9] for out in ar_split_mig if out[1][4] == 0]

df = pd.DataFrame(data=success, \
                  columns=['nu1_0','nu2_0', 'T_0', 'm_0', 'nu1_opt', 'nu2_opt', 'T_opt', 'm_opt', '-logL'])

df.sort_values(by='-logL', ascending=True)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
This model is hard to fit to the data. The best-fit parameter combinations contain an extremely long divergence time T of around 3, i. e. $6 \times N_{ref}$ generations. No parameter combination with higher likelihood could be found than in the previous runs and they are all much lower than from the split_asym_mig model (-18104), which imposed a stepwise size change coincident with the population split and otherwise constant population sizes.
plot residuals
Can you write Python code for it?

popt = np.array(df.sort_values(by='-logL', ascending=True).iloc[0, 6:12])
popt

func_ex = dadi.Numerics.make_extrap_log_func(IM)

model_spectrum = func_ex(popt, ns, pts_l)

dadi.Plotting.plot_2d_comp_multinom(model_spectrum, sfs2d, vmin=1)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
This model specifies a split of the ancestral population $T \times 2N_{ref}$ generations ago. The two daughter populations have a constant population size until the present of $\nu_1 \times N_{ref}$ and $\nu_2 \times N_{ref}$, respectively.
Can you write Python code for it?

# load optimisation results from file

ar_split_nomig = []

for filename in glob("OUT_2D_models/split_nomig*dill"):
    ar_split_nomig.append(dill.load(open(filename)))

success = [flatten(out)[:7] for out in ar_split_nomig if out[1][4] == 0]

df = pd.DataFrame(data=success, \
                  columns=['nu1_0','nu2_0', 'T_0', 'nu1_opt', 'nu2_opt', 'T_opt', '-logL'])

df.sort_values(by='-logL', ascending=True)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
This model therefore indicates a much stronger introgression from parallelus into erythropus than in the opposite direction. Is the asymmetrical gene flow significant?
LRT
Can you write Python code for it?

ll_c = dadi.Inference.ll_multinom(model_spectrum_asym_mig_scaled, sfs2d)
ll_c

# get maximum log likelihood of best fit parameter combination for the model
# with symmetrical migration rates

func = dadi.Demographics2D.split_mig

func_ex = dadi.Numerics.make_extrap_log_func(func)

# best-fit paramter combination for the split_mig model (see section 3 above)
popt_split_mig = [0.994185, 1.766127, 0.922632, 0.250688]

model_spectrum_sym_mig = func_ex(popt_split_mig, ns, pts_l)

model_spectrum_sym_mig_scaled = dadi.Inference.optimally_scaled_sfs(model_spectrum_sym_mig, sfs2d)

ll_s = dadi.Inference.ll_multinom(model_spectrum_sym_mig, sfs2d)
ll_s

D = 2*(ll_c - ll_s)
D

# calculate p-value for Chi-square dist. with 1 degree of freedom
p = dadi.Godambe.sum_chi2_ppf(D)
p


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
compare asym_mig with IM model
Can you write Python code for it?

# optimally scale the best-fit IM model spectrum,
# by multiplication with optimal theta
model_spectrum_scaled = dadi.Inference.optimally_scaled_sfs(model_spectrum, sfs2d)

# make extraplating function from `split_asym_mig` model function
func_ex = dadi.Numerics.make_extrap_log_func(split_asym_mig)

# these were the optimal parameters for the `split_asym_mig` model
popt_asym_mig = [0.92091098,  2.2386172 ,  1.24468148,  0.08414741,  0.49491818]

# get (unscaled) expected SFS for optimal `split_asym_mig` model
model_spectrum_asym_mig = func_ex(popt_asym_mig, ns, pts_l)

# optimally scale spectrum
model_spectrum_asym_mig = dadi.Inference.optimally_scaled_sfs(model_spectrum_asym_mig, sfs2d)

dadi.Plotting.plot_2d_comp_multinom(data=model_spectrum_asym_mig.fold(), model=model_spectrum_scaled.fold(), \
                                    vmin=1, title=['asym_mig', 'IM'], pop_ids=['ery', 'par'])


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
divergence in isolation model
Can you write Python code for it?

def split_nomig(params, ns, pts):
    """
    params = (nu1,nu2,T)
    ns = (n1,n2)

    Split into two populations of specifed size, no migration.

    nu1: Size of population 1 after split.
    nu2: Size of population 2 after split.
    T: Time in the past of split (in units of 2*Na generations)
    n1,n2: Sample sizes of resulting Spectrum
    pts: Number of grid points to use in integration.
    """
    nu1,nu2,T = params

    xx = dadi.Numerics.default_grid(pts)

    phi = dadi.PhiManip.phi_1D(xx)
    phi = dadi.PhiManip.phi_1D_to_2D(xx, phi)

    phi = dadi.Integration.two_pops(phi, xx, T, nu1, nu2, m12=0, m21=0)

    fs = dadi.Spectrum.from_phi(phi, ns, (xx,xx))
    return fs


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
parameter interpretation
Can you write Python code for it?

theta = dadi.Inference.optimal_sfs_scaling(model_mig_iso, sfs2d)
theta

mu = 3e-9
L = sfs2d.data.sum() # this sums over all entries in the spectrum, including masked ones, i. e. also contains invariable sites
print "The total sequence length for the 2D spectrum is {0:,}.".format(int(L))
N_ref = theta/L/mu/4
print "The effective size of the ancestral population of ery and par (in number of diploid individuals) implied by this theta is: {0:,}.".format(int(N_ref))

nu1, nu2, Tc, m1, m2, Ti = popt_mig_iso

popt = popt_mig_iso
print "The ancestral population split apart {0:,} generations ago.".format(int((Tc+Ti)*2*N_ref)),
print "Immediately after the split the ERY population changed to a size of {0:,} and the PAR population to {1:,}.".format(int(nu1*N_ref), int(nu2*N_ref)),
print "Since the split of the ancestral population, PAR received 1 individual from ERY every {0:.2f} generations,".format(1.0/(m1/2*nu2)),
print "while ERY received 1 PAR individual every {0:.2f} generations.".format(1.0/(m2/2*nu1)),
print "Put another way: The PAR population contained a constant proportion of {0:.2e} of new immigrant alleles each generation".format(m1/2/N_ref/nu2),
print "and the ERY population contained a constant proportion of {0:.2e} of new immigrant alleles each generation.".format(m2/2/N_ref/nu1),
print "ERY and PAR remained in contact for {0:,} generations.".format(int(Tc*2*N_ref)),
print "{0:,} generations ago gene flow between ERY and PAR had ceased.".format(int(Ti*2*N_ref))


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_2D_models_synthesis.ipynb".
The first task is:
plot residuals
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
$\theta$ and implied $N_{ref}$
Can you write Python code for it?

theta = dadi.Inference.optimal_sfs_scaling(model_folded, fs_ery)

print 'The optimal value of theta is {0:.3f}.'.format(theta)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
According to the number of segregating sites, this spectrum should have good power to distinguish between alternative demographic models (see Adams2004). However, the noise in the data is extreme, as can be seen below, which might compromise this power and maybe even lead to false inferences.
Plot the data
Can you write Python code for it?

%pdoc dadi.Plotting.plot_1d_fs

pylab.rcParams['figure.figsize'] = [12.0, 10.0]

dadi.Plotting.plot_1d_fs(fs_ery, show=False)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
Both parameters seem to be correlated. With the available data, it may not be possible to distinguish between a moderate reduction in population size a long time ago (topright in the above figure) and a drastic reduction in population size a short time ago (bottomleft in the above figure).
Bottleneck then exponential growth
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
Built-in 1D models
Can you write Python code for it?

# show modules within dadi

dir(dadi)

dir(dadi.Demographics1D)

# show the source of the 'Demographics1D' method

%psource dadi.Demographics1D


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
End Digression
Can you write Python code for it?

model_folded * theta

pylab.semilogy(model_folded * theta, "bo-", label='SNM')
pylab.plot(fs_ery, "ro-", label='ery')
pylab.legend()

%psource dadi.Plotting.plot_1d_comp_Poisson

# compare model prediction and data visually with dadi function

dadi.Plotting.plot_1d_comp_multinom(model_folded[:19], fs_ery[:19], residual='linear')


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
I am going to fold manually now.
Can you write Python code for it?

# reverse spectrum and add to itself

model_fold = model.data + model.data[::-1]
model_fold

# discard all count classes >n/2

model_fold = model_fold[:19]
model_fold


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
I have copied the unfolded SFS into the current directory.
Can you write Python code for it?

% ll

% cat ERY.unfolded.sfs

# load unfolded spectrum

fs_ery_unfolded_by_ANGSD = dadi.Spectrum.from_file('ERY.unfolded.sfs')
fs_ery_unfolded_by_ANGSD

# fold unfolded spectrum

fs_ery_unfolded_by_Angsd_folded = fs_ery_unfolded_by_ANGSD.fold()
fs_ery_unfolded_by_Angsd_folded

# plot the two spectra

pylab.rcParams['figure.figsize'] = [12.0, 10.0]

pylab.plot(fs_ery_folded_by_Angsd, 'ro-', label='folded by ANGSD')
pylab.plot(fs_ery_unfolded_by_Angsd_folded, 'bo-', label='folded by DADI')
pylab.legend()
pylab.savefig('ery_fold_comp.png')

%psource dadi.Plotting.plot_1d_comp_Poisson

dadi.Plotting.plot_1d_comp_Poisson(fs_ery_folded_by_Angsd[:19], fs_ery_unfolded_by_Angsd_folded[:19], \
                                   residual='linear')


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
I now have connections to 11 engines. I started the engines with ipcluster start -n 11 & in the terminal.
Can you write Python code for it?

# create load balanced view of the engines

lbview = cl.load_balanced_view()

lbview.block

# create direct view of all engines

dview = cl[:]


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
Load unfolded SFS
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
Now, the reported sample size is correct and we have a Spectrum object that dadi can handle correctly.
To fold or not to fold by ANGSD
Does estimating an unfolded spectrum with ANGSD and then folding yield a sensible folded SFS when the sites are not polarised with respect to an ancestral allele but with respect to the reference allele? Matteo Fumagalli thinks that this is sensible.
Load SFS folded by ANGSD
Can you write Python code for it?

% cat dadiExercises/ERY.FOLDED.sfs.dadi_format

# load the spectrum that was created from folded SAF's

fs_ery_folded_by_Angsd = dadi.Spectrum.from_file('dadiExercises/ERY.FOLDED.sfs.dadi_format')
fs_ery_folded_by_Angsd

# extract unmasked entries of the SFS

m = fs_ery_folded_by_Angsd.mask
fs_ery_folded_by_Angsd[m == False]


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
See this thread on the dadi forum.

Exponential growth model
Can you write Python code for it?

# show the source of the 'Demographics1D' method

%psource dadi.Demographics1D.growth

# create link to function that specifies a simple growth or decline model

func = dadi.Demographics1D.growth

# create extrapolating version of the function

func_ex = dadi.Numerics.make_extrap_log_func(func)

# set lower and upper bounds to nu and T

upper_bound = [100, 3]
lower_bound = [1e-2, 0]

# set starting value

p0 = [1, 1] # corresponds to constant population size

%pdoc dadi.Misc.perturb_params

# perturb starting values by up to a factor of 2

p0 = dadi.Misc.perturb_params(p0, fold=1, upper_bound=upper_bound, lower_bound=lower_bound)

p0

%psource dadi.Inference.optimize_log

# run optimisation of paramters

popt = dadi.Inference.optimize_log(p0=p0, data=fs_ery, model_func=func_ex, pts=pts_l, \
                                   lower_bound=lower_bound, upper_bound=upper_bound, \
                                   verbose=0, maxiter=100, full_output=False)

popt


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
The folded expected spectrum is correct. Also, see figure 4.5 in Wakeley2009.
How to fold an unfolded spectrum
Can you write Python code for it?

# fold the unfolded model

model_folded = model.fold()
#model_folded = model_folded[:(ns[0]+1)]
model_folded.pop_ids = ['ery'] # be sure to give an array, not a scalar string
model_folded

ll_model_folded = dadi.Inference.ll_multinom(model_folded, fs_ery)

print 'The log composite likelihood of the observed ery spectrum given a standard neutral model is {0:.3f}.'.format(ll_model_folded)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
The lower log-likelihood for the last set of parameters inferred indicates that the optimisation got trapped in a local minimum in the last run of the optimisation.
What the majority of the parameter sets seem to indicate is that at about time $0.007 \times 2 N_{ref}$ generations in the past the ancestral population started to shrink exponentially, reaching a population size of about $0.14 \times N_{ref}$ at present.
Can you write Python code for it?

print 'The model suggests that exponential decline in population size started {0:.0f} generations ago.'.format(popt[0][1] * 2 * N_ref)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
The lower plot is for the scaled Poisson residuals.
$$
residuals = (model - data)/\sqrt{model}
$$
The model is the expected counts in each frequency class. If these counts are Poisson distributed, then their variance is equal to their expectation. The differences between model and data are therefore scaled by the expected standard deviation of the model counts.
The observed counts deviate by up to 30 standard deviations from the model!
What could be done about this?
The greatest deviations are seen for the first two frequency classes, the ones that should provide the greatest amount of information (Fu1994) for theta and therefore probably also other parameters. Toni has suggested that the doubleton class is inflated due to "miscalling" heterozygotes as homozygotes. When they contain a singleton they will be "called" as homozygote and therefore contribute to the doubleton count. This is aggravated by the fact that the sequenced individuals are all male which only possess one X chromosome. The X chromosome is the fourth largest of the 9 chromosomes of these grasshoppers (8 autosomes + X) (see Gosalvez1988, fig. 2). That is, about 1/9th of the sequenced RAD loci are haploid but ANGSD assumes all loci to be diploid. The genotype likelihoods it calculates are all referring to diploid genotypes.
I think one potential reason for the extreme deviations is that the genotype likelihoods are generally biased toward homozygote genotypes (i. e. also for autosomal loci) due to PCR duplicates (see eq. 1 in Nielsen2012). So, one potential improvement would be to remove PCR duplicates.
Another potential improvement could be found by subsampling 8/9th to 8/10th of the contigs in the SAF files and estimating an SFS from these. Given enough subsamples, one should eventually be found that maximally excludes loci from the X chromosome. This subsample is expected to produce the least squared deviations from an expected SFS under the standard neutral model. However, one could argue that this attempt to exclude problematic loci...
Can you write Python code for it?

fs_ery

# make copy of spectrum array
data_abc = fs_ery.data.copy()

# resize the array to the unfolded length

data_abc.resize((37,))
data_abc

fs_ery_ext = dadi.Spectrum(data_abc)
fs_ery_ext

fs_ery_ext.fold()

fs_ery_ext = fs_ery_ext.fold()
fs_ery_ext.pop_ids = ['ery']
fs_ery_ext

fs_ery_ext.sample_sizes


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
The sizes of the residuals (scaled by the Poisson standard deviations) indicate that the two versions of the folded SFS of ery are significantly different.
Now, what does the parallelus data say?
Can you write Python code for it?

% ll dadiExercises/

% cat dadiExercises/PAR.FOLDED.sfs.dadi_format

# load the spectrum folded by ANGSD

fs_par_folded_by_Angsd = dadi.Spectrum.from_file('dadiExercises/PAR.FOLDED.sfs.dadi_format')
fs_par_folded_by_Angsd

% cat PAR.unfolded.sfs

# load spectrum that has been created from unfolded SAF's

fs_par_unfolded_by_Angsd = dadi.Spectrum.from_file('PAR.unfolded.sfs')
fs_par_unfolded_by_Angsd

fs_par_unfolded_by_Angsd_folded = fs_par_unfolded_by_Angsd.fold()
fs_par_unfolded_by_Angsd_folded

dadi.Plotting.plot_1d_comp_Poisson(fs_par_folded_by_Angsd[:19], fs_par_unfolded_by_Angsd_folded[:19], \
                                   residual='linear')

#pylab.subplot(2,1,1)
pylab.plot(fs_par_folded_by_Angsd[:19], 'ro-', label='folded by ANGSD')

#pylab.subplot(2,1,2)
pylab.plot(fs_par_unfolded_by_Angsd_folded, 'bo-', label='folded by DADI')
pylab.legend()
pylab.savefig('par_fold_comp.png')


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
There is no convergence of parameters estimates. The parameter combinations stand for vastly different demographic scenarios. Most seem to suggest a population increase (up to 100 times the ancestral population size), followed by exponential decrease to about the ancestral population size.
Three epochs
Can you write Python code for it?

func = dadi.Demographics1D.three_epoch
func_ex = dadi.Numerics.make_extrap_log_func(func)

%psource dadi.Demographics1D.three_epoch


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
This model does not converge on a set of parameter values.
Can you write Python code for it?

nu = [i[0] for i in popt]
nu

T = [i[1] for i in popt]
T

pylab.rcParams['font.size'] = 14.0

pylab.loglog(nu, T, 'bo')
pylab.xlabel(r'$\nu$')
pylab.ylabel('T')


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
This model has three parameters. $\nu_B$ is the ratio of the population size (with respect to the ancestral population size $N_{ref}$) after the first stepwise change at time T in the past. The population is then asumed to undergo exponential growth/decline to a ratio of population size $\nu_F$ at present.
Can you write Python code for it?

func = dadi.Demographics1D.bottlegrowth
func_ex = dadi.Numerics.make_extrap_log_func(func)

upper_bound = [100, 100, 3]
lower_bound = [1e-3, 1e-3, 0]
pts_l = [40, 50, 60]

def bottleneck_growth(x):
    p0 = [1, 1, 1] # corresponds to constant population size

        # perturb initial parameter values randomly by up to 2 * fold
    p0 = dadi.Misc.perturb_params(p0, fold=1.5, \
                                  upper_bound=upper_bound, lower_bound=lower_bound)

    # run optimisation
    popt = dadi.Inference.optimize_log(p0, fs_ery_ext, func_ex, pts_l, \
                                      upper_bound=upper_bound, lower_bound=lower_bound,
                                      verbose=0, maxiter=10)

    return popt

%%time

popt = map(bottleneck_growth, range(10))

popt


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
This model specifies a stepwise change in population size some time ago. It assumes that the population size has stayed constant since the change.
Can you write Python code for it?

func = dadi.Demographics1D.two_epoch
func_ex = dadi.Numerics.make_extrap_log_func(func)

upper_bound = [10, 3]
lower_bound = [1e-3, 0]
pts_l = [40, 50, 60]

def stepwise_pop_change(x):
    # set initial values
    p0 = [1, 1]

    # perturb initial parameter values randomly by up to 2 * fold
    p0 = dadi.Misc.perturb_params(p0, fold=1.5, \
                                  upper_bound=upper_bound, lower_bound=lower_bound)

    # run optimisation
    popt = dadi.Inference.optimize_log(p0, fs_ery_ext, func_ex, pts_l, \
                                      upper_bound=upper_bound, lower_bound=lower_bound,
                                      verbose=0, maxiter=10)

    return popt

stepwise_pop_change(1)

stepwise_pop_change(1)

popt = map(stepwise_pop_change, range(10))

popt


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
This model tries to estimate three parameters. The populations is expected to undergo a stepwise population size change (bottleneck) at time TF + TB. At time TF it is expected to recover immediately to the current population size.
Can you write Python code for it?

upper_bound = [100, 100, 3, 3]
lower_bound = [1e-3, 1e-3, 0, 0]
pts_l = [40, 50, 60]

def opt_three_epochs(x):
    p0 = [1, 1, 1, 1] # corresponds to constant population size

        # perturb initial parameter values randomly by up to 2 * fold
    p0 = dadi.Misc.perturb_params(p0, fold=1.5, \
                                  upper_bound=upper_bound, lower_bound=lower_bound)

    # run optimisation
    popt = dadi.Inference.optimize_log(p0, fs_ery_ext, func_ex, pts_l, \
                                      upper_bound=upper_bound, lower_bound=lower_bound,
                                      verbose=0, maxiter=10)

    return popt

%%time

popt = map(opt_three_epochs, range(10))

popt


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
Unfortunately, parallelisation is not as straightforward as it should be.
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
What is the log likelihood of the model given these two different parameter sets?
Can you write Python code for it?

model_one = func_ex(popt[0], ns, pts_l)
ll_model_one = dadi.Inference.ll_multinom(model_one, fs_ery_ext)
ll_model_one

model_two = func_ex(popt[9], ns, pts_l)
ll_model_two = dadi.Inference.ll_multinom(model_two, fs_ery_ext)
ll_model_two


I am working on the file "Data_analysis/SNP-indel-calling/dadi/1D_models.ipynb".
The first task is:
What's happening in the 18th count class?
Can you write Python code for it?

# get the source of the fold method, which is part of the Spectrum object

%psource dadi.Spectrum.fold

# get the docstring of the Spectrum object

%pdoc dadi.Spectrum

# retrieve the spectrum array from the Spectrum object

model.data


I am working on the file "Data_analysis/SNP-indel-calling/dadi/DEDUPLICATED/deduplicated_spectra.ipynb".
The first task is:
This doesn't look good!
Can you write Python code for it?

# import 1D spectrum of ery
fs_ery = dadi.Spectrum.from_file('ERY.unfolded.sfs.dadi')

fs_par = dadi.Spectrum.from_file('PAR.unfolded.sfs.dadi')

import pylab

%matplotlib inline

pylab.rcParams['figure.figsize'] = [14, 12]

pylab.plot(fs_par.fold(), 'go-', label='par')
pylab.plot(fs_ery.fold(), 'rs-', label='ery')
pylab.xlabel('minor allele frequency')
pylab.ylabel('SNP count')
pylab.legend()


I am working on the file "Data_analysis/SNP-indel-calling/dadi/DEDUPLICATED/deduplicated_spectra.ipynb".
The first task is:
Unfortunately, the Big Data set from standard RAD sequencing does not provide enough information for genotype likelihoods to estimate allele frequency spectra from. It may be that the coverage filtering just leaves too few sites that can provide enough signal to overcome the noise in the data.
read data in at least 15 ind
Using the sites that had at least 3x coverage in at least 10 individuals across the two populations, I have run the SAF calculation again with -minInd 15, that is requiring at least 15 (of 18) individuals with read data (i. e. at least 1x coverage).
Can you write Python code for it?

# import 1D spectrum of ery
fs_ery = dadi.Spectrum.from_file('../../ANGSD/DEDUPLICATED/afs/ery/ery.unfolded.15.sfs.dadi')

fs_par = dadi.Spectrum.from_file('../../ANGSD/DEDUPLICATED/afs/par/par.unfolded.15.sfs.dadi')

fs_ery.S()

pylab.plot(fs_par.fold(), 'go-', label='par')
pylab.plot(fs_ery.fold(), 'rs-', label='ery')
pylab.xlabel('minor allele frequency')
pylab.ylabel('SNP count')
pylab.legend()


I am working on the file "Data_analysis/SNP-indel-calling/dadi/DEDUPLICATED/deduplicated_spectra.ipynb".
The first task is:
Unfortunately, this is still completely unusable. Only few sites are inferred variable.
I think that the coverage is too low for ANGSD to be able to destinguish alleles from sequencing errors. The ML SFS is therefore probably very unstable.
After more stringent coverage filtering
I have applied a slightly more stringent minimum coverage filtering: instead of at least 1x in 15 ind. I have applied 3x in 10 ind. This retained 368,764 sites on 9,119 contigs. Other than that, I have applied exactly the same filtering as above. Before HWE filtering there were 414,122 sites from originally 85,488,084 sites (0.48%). There were only 125,875 sites if I had filtered for 3x coverage in at least 15 individuals.
Can you write Python code for it?

% ll ../../ANGSD/DEDUPLICATED/afs/ery

# import 1D spectrum of ery
fs_ery = dadi.Spectrum.from_file('../../ANGSD/DEDUPLICATED/afs/ery/ery.unfolded.sfs.dadi')

fs_par = dadi.Spectrum.from_file('../../ANGSD/DEDUPLICATED/afs/par/par.unfolded.sfs.dadi')

fs_ery.S()

pylab.plot(fs_par.fold(), 'go-', label='par')
pylab.plot(fs_ery.fold(), 'rs-', label='ery')
pylab.xlabel('minor allele frequency')
pylab.ylabel('SNP count')
pylab.legend()


I am working on the file "Data_analysis/SNP-indel-calling/dadi/DEDUPLICATED/deduplicated_spectra.ipynb".
The first task is:
What is going on here?
After new coverage filtering
I have done new excess and minimum coverage filtering. See excess_coverage_filter.py and minimum_coverage_filter.py. This resulted in many more positions, contigs and reads retained, i. e. more info to determine SFS.
I have created SAF's with a minimum of 9 individuals with read data and used the accellerated version of the EM algorithm in realSFS.
Can you write Python code for it?

# load dadi module

import sys

sys.path.insert(0, '/home/claudius/Downloads/dadi')

import dadi

# import 1D spectrum of ery
fs_ery = dadi.Spectrum.from_file('Ery.unfolded.sfs.dadi')

fs_par = dadi.Spectrum.from_file('Par.unfolded.sfs.dadi')

fs_ery.S()

import pylab

%matplotlib inline

pylab.rcParams['figure.figsize'] = [12, 10]

pylab.plot(fs_par.fold(), 'go-', label='par')
pylab.plot(fs_ery.fold(), 'rs-', label='ery')
pylab.xlabel('minor allele frequency')
pylab.ylabel('SNP count')
pylab.legend()


I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_1D_models.ipynb".
The first task is:
ERY
Can you write Python code for it?

%%px --local

# set up global variables on engines required for run_dadi function call

ns = fs_ery.sample_sizes # both populations have the same sample size
# setting the smallest grid size slightly larger than the largest population sample size (36)
pts_l = [40, 50, 60]
dadi_opt_func = dadi.Inference.optimize_log_fmin # uses Nelder-Mead algorithm
sfs = fs_ery
perturb = True
fold = 2 # perturb randomly up to `fold` times 2-fold
maxiter = 100 # run a maximum of 300 iterations
verbose = 0
full_output = True # need to have full output to get the warnflags (see below)
outname = "MODIFIED_SPECTRA/OUT_1D_models/expgrowth" # set file name stub for opt. result files
fixed_params = None

# set starting values for perturbation
p0 = [1, 1]

#ar_ery = lbview.map(run_dadi, repeat(p0, 10))

ar_ery.get()

# set starting values for perturbation
p0 = [0.1, 0.1]

#ar_ery = lbview.map(run_dadi, repeat(p0, 10))

# set starting values for perturbation
p0 = [10, 0.1]

#ar_ery1 = lbview.map(run_dadi, repeat(p0, 10))

ar_ery = []

for filename in glob("OUT_1D_models/expgrowth*dill"):
    ar_ery.append(dill.load(open(filename)))

get_flag_count(ar_ery, NM=True)

import pandas as pd

l = 2*len(p0)+1

# show all parameter combinations
returned = [flatten(out)[:l] for out in ar_ery]

df = pd.DataFrame(data=returned, \
                  columns=['nu1_0', 'T_0', 'nu1_opt', 'T_opt', '-logL'])

df.sort_values(by='-logL', ascending=True)

%%px --local

# set lower and upper bounds to nu1 and T
upper_bound = [1e4, 6] # increasing upper bound of time parameter
lower_bound = [1e-4, 0]

# set starting values for perturbation
p0 = [2, 1]

#ar_ery1 = lbview.map(run_dadi, repeat(p0, 10))

ar_ery = []

for filename in glob("OUT_1D_models/expgrowth*dill"):
    ar_ery.append(dill.load(open(filename)))

l = 2*len(p0)+1

# show all parameter combinations
returned = [flatten(out)[:l] for out in ar_ery]

df = pd.DataFrame(data=returned, \
                  columns=['nu1_0', 'T_0', 'nu1_opt', 'T_opt', '-logL'])

df.sort_values(by='-logL',...


%%px --local

# set up global variables on engines required for run_dadi function call

ns = fs_ery.sample_sizes # both populations have the same sample size
# setting the smallest grid size slightly larger than the largest population sample size (36)
pts_l = [40, 50, 60]
dadi_opt_func = dadi.Inference.optimize_log_fmin # uses Nelder-Mead algorithm
sfs = fs_ery
perturb = True
fold = 2 # perturb randomly up to `fold` times 2-fold
maxiter = 100 # run a maximum of 100 iterations
verbose = 0
full_output = True # need to have full output to get the warnflags (see below)
outname = "MODIFIED_SPECTRA/OUT_1D_models/ERY_twoEpoch" # set file name stub for opt. result files
fixed_params = None

# set starting values for perturbation
p0 = [1, 1]

ar_ery = lbview.map(run_dadi, repeat(p0, 10))

ar_ery = []

for filename in glob("OUT_1D_models/ERY_twoEpoch*dill"):
    ar_ery.append(dill.load(open(filename)))

l = 2*len(p0)+1

# show all parameter combinations
returned = [flatten(out)[:l] for out in ar_ery]

df = pd.DataFrame(data=returned, \
                  columns=['nu1_0', 'T_0', 'nu1_opt', 'T_opt', '-logL'])

df.sort_values(by='-logL', ascending=True)

%%px --local

# set lower and upper bounds to nu1 and T
upper_bound = [1e4, 8]
lower_bound = [1e-4, 0]

# set starting values for perturbation
p0 = [1, 1]

ar_ery = lbview.map(run_dadi, repeat(p0, 10))

ar_ery = []

for filename in glob("OUT_1D_models/ERY_twoEpoch*dill"):
    ar_ery.append(dill.load(open(filename)))

l = 2*len(p0)+1

# show all parameter combinations
returned = [flatten(out)[:l] for out in ar_ery]

df = pd.DataFrame(data=returned, \
                  columns=['nu1_0', 'T_0', 'nu1_opt', 'T_opt', '-logL'])

df.sort_values(by='-logL', ascending=True)


%%px --local

# set up global variables on engines required for run_dadi function call

ns = fs_ery.sample_sizes # both populations have the same sample size
# setting the smallest grid size slightly larger than the largest population sample size (36)
pts_l = [40, 50, 60]
dadi_opt_func = dadi.Inference.optimize_log_fmin # uses Nelder-Mead algorithm
sfs = fs_ery
perturb = True
fold = 2 # perturb randomly up to `fold` times 2-fold
maxiter = 100 # run a maximum of 100 iterations
verbose = 0
full_output = True # need to have full output to get the warnflags (see below)
outname = "MODIFIED_SPECTRA/OUT_1D_models/ERY_threeEpoch" # set file name stub for opt. result files
fixed_params = None

# set starting values for perturbation
p0 = [10, 1, 1, 1]

#ar_ery = lbview.map(run_dadi, repeat(p0, 20))

ar_ery = []

for filename in glob("OUT_1D_models/ERY_threeEpoch*dill"):
    ar_ery.append(dill.load(open(filename)))

l = 2*len(p0)+1

# show all parameter combinations
returned = [flatten(out)[:l] for out in ar_ery]

df = pd.DataFrame(data=returned, \
                  columns=['nuB_0', 'nuF_0', 'TB_0', 'TF_0', 'nuB_opt', 'nuF_opt', 'TB_opt', 'TF_opt', '-logL'])

df.sort_values(by='-logL', ascending=True)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_1D_models.ipynb".
The first task is:
Exponential growth
Can you write Python code for it?

dadi.Demographics1D.growth?

%%px --local

func_ex = dadi.Numerics.make_extrap_log_func(dadi.Demographics1D.growth)

%%px --local

# set lower and upper bounds to nu1 and T
upper_bound = [1e4, 4]
lower_bound = [1e-4, 0]


I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_1D_models.ipynb".
The first task is:
I think such a high effective population size is not realistic.
PAR
Can you write Python code for it?

%%px --local

# set up global variables on engines required for run_dadi function call

ns = fs_ery.sample_sizes # both populations have the same sample size
# setting the smallest grid size slightly larger than the largest population sample size (36)
pts_l = [40, 50, 60]
dadi_opt_func = dadi.Inference.optimize_log_fmin # uses Nelder-Mead algorithm
sfs = fs_par
perturb = True
fold = 2 # perturb randomly up to `fold` times 2-fold
maxiter = 100 # run a maximum of 100 iterations
verbose = 0
full_output = True # need to have full output to get the warnflags (see below)
outname = "MODIFIED_SPECTRA/OUT_1D_models/PAR_bottlegrowth" # set file name stub for opt. result files
fixed_params = None

%%px --local

# set lower and upper bounds to nu1 and T
upper_bound = [1e4, 1e4, 6]
lower_bound = [1e-4, 1e-4, 0]

# set starting values for perturbation
p0 = [1, 1, 1]

#ar_par = lbview.map(run_dadi, repeat(p0, 10))

ar_par = []

for filename in glob("OUT_1D_models/PAR_bottlegrowth*dill"):
    ar_par.append(dill.load(open(filename)))

l = 2*len(p0)+1

# show all parameter combinations
returned = [flatten(out)[:l] for out in ar_par]

df = pd.DataFrame(data=returned, \
                  columns=['nuB_0', 'nuF_0', 'T_0', 'nuB_opt', 'nuF_opt', 'T_opt', '-logL'])

df.sort_values(by='-logL', ascending=True)

cl[:]['maxiter'] = 100

# set starting values for perturbation
p0 = [100, 2, 1.2]

ar_par = lbview.map(run_dadi, repeat(p0, 10))

ar_par = []

for filename in glob("OUT_1D_models/PAR_bottlegrowth*dill"):
    ar_par.append(dill.load(open(filename)))

l = 2*len(p0)+1

# show all parameter combinations
returned = [flatten(out)[:l] for out in ar_par]

df = pd.DataFrame(data=returned, \
                  columns=['nuB_0', 'nuF_0', 'T_0', 'nuB_opt', 'nuF_opt', 'T_opt', '-logL'])

df.sort_values(by='-logL', ascending=True).head(10)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_1D_models.ipynb".
The first task is:
PAR
Can you write Python code for it?

%%px --local

# set up global variables on engines required for run_dadi function call

ns = fs_ery.sample_sizes # both populations have the same sample size
# setting the smallest grid size slightly larger than the largest population sample size (36)
pts_l = [40, 50, 60]
dadi_opt_func = dadi.Inference.optimize_log_fmin # uses Nelder-Mead algorithm
sfs = fs_par
perturb = True
fold = 2 # perturb randomly up to `fold` times 2-fold
maxiter = 100 # run a maximum of 100 iterations
verbose = 0
full_output = True # need to have full output to get the warnflags (see below)
outname = "MODIFIED_SPECTRA/OUT_1D_models/PAR_threeEpoch" # set file name stub for opt. result files
fixed_params = None

# set starting values for perturbation
p0 = [100, 2, 1, 1]

#ar_par = lbview.map(run_dadi, repeat(p0, 20))

ar_par = []

for filename in glob("OUT_1D_models/PAR_threeEpoch*dill"):
    ar_par.append(dill.load(open(filename)))

l = 2*len(p0)+1

# show all parameter combinations
returned = [flatten(out)[:l] for out in ar_par]

df = pd.DataFrame(data=returned, \
                  columns=['nuB_0', 'nuF_0', 'TB_0', 'TF_0', 'nuB_opt', 'nuF_opt', 'TB_opt', 'TF_opt', '-logL'])

df.sort_values(by='-logL', ascending=True)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_1D_models.ipynb".
The first task is:
This looks like convergence.
Interpretation
Can you write Python code for it?

popt = np.array( df.sort_values(by='-logL', ascending=True).iloc[0, 3:6] )
popt

# calculate best-fit model spectrum

model_spectrum = func_ex(popt, ns, pts_l)

theta = dadi.Inference.optimal_sfs_scaling(model_spectrum, fs_ery)

mu = 3e-9
L = fs_ery.data.sum()
print "The optimal value of theta per site for the ancestral population is {0:.4f}.".format(theta/L)

Nref = theta/L/mu/4
Nref

print "At time {0:,} generations ago, the ERY population size instantaneously increased by almost 55-fold (to {1:,}).".format(int(popt[2]*2*Nref), int(popt[0]*Nref))


I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_spectra_synthesis.ipynb".
The first task is:
Note the asymmetry in the model spectrum for frequency classes [1, x] and [x, 1].
Exponential size change
So far, the split coincided with a stepwise population size change. This model adds a proportion $s$ of the ancestral population size that is allocated to each daughter population after the split and exponential population size change afterwards.
None of the parameter combinations has higher likelihood (12888.96) than the best-fit parameter combinations of the previous model (lowest neg. logL 12,359). Also the three best-fit parameter combinations are very close to the upper limit of parameter bound that I set for the time parameter $T$ and the population size ratio for PAR implies a current effective population size of several millions (something I think is unlikely given the disperal rate and the patchiness of the habitat of Chorthippus parallelus).
Linear size change
This model changes the population size growth or decline from exponential to linear.
There is no good convergence, the inferred $s$ is very high (close to 1 or 0) and the inferred T is close to the upper boundary that I set. Furthermore, the best -logL (12516.95) is still higher than the best -logL achieved with the asymmetric migration model (12,359).
Ancient migration
Can you write Python code for it?

def split_asym_mig_iso(params, ns, pts):
    """
    params = (nu1,nu2,Tc,m1,m2,Ti)
    ns = (n1,n2)

    Split into two populations of specifed size, with potentially asymmetric migration
    for a time Tc followed by complete isolation until present.

    nu1: population size ratio of population 1 after split (with respect to Na)
    nu2: population size ratio of population 2 after split (with respect to Na)
    Tc: Time of gene flow after split (in units of 2*Na generations)
    m1: Migration rate from ery into par (in units of 2*Na ind per generation)
    m2: Migration rate from par into ery (in units of 2*Na ind per generation)
    Ti: Time of isolation after cessation of gene flow
    The split lies Tc+Ti * 2Na generations in the past.
    n1,n2: Sample sizes of resulting Spectrum
    pts: Number of grid points to use in integration.
    """
    nu1,nu2,Tc,m1,m2,Ti = params

    xx = dadi.Numerics.default_grid(pts)

    phi = dadi.PhiManip.phi_1D(xx)

    # split
    phi = dadi.PhiManip.phi_1D_to_2D(xx, phi)

    # divergence with potentially asymmetric migration
    phi = dadi.Integration.two_pops(phi, xx, Tc, nu1, nu2, m12=m2, m21=m1)

    # divergence without gene flow
    phi = dadi.Integration.two_pops(phi, xx, Ti, nu1, nu2, m12=0, m21=0)

    fs = dadi.Spectrum.from_phi(phi, ns, (xx,xx))
    return fs

ar_split_asym_mig_iso = []

for filename in glob("OUT_2D_models/split_asym_mig_iso*dill"):
    ar_split_asym_mig_iso.append(dill.load(open(filename)))

l = 2*6+1

success = [flatten(out)[:l] for out in ar_split_asym_mig_iso]

df = pd.DataFrame(data=success, \
                  columns=['nu1_0','nu2_0', 'Tc_0', 'm1_0', 'm2_0', 'Ti_0', 'nu1_opt', 'nu2_opt', 'Tc_opt', 'm1_opt', 'm2_opt', 'Ti_opt', '-logL'])

df.sort_values(by='-logL', ascending=True).head(10)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_spectra_synthesis.ipynb".
The first task is:
The 2D SFS was corrected with a $p$ of 0.35, the optimal $p$ when fit to a simple divergence in isolation model with parameters $\nu_1=1$, $\nu_2=1$ and $T=1$.
Can you write Python code for it?

# number of SNP's in the spectrum
sfs2d.S()

ns = sfs2d.sample_sizes # both populations have the same sample size
# setting the smallest grid size slightly larger than the largest population sample size (36)
pts_l = [40, 50, 60]


I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_spectra_synthesis.ipynb".
The first task is:
The PAR spectrum was corrected with a $p$ of 0.36 and the ERY spectrum with a $p$ of 0.34, the optimal values of $p$ for each spectrum.
exponential growth model
For ERY as well as for PAR the time parameter is hitting the upper parameter boundary (8) with the corrected spectrum. With uncorrected spectrum, optimisation was successful for ERY but not for PAR, because of lack of convergence.
two epoch model
The two epoch model could not be fit to the ERY nor the PAR spectrum with Ludovic's correction applied due hitting of upper boundary on the time parameter (8). A 2-epoch model could also not be fit to the ERY nor PAR with the uncorrected spectra due to hitting the lower and upper boundary on the time parameter, respectively.
bottleneck with exponential growth model
With the uncorrected spectra, the optimal parameter values for ERY suggested an ancient population size of >21 million, which is unrealistic. For PAR the optimisation did not converge.
With the corrected spectra, the optimal parameter values imply an effective population size of >14 million for erythropus (before decline). I think such a high effective population size is unrealistic. The fitting to the corrected spectrum of PAR even estimated an initial increase to 36 million. Such high effective population sizes are obviously unrealistic. I must therefore regard the fitting of this model to the corrected spectra as failed.
three epoch model
With the corrected spectrum for ERY, this model unambiguously (i. e. in all optimisation runs) infers a long time with around 10-fold increased population size followed by a stepwise decrease to almost the size of the ancient population. The inference from the uncorrected spectrum ranged from a 186-fold increase to a reduction to 1/3 of the ancient population size.
Similar to the ERY spectrum, the inference from the corrected PAR spectrum showed a consistent population size increase (ranging from 38-fold to a ridiculous 7772-fold). This was followed by a very recent radical reduction in population size to only...
Can you write Python code for it?

# load spectrum modified with Ludovic's correction, p=0.35

sfs2d = dadi.Spectrum.from_file("EryPar_modified.2dsfs")

dadi.Plotting.plot_single_2d_sfs(sfs2d, vmin=1, cmap=pylab.cm.jet)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_spectra_synthesis.ipynb".
The first task is:
The absolute values of the two time parameters are striking. This model infers a very ancient origin of ERY and PAR and a very recent cessation of gene flow. Obviously, this model is an abstraction of the certainly more complicated history of the two subspecies, as one would expect given that there have been at least three ice ages since the inferred origin of ERY and PAR.
The following table compares the inferred parameters for the ancient migration model from the uncorrected and corrected spectrum in their absolute units:
| parameter      | uncorrected | corrected |
| :----------:   | ----------- | --------- |
| $N_{a}$        | 125,977     | 113,522   |
| $N_{ery}$      | 363,349     | 389,237   |
| $N_{par}$      | 886,821     | 950,409   |
| $T_c$          | 1,748,505   | 1,525,640 |
| $p_{ery->par}$ | 2.21e-08    | 2.14e-08  |
| $p_{par->ery}$ | 3.27e-07    | 2.60e-07  |
| $T_i$          | 43,628      | 26,970    |
| D              | 1044        | 406       |
$N_x$ have unit individuals, $T_x$ has unit generations, $p_x$ are proportions of new immigrant alleles per generation and D is two times the log likelihood ratio between the best fit parameters of this model with the hitherto best performing asymmetric migration model.
Note that with the uncorrected spectrum, I could not achieve satisfactory convergence of parameter estimations. The parameters inferred with the corrected spectrum are therefore more reliable. The inferred population sizes for ERY and PAR aren't very different between corrected and uncorrected spectra and so is the inferred time of contact ($T_c$). The migration rates are slightly reduced with the corrected spectrum and the time since cessation of gene flow ($T_i$) is inferred to be even more recent than with the uncorrected spectrum.
The addition of $T_i$ improves the model by 203 log likelihood units as compared to the model without a period of complete isolation (split_asym_mig above). If the better fit is not just due to fitting noise and bias in the data, then this means that...
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_spectra_synthesis.ipynb".
The first task is:
The difference in the frequency class [1,1] between the observed (corrected) and model spectrum has dramatically decreased after introducing migration. The introduction of migration has improved the log likelihood of the model by 2982 units. So there is a strong signal of gene flow in the 2D spectrum.
Influence of the degree of correction
I have also fit the above two models to spectra corrected with p=0.30 and p=0.39, both derived by using a slightly different null model for the optimisation of p, that is with $T=2$ and $T=0.5$ respectively (see notebook).
Since the degree of correction does not seem to have a strong influence on parameter estimation (see notebook), I am going to use only the spectrum corrected with $p=0.35$ in the following.
divergence with asymmetric migration
Can you write Python code for it?

def split_asym_mig(params, ns, pts):
    """
    params = (nu1,nu2,T,m1,m2)
    ns = (n1,n2)

    Split into two populations of specifed size, with potentially asymmetric migration.

    nu1: population size ratio of population 1 after split (with respect to Na)
    nu2: population size ratio of population 2 after split (with respect to Na)
    T: Time in the past of split (in units of 2*Na generations)
    m1: Migration rate from ery into par (in units of 2*Na ind per generation)
    m2: Migration rate from par into ery (in units of 2*Na ind per generation)
    n1,n2: Sample sizes of resulting Spectrum
    pts: Number of grid points to use in integration.
    """
    nu1,nu2,T,m1,m2 = params

    xx = dadi.Numerics.default_grid(pts)

    phi = dadi.PhiManip.phi_1D(xx)

    # split
    phi = dadi.PhiManip.phi_1D_to_2D(xx, phi)

    # divergence with potentially asymmetric migration
    phi = dadi.Integration.two_pops(phi, xx, T, nu1, nu2, m12=m2, m21=m1)

    fs = dadi.Spectrum.from_phi(phi, ns, (xx,xx))
    return fs

ar_split_asym_mig = []

for filename in glob("OUT_2D_models/split_asym_mig_[0-9]*dill"):
    ar_split_asym_mig.append(dill.load(open(filename)))

l = 2*5+1

returned = [flatten(out)[:l] for out in ar_split_asym_mig]

df = pd.DataFrame(data=returned, \
                  columns=['nu1_0','nu2_0', 'T_0', 'm1_0', 'm2_0', 'nu1_opt', 'nu2_opt', 'T_opt', 'm1_opt', 'm2_opt', '-logL'])

df.sort_values(by='-logL', ascending=True)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_spectra_synthesis.ipynb".
The first task is:
The following table compares the best-fit parameter values from the uncorrected and corrected spectrum:
| parameter | uncorrected | corrected |
| :-------: | ----------- | --------- |
| $N_a$     |  688,875    | 631,166   |
| $\nu_1$   |  438,776    | 481,209   |
| $\nu_2$   |  847,045    | 963,542   |
| T         |  400,188    | 381,147   |
| -logL     |  20387      | 15791     |
The values are translated to absolute units, i. e. $\nu_x$ in individuals and T in generations.
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_spectra_synthesis.ipynb".
The first task is:
The following table compares the inferred parameters for the asymmetric migration model from the uncorrected and corrected spectrum in their absolute units:
| parameter      | uncorrected | corrected |
| :----------:   | ----------- | --------- |
| $N_{a}$        | 410,678     | 255,715   |
| $N_{ery}$      | 378,198     | 397,740   |
| $N_{par}$      | 919,352     | 975,012   |
| T              | 1,022,329   | 1,198,692 |
| $p_{ery->par}$ | 4.58e-08    | 3.91e-08  |
| $p_{par->ery}$ | 6.54e-07    | 4.37e-07  |
| -logL          | 18104       | 12359     |
$N_x$ have unit individuals, T has unit generations, $p_x$ are proportions of new immigrant alleles per generation.
There is marked difference in the inferred size of the ancestral population between corrected and uncorrected spectra. The inferred time of the split in number of generations for the unmodified spectrum was 1,022,329. Not a dramatic difference from the one inferred with this corrected spectrum, but still 176,000 years difference.
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_spectra_synthesis.ipynb".
The first task is:
The following table compares the parameter estimates for the split-migration model derived with the uncorrected and the corrected spectrum in their absolute units:
| parameter      | uncorrected | corrected   |
| :-------:      | ----------- | ---------   |
| $N_a$          |  468,295    |  308,029    |
| $N_{ERY}$      |  465,572    |  488,151    |
| $N_{PAR}$      |  827,069    |  881,645    |
| T (gen.)       |  864,128    |  1,063,520  |
| $p_{par->ery}$ |  2.69e-07   |  2.16e-07   |
| $p_{ery->par}$ |  1.52e-07   |  1.20e-07   |
| -logL          |  18574      |  12809      |
$N_x$ have unit individuals, T has unit generations, $p_x$ are proportions of new immigrant alleles per generation.
There is a marked difference in the inferred ancestral population size ($N_a$ eq. to $N_{ref}$), which affects all other parameters. The estimated contemporary population sizes for ERY and PAR are slightly higher for the corrected spectrum, but not much so. The inferred divergence time is 1/4 higher with the corrected spectrum. The inferred migration rates (as proportion of new immigrant individuals per generation) are both slightly smaller with the corrected spectrum, but not dramatically.
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_spectra_synthesis.ipynb".
The first task is:
The scaling of the neutral model spectrum towards the ery or par spectrum is almost identical.
Can you write Python code for it?

pylab.plot(snm_ery, 'bs-', label="snm")
pylab.plot(fs_ery, 'ro-', label="ery", linewidth=2.0, markersize=10)
pylab.plot(fs_par, 'g^-', label="par", linewidth=2.0, markersize=10)

pylab.grid()
pylab.legend()
pylab.xlabel("minor allele frequency")
pylab.ylabel("number of SNP's")
pylab.title("corrected spectra")


I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_spectra_synthesis.ipynb".
The first task is:
This is good convergence.
The best-fit ancient migration model had a log likelihood of -12156. So only one log likelihood unit worse.
Can you write Python code for it?

ll_s = 12156.061309
ll_c = 12155.538032
D = 2 * (ll_s - ll_c)
D

# calculate p-value for Chi-square dist.
# the weights specify a weighted sum of chi^2 distributions with 0 and 1 d.o.f
# this is because Tsc is 0 in the split_asym_mig_iso model and at the boundary of the parameter space
p = dadi.Godambe.sum_chi2_ppf(D, weights=(0.5, 0.5))
p


I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_spectra_synthesis.ipynb".
The first task is:
This is how convergence looks like.
Interpretation
Can you write Python code for it?

# get best-fit parameter combination
popt = np.array(df.sort_values(by='-logL', ascending=True).iloc[0, 3:6])

# create extrapolating version of the model function
func_ex = dadi.Numerics.make_extrap_log_func(split_nomig)

# get unscaled, best-fit model spectrum
model = func_ex(popt, ns, pts_l)

# get logL of best-fit model
ll_model = dadi.Inference.ll_multinom(model, sfs2d)

# get theta
theta = dadi.Inference.optimal_sfs_scaling(model, sfs2d)

# get total sequence length in spectrum
L = sfs2d.data.sum()

# get theta per site
print "The optimal value of theta per site for the ancestral population is {0:.4f}.".format(theta/L)

mu = 3e-9 # assumed per generation per site mutation rate

# get ancestral population size
N_ref = theta/L/mu/4

print "The effective size of the ancestral population of ERY and PAR (in number of diploid individuals) implied by this theta is:\n {0:,}.".format(int(N_ref))

print "The ancestral population of ERY and PAR split apart {0:,} generations ago.".format(int(popt[2]*2*N_ref)),
print "Since then until present the ERY population had a constant size of {0:,} and the PAR population of {1:,}.".format(int(popt[0]*N_ref), int(popt[1]*N_ref))


I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_spectra_synthesis.ipynb".
The first task is:
This is quite a small effective population size for an insect.
Can you write Python code for it?

nu1, nu2, Tc, m1, m2, Ti = popt

print "The ancestral population split apart {0:,} generations ago.".format(int((Tc+Ti)*2*N_ref)),
print "Immediately after the split the ERY population changed to a size of {0:,} and the PAR population to {1:,}.".format(int(nu1*N_ref), int(nu2*N_ref)),
print "Since the split of the ancestral population, PAR received 1 individual from ERY every {0:.2f} generations,".format(1.0/(m1/2*nu2)),
print "while ERY received 1 PAR individual every {0:.2f} generations.".format(1.0/(m2/2*nu1)),
print "Put another way: The PAR population contained a constant proportion of {0:.2e} of new immigrant alleles each generation".format(m1/2/N_ref/nu2),
print "and the ERY population contained a constant proportion of {0:.2e} of new immigrant alleles each generation.".format(m2/2/N_ref/nu1),
print "ERY and PAR remained in contact for {0:,} generations.".format(int(Tc*2*N_ref)),
print "{0:,} generations ago gene flow between ERY and PAR had ceased.".format(int(Ti*2*N_ref))


I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_spectra_synthesis.ipynb".
The first task is:
This looks like convergence. Allowing for asymmetric migration rates improves the model fit by 450 logL units and is therefore highly significant.
Interpretation
Can you write Python code for it?

popt = np.array(df.sort_values(by='-logL', ascending=True).iloc[0, 5:10])

func_ex = dadi.Numerics.make_extrap_log_func(split_asym_mig)

# calculate best-fit model spectrum
model = func_ex(popt, ns, pts_l)

theta = dadi.Inference.optimal_sfs_scaling(model, sfs2d)
L = sfs2d.data.sum()
print "The optimal value of theta per site for the ancestral population is {0:.5f}.".format(theta/L)

mu = 3e-9
N_ref = theta/L/mu/4
print "The effective size of the ancestral population of ERY and PAR (in number of diploid individuals) implied by this theta is:\n {0:,}.".format(int(N_ref))

nu1, nu2, T, m1, m2 = popt

print "The ancestral population split apart {0:,} generations ago.".format(int(T*2*N_ref)),
print "Immediately after the split the ERY population changed to a size of {0:,} and the PAR population to {1:,}.".format(int(nu1*N_ref), int(nu2*N_ref)),
print "Since the split of the ancestral population, PAR received 1 individual from ERY every {0:.2f} generations,".format(1.0/(m1/2*nu2)),
print "while ERY received 1 PAR individual every {0:.2f} generations.".format(1.0/(m2/2*nu1)),
print "Put another way: The PAR population contained a constant proportion of {0:.2e} of new immigrant alleles each generation".format(m1/2/N_ref/nu2),
print "and the ERY population contained a constant proportion of {0:.2e} of new immigrant alleles each generation.".format(m2/2/N_ref/nu1)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/dadiExercises/First_Steps_with_dadi.ipynb".
The first task is:
$\delta$a$\delta$i is detecting that the spectrum is folded (as given in the input file), but it is also automatically masking the 0th and 18th count category. This is a not a good behaviour.
Can you write Python code for it?

# number of segregating sites

fs_ery.data[1:].sum()


I am working on the file "Data_analysis/SNP-indel-calling/dadi/dadiExercises/First_Steps_with_dadi.ipynb".
The first task is:
$\delta$a$\delta$i is issuing the following message when executing the above command:
WARNING:Spectrum_mod:Creating Spectrum with data_folded = True, but mask is not True for all entries which are nonsensical for a folded Spectrum.
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/dadiExercises/First_Steps_with_dadi.ipynb".
The first task is:
$\delta$a$\delta$i seems to do the right thing. Note, that the estimate of Tajima's D from this marginal spectrum of ery is slightly different from the estimate derived from the folded 1D spectrum of ery (see /data3/claudius/Big_Data/ANGSD/SFS/SFS.Rmd). The folded 1D spectrum resulted in a Tajima's D estimate of $\sim$0.05, i. e. a difference of almost 0.1. Again, the 2D spectrum is based on only those sites for which there were at least 9 individiuals with data in both populations, whereas the 1D folded spectrum of ery included all sites for which there were 9 ery individuals with data (see line 1571 onwards in assembly.sh).
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/dadiExercises/First_Steps_with_dadi.ipynb".
The first task is:
The sum over the marginalised 1D spectra should be the same as the sum over the 2D spectrum !
Can you write Python code for it?

# from dadi's marginalise function:
fs_ery.data

sfs2d = EryPar_unfolded_2dsfs.copy()

# this should get the marginal spectrum for ery
ery_mar = [pylab.sum(sfs2d.data[i]) for i in range(0, len(sfs2d))]
ery_mar

# this should get the marginal spectrum for ery and then take the sum over it
sum([pylab.sum(sfs2d.data[i]) for i in range(0, len(sfs2d))])

# look what happens if I include masking
sum([pylab.sum(sfs2d[i]) for i in range(0, len(sfs2d))])

fs_ery.data - ery_mar


I am working on the file "Data_analysis/SNP-indel-calling/dadi/dadiExercises/First_Steps_with_dadi.ipynb".
The first task is:
Apart from the incorrect small sample size correction by $\delta$a$\delta$i in case of folded spectra ($n$ refers to sampled sequences, not individuals), Gillespie's formula leads to a much higher estimate of $\pi_{site}$ than Wakeley's. Why is that?
Can you write Python code for it?

# with correct small sample size correction
2 * n / (2* n-1.0) * 2 * pylab.sum(fs_ery * p*(1-p)) / pylab.sum(fs_ery.data)

# Calculating pi with the formula from Gillespie:

n = 18
p = pylab.arange(0, n+1)/float(n)
p = p/2 # with a folded spectrum, we are summing over minor allele freqs only
pi_Gillespie = 2*n / (2*n-1.0) * 2 * pylab.sum(fs_ery * p*(1-p)) / pylab.sum(fs_ery.data)
pi_Gillespie

pi_Wakeley - pi_Gillespie


I am working on the file "Data_analysis/SNP-indel-calling/dadi/dadiExercises/First_Steps_with_dadi.ipynb".
The first task is:
Curve_fit is another function that can be used for optimization.
Can you write Python code for it?

?curve_fit

def model(i, theta):
    """
    i: indpendent variable, here minor SNP frequency classes
    theta: scaled population mutation rate parameter [scalar]

    returns a numpy array
    """
    n = len(i)
    delta = pylab.where(i == n-i, 1, 0)
    return theta * 1/i + 1/(n-i) / (1 + delta)

i = pylab.arange(1, fs_ery.size)

popt, pcov = curve_fit(model, i, fs_ery.data[1:])

# optimal theta
print popt

perr = pylab.sqrt(pcov)
perr

print str(int(popt[0] - 1.96*perr[0])) + ' < ' + str(int(popt[0])) + ' < ' + str(int(popt[0] + 1.96*perr[0]))

popt, pcov = curve_fit(model, i, fs_par.data[1:])
perr = pylab.sqrt(pcov)
print str(int(popt[0] - 1.96*perr[0])) + ' < ' + str(int(popt[0])) + ' < ' + str(int(popt[0] + 1.96*perr[0]))


I am working on the file "Data_analysis/SNP-indel-calling/dadi/dadiExercises/First_Steps_with_dadi.ipynb".
The first task is:
Here is how to flatten an array of arrays with list comprehension:
Can you write Python code for it?

mask = [[True], [False] * 18, [True] * 18]
print mask

print [elem for a in mask for elem in a]


I am working on the file "Data_analysis/SNP-indel-calling/dadi/dadiExercises/First_Steps_with_dadi.ipynb".
The first task is:
I am not sure whether these standard errors (perr) are correct. It may be that it is assumed that errors are normally distributed, which they are not exactly in this case. They should be close to Poisson distributed (see Fu1995), which should be fairly similar to normal with such high expected values as here.
If the standard errors are correct, then the large overlap of the 95% confidence intervals would indicate that the data do not provide significant support for a difference in $\theta$ between par and ery.
Parametric bootstrap from the observed SFS
Can you write Python code for it?

%pwd

% ll

! cat ERY.FOLDED.sfs.dadi_format

fs_ery = dadi.Spectrum.from_file('ERY.FOLDED.sfs.dadi_format', mask_corners=False)

fs_ery

fs_ery.pop_ids = ['ery']

# get a Poisson sample from the observed spectrum

fs_ery_param_boot = fs_ery.sample()

fs_ery_param_boot

fs_ery_param_boot.data

%psource fs_ery.sample


I am working on the file "Data_analysis/SNP-indel-calling/dadi/dadiExercises/First_Steps_with_dadi.ipynb".
The first task is:
I do not understand this warning from $\delta$a$\delta$i. The 18-count category is sensical for a folded spectrum with even sample size, so should not be masked. Anyway, I do not understand why $\delta$a$\delta$i is so reluctant to keep all positions, including the non-variable one.
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/dadiExercises/First_Steps_with_dadi.ipynb".
The first task is:
I have cloned the $\delta$a$\delta$i repository into '/home/claudius/Downloads/dadi' and have compiled the code. Now I need to add that directory to the PYTHONPATH variable:
Can you write Python code for it?

sys.path.insert(0, '/home/claudius/Downloads/dadi')

sys.path


I am working on the file "Data_analysis/SNP-indel-calling/dadi/dadiExercises/First_Steps_with_dadi.ipynb".
The first task is:
I have next added a masking line to the input file, setting it to '1' for the first position, i. e. the 0-count category.
Can you write Python code for it?

fs_ery = dadi.Spectrum.from_file('ERY.FOLDED.sfs.dadi_format', mask_corners=False)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/dadiExercises/First_Steps_with_dadi.ipynb".
The first task is:
I think for now it would be best to import unfolded spectra from realSFS and fold them if necessary in dadi.
Can you write Python code for it?

fs_par = dadi.Spectrum.from_file('PAR.FOLDED.sfs.dadi_format')

pylab.plot(fs_ery, 'r', label='ery')
pylab.plot(fs_par, 'g', label='par')
pylab.legend()


I am working on the file "Data_analysis/SNP-indel-calling/dadi/dadiExercises/First_Steps_with_dadi.ipynb".
The first task is:
It would be good to get the 5% and 95% quantiles from the bootstrap samples of each frequency class and add those intervals to the plot of the observed frequency spectrum and the fitted neutral spectrum. This would require to find a quantile function and to find out how to add lines to a plot with matplotlib.
It would also be good to use the predicted counts from the neutral model above with the fitted $\theta$ as parameters for the bootstrap with sample() and add 95% confidence intervals to the predicted neutral SFS. I have done this in R instead (see /data3/claudius/Big_Data/ANGSD/SFS/SFS.Rmd)

Using unfolded spectra
I edited the 2D SFS created for estimating $F_{ST}$ by realSFS. I have convinced myself that realSFS outputs a flattened 2D matrix as expected by $\delta$a$\delta$i's Spectrum.from_file function (see section 3.1 of the manual with my comments). Note, that in the manual, "samples" stands for number of allele copies, so that the correct specification of dimensions for this 2D unfolded SFS of 18 diploid individuals in each of 2 populations is 37 x 37.
Can you write Python code for it?

# read in the flattened 2D SFS
EryPar_unfolded_2dsfs = dadi.Spectrum.from_file('EryPar.unfolded.2dsfs.dadi_format', mask_corners=True)

# check dimension
len(EryPar_unfolded_2dsfs[0,])

EryPar_unfolded_2dsfs.sample_sizes

# add population labels
EryPar_unfolded_2dsfs.pop_ids = ["ery", "par"]

EryPar_unfolded_2dsfs.pop_ids


I am working on the file "Data_analysis/SNP-indel-calling/dadi/dadiExercises/First_Steps_with_dadi.ipynb".
The first task is:
Note, that these marginalised 1D SFS's are not identical to the 1D SFS estimated directly with realSFS. This is because, for the estimation of the 2D SFS, realSFS has only taken sites that had data from at least 9 individuals in each population (see assembly.sh, lines 1423 onwards).
The SFS's of par and ery had conspicuous shape differences. It would therefore be good to plot them to see, whether the above commands have done the correct thing.
Can you write Python code for it?

# plot 1D spectra for each population
pylab.plot(fs_par, 'g', label="par")
pylab.plot(fs_ery, 'r', label="ery")
pylab.legend()


I am working on the file "Data_analysis/SNP-indel-calling/dadi/dadiExercises/First_Steps_with_dadi.ipynb".
The first task is:
Now, I should be able to import $\delta$a$\delta$i
Can you write Python code for it?

import dadi

dir(dadi)

import pylab

%matplotlib inline

x = pylab.linspace(0, 4*pylab.pi, 1000)

pylab.plot(x, pylab.sin(x), '-r')

%%sh
# this allows me to execute a shell command

ls


I am working on the file "Data_analysis/SNP-indel-calling/dadi/dadiExercises/First_Steps_with_dadi.ipynb".
The first task is:
Plotting
Can you write Python code for it?

dadi.Plotting.plot_single_2d_sfs(EryPar_unfolded_2dsfs, vmin=1)

dadi.Plotting.plot_single_2d_sfs(EryPar_folded_2dsfs, vmin=1)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/dadiExercises/First_Steps_with_dadi.ipynb".
The first task is:
Single population statistics
$\pi$
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/dadiExercises/First_Steps_with_dadi.ipynb".
The first task is:
So, 1.6 million sites went into the ery spectrum.
Can you write Python code for it?

# pi per site
n / (n-1.0) * 2 * pylab.sum(fs_ery * p*(1-p)) / pylab.sum(fs_ery.data)