I am working on the file "Data_analysis/SNP-indel-calling/dadi/01_dadi_1D_exp_growth.ipynb".
The first task is:
Now, define a new run_dadi function that pickles output to file:
Can you write Python code for it?

def run_dadi(p_init): # for the function to be called with map, it needs to have one input variable
    """
    p_init: initial parameter values to run optimisation from
    """
    if perturb == True:
        p_init = dadi.Misc.perturb_params(p_init, fold=fold,
                                      upper_bound=upper_bound, lower_bound=lower_bound)
        # note upper_bound and lower_bound variables are expected to be in the namespace of each engine
    # run optimisation of paramters
    popt = dadi_opt_func(p0=p_init, data=sfs, model_func=func_ex, pts=pts_l, \
                                   lower_bound=lower_bound, upper_bound=upper_bound, \
                                   verbose=verbose, maxiter=maxiter, full_output=full_output)
    # pickle to file
    import dill
    name = outname[:] # make copy of file name stub!
    for p in p_init:
        name += "_%.4f" % (p)
    with open(name + ".dill", "w") as fh:
        dill.dump((p_init, popt), fh)

    return p_init, popt

%ll

ar_ery = lbview.map(run_dadi, product(p0_nu, p0_T), block=False, order=True)

ar_ery.progress


I am working on the file "Data_analysis/SNP-indel-calling/dadi/01_dadi_1D_exp_growth.ipynb".
The first task is:
OK, that looks promising. Two quite different parameter combinations have lead to the same optimal parameter combination.
Can you write Python code for it?

p_init = np.array( df.sort_values(by='-logL').iloc[0, 2:4] )
p_init

ar_ery = lbview.map(run_dadi, repeat(p_init, 20), block=False, order=False)

ar_ery.elapsed

get_flag_count(ar_ery, NM=True)

i = 4 # where to find flag, 6 for BFGS, 4 for Nelder-Mead

successfull_popt_ery = [flatten(out)[:5] for out in ar_ery if out[1][i] == 0]

df = pd.DataFrame(data=successfull_popt_ery, \
                  columns=['nu_0', 'T_0', 'nu_opt', 'T_opt', '-logL'])

df.sort_values(by='-logL', ascending=True) # smaller is better


I am working on the file "Data_analysis/SNP-indel-calling/dadi/01_dadi_1D_exp_growth.ipynb".
The first task is:
OK. Three successful optimisations.
Can you write Python code for it?

i = 4 # where to find flag, 6 for BFGS, 4 for Nelder-Mead

successfull_popt_ery = [flatten(out)[:5] for out in ar_ery if out[1][i] == 0]

df = pd.DataFrame(data=successfull_popt_ery, \
                  columns=['nu_0', 'T_0', 'nu_opt', 'T_opt', '-logL'])

df.sort_values(by='-logL', ascending=True) # smaller is better


I am working on the file "Data_analysis/SNP-indel-calling/dadi/01_dadi_1D_exp_growth.ipynb".
The first task is:
Par
Since starting optimisations from a broad range of initial values did not lead to useful results with the ery spectrum, I am going to use the same strategy that led to a successful optimisation with ery: creating initial parameter values from a perturbation of parameters for a neutral model.
Can you write Python code for it?

%%px

# set up global variables on engines required for run_dadi function call

dadi_opt_func = dadi.Inference.optimize_log_fmin # uses Nelder-Mead algorithm
sfs = fs_par # use PAR spectrum
perturb = True
fold = 3
maxiter = 50 # run a maximum of 50 iterations
verbose = 0
full_output = True # need to have full output to get the warnflags (see below)
outname = "OUT_exp_growth_model/PAR_perturb"

p_init = [1, 1]

ar_par = lbview.map(run_dadi, repeat(p_init, 20), block=False, order=False)

ar_par.elapsed/60

ar_par.progress


I am working on the file "Data_analysis/SNP-indel-calling/dadi/01_dadi_1D_exp_growth.ipynb".
The first task is:
So, if I could believe the optimal parameter combination found here to be the global maximum, it would imply that the ancestral populations of the parallelus sample have undergone an exponential decline to about 1/1000th of the ancient population size, $N_{ref}$ (i. e. very strong bottleneck) and this happend in the very recent past, i. e. only $2.3 \times 10^{-5} \times 2N_{ref}$ generations ago.
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/01_dadi_1D_exp_growth.ipynb".
The first task is:
That is hitting the upper limit for the time parameter.
Maybe a sweep through the parameter space could help to find an optimum.
Can you write Python code for it?

%%px

# set up global variables on engines required for run_dadi function call

dadi_opt_func = dadi.Inference.optimize_log # uses BFGS algorithm
sfs = fs_par # use PAR spectrum
perturb = False
fold = 3
maxiter = 10 # run a maximum of 10 iterations
verbose = 0
full_output = True # need to have full output to get the warnflags (see below)
outname = "OUT_exp_growth_model/PAR"

ar_par = lbview.map(run_dadi, product(p0_nu, p0_T), block=False, order=False)

ar_par.progress

ar_par.elapsed/60


I am working on the file "Data_analysis/SNP-indel-calling/dadi/01_dadi_1D_exp_growth.ipynb".
The first task is:
The parameter combination $\nu = 0.001049$ and $T = 0.000023$ has only slightly smaller log likelihood. This parameter combination corresponds to a much more recent and much more drastic population reduction. I think a likelihood ratio test would be required to see whether the data can distinguish between these two scenarios.
Maybe I should do a longer run, starting from the best-fit parameter sets found here and setting maxiter to something ridiculously high, so that it is guaranteed that the ML values will be found within gtol deviation.
Can you write Python code for it?

p0 = df.sort_values(by='e_log ll', ascending=False).iloc[0, 2:4]
p_init = np.array(p0)
p_init

# create direct view of engines
dview = cl[:]

# define function to run dadi with perturbed starting values
# note the high maxiter
def run_dadi_on_ery(p_init): # for the function to be called with map, it needs to have one input variable
    """
    p_init: initial parameter values to run optimisation from
    """
    # perturb starting values by up to a factor of 2 times 'fold'
    p0 = dadi.Misc.perturb_params(p_init, fold=2, upper_bound=upper_bound, lower_bound=lower_bound)
    # run optimisation of paramters
    popt = dadi.Inference.optimize_log(p0=p0, data=fs_ery, model_func=func_ex, pts=pts_l, \
                                   lower_bound=lower_bound, upper_bound=upper_bound, \
                                   verbose=0, maxiter=1e9, full_output=False)
    return popt # returns a tuple with starting values and optimal nu and T

%pdoc run_dadi_on_ery

# run optimisation with perturbed initial values on all 11 engines
fine_opt_res = dview.apply_async( run_dadi_on_ery, p_init )

# get running time
fine_opt_res.wall_time


I am working on the file "Data_analysis/SNP-indel-calling/dadi/01_dadi_1D_exp_growth.ipynb".
The first task is:
The parameter combination with the highest likelihood is not found several times from different starting values. This does not give me much confidence in the maximum likelihood parameter set.
Can you write Python code for it?

plt.rcParams['font.size'] = 20.0
plt.figure(figsize=[12.0, 10.0])
plt.scatter(np.log(nu), np.log(T), c=ll, s=75, alpha=0.5)
plt.xlabel(r'log $\nu$')
plt.ylabel("log T")
plt.title("log likelihoods for parameter combinations")
plt.grid()
plt.colorbar()

# define function to run dadi with perturbed starting values
# note the high maxiter
def run_dadi_on_par(p_init): # for the function to be called with map, it needs to have one input variable
    """
    p_init: initial parameter values to run optimisation from
    """
    # perturb starting values by up to a factor of 2 times 'fold'
    p0 = dadi.Misc.perturb_params(p_init, fold=2, upper_bound=upper_bound, lower_bound=lower_bound)
    # run optimisation of paramters
    popt = dadi.Inference.optimize_log(p0=p0, data=fs_par, model_func=func_ex, pts=pts_l, \
                                   lower_bound=lower_bound, upper_bound=upper_bound, \
                                   verbose=0, maxiter=1e9, full_output=False)
    return popt # returns optimal nu and T

p_init

# get parameter combination with highest likelihood as starting values
p_init = np.array( df.sort_values(by='e_log ll', ascending=False).head().iloc[0, 2:4] )

# run optimisation with perturbed initial values on all 11 engines
fine_opt_res = dview.apply_async( run_dadi_on_par, p_init )

fine_opt_res.wall_time

ll = get_ll.map(fine_opt_res.get())

nu_opt = [p[0] for p in fine_opt_res.get()]
T_opt = [p[1] for p in fine_opt_res.get()]

df = pd.DataFrame({
        'a_nu' : nu_opt,
        'b_T' : T_opt,
        'c_ll' : ll
    })
df


I am working on the file "Data_analysis/SNP-indel-calling/dadi/01_dadi_1D_exp_growth.ipynb".
The first task is:
There are still three outlier results. Maybe I need to use a different optimsiation function.
residual plots
I would like to comparisons between model predicted and observed SFS for a couple of optimisation results.
Can you write Python code for it?

p_init

# calculate the best-fit model SFS given the paramter combination with the highest likelihood
expected_sfs = func_ex(p_init, ns, pts_l)
expected_sfs.fold()

plt.rcParams['figure.figsize'] = [12.0, 10.0]
print(p_init)
dadi.Plotting.plot_1d_comp_multinom(expected_sfs.fold()[:19], fs_ery[:19], residual='linear')


I am working on the file "Data_analysis/SNP-indel-calling/dadi/01_dadi_1D_exp_growth.ipynb".
The first task is:
These starting values cover a very wide range of possible parameter combinations. This should be better than merely randomly perturbing a single parameter combination a certain number of times (and running the optimisation from those perturbed starting values).
ery
run optimisation on 1D SFS of
Can you write Python code for it?

# create load balanced view of engines

lbview = cl.load_balanced_view()

%pdoc dadi.Inference.optimize_log


I am working on the file "Data_analysis/SNP-indel-calling/dadi/01_dadi_1D_exp_growth.ipynb".
The first task is:
This lets the optimisations run with the faster BFGS algorithm and stop after a maximum of 10 iterations. This is intended for a broad sweep across the parameter space. Also note that this sets the optimisations to fit the ery spectrum.
Can you write Python code for it?

%%px

# set lower and upper bounds to nu and T
upper_bound = [1e4, 4]
lower_bound = [1e-4, 0]

p0_nu = np.logspace(-3, 3, base=10.0, num=15)
p0_T = np.logspace(-3, np.log10(3), base=10, num=15)

from itertools import product

# DO NOT RUN
#ar_ery = lbview.map(run_dadi, product(p0_nu, p0_T), block=False, order=True)

ar_ery.progress

ar_ery.get()[0]


I am working on the file "Data_analysis/SNP-indel-calling/dadi/01_dadi_1D_exp_growth.ipynb".
The first task is:
This runs definitely too long! run_dadi_on_par spends most of its time on the last three tasks. A maxiter of 100, therefore, does not seem to be limiting the search. Set to lower maxiter next time. Maybe 50.
Recommended parameter boundaries
Ryan Gutenkunst writes on the dadi forum:

The parameter bounds are primarily about avoiding really slow fits. Calculation with high migration rates and times or small population sizes is much slower, and the optimizer can explore extreme values before settling down. So typically we set migrations rates to be bounded [0, 20ish], times to be [0, 5ish], population sizes to be [1e-3, 1e6ish].

Maybe I should set the time parameter to something above 0.
Can you write Python code for it?

%psource dadi.Inference.optimize_log

# save optimsation results to file
# can be loaded back into another IPython session with: ar = pickle.load(filehandle)
# unfortunately, I cannot save the AsyncResult object with pickle (dill can save it, but emits a
# "maximum recursion depth exceeded" error when I try to load the object from file again).
# I need to save the result as a plain list

import dill

with open("exp_growth_optim_res_PAR.pickle", "w") as fhandle:
    dill.dump(list(ar.get()), fhandle)

# get the optimized parameter values
popt = [p for i, p in enumerate(ar) if not (i+1) % 2]

# get the starting values
pstart = [p for i, p in enumerate(ar) if (i+1) % 2]

# get arrays of individual parameters from each optimisation run
nu = [p_set[0] for p_set in popt]
T = [p_set[1] for p_set in popt]
nu_start = [p_set[0][0] for p_set in pstart]
T_start = [p_set[0][1] for p_set in pstart]

# run get_ll on all sets of parameters in parallel
ll = get_ll.map(popt)

# create pandas data frame
df = pd.DataFrame({
        'a_nu_start' : nu_start,
        'b_T_start' : T_start,
        'c_nu' : nu,
        'd_T' : T,
        'e_log ll' : ll
    })
df.head()

df.sort_values(by='e_log ll', ascending=False).head(30)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/01_dadi_1D_exp_growth.ipynb".
The first task is:
This toke only 6 seconds to compute on 11 engines.
Can you write Python code for it?

# get optimised parameters
fine_opt_res.get()

# run get_ll on all sets of parameters in parallel
ll = get_ll.map(fine_opt_res.get())
ll

nu_opt = [p[0] for p in fine_opt_res.get()]
T_opt = [p[1] for p in fine_opt_res.get()]

df = pd.DataFrame({
        'a_nu' : nu_opt,
        'b_T' : T_opt,
        'c_ll' : ll
    })
df


I am working on the file "Data_analysis/SNP-indel-calling/dadi/01_dadi_1D_exp_growth.ipynb".
The first task is:
Unfortunately, this is too slow. I need to abort this by stopping the cluster of engines.
Can you write Python code for it?

import glob, dill

ar_par = []

for filename in glob.glob('OUT_exp_growth_model/PAR_[0123456789]*'):
    try:
        ar_par.append( dill.load( open(filename) ) )
    except:
        pass

get_flag_count(ar_par, NM=False)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/01_dadi_1D_exp_growth.ipynb".
The first task is:
Use unfolded ANGSD spectra and fold with dadi:
Can you write Python code for it?

%%px --local

# import 1D spectrum of ery on all engines:
fs_ery = dadi.Spectrum.from_file('ERY.unfolded.sfs').fold()

# import 1D spectrum of ery on all engines:
fs_par = dadi.Spectrum.from_file('PAR.unfolded.sfs').fold()

cl[0]['fs_ery']

%%px --local

ns = fs_ery.sample_sizes # both populations have the same sample size

fs_ery.pop_ids = ['ery']
fs_par.pop_ids = ['par']

# setting the smallest grid size slightly larger than the largest population sample size (36)
pts_l = [40, 50, 60]

%%px --local

# create link to function that specifies the model
func = dadi.Demographics1D.growth

# create extrapolating version of the model function
func_ex = dadi.Numerics.make_extrap_log_func(func)

%pdoc dadi.Demographics1D.growth

# create load balanced view of engines

lbview = cl.load_balanced_view()

def run_dadi(p_init): # for the function to be called with map, it needs to have one input variable
    """
    p_init: initial parameter values to run optimisation from
    """
    if perturb == True:
        p_init = dadi.Misc.perturb_params(p_init, fold=fold,
                                      upper_bound=upper_bound, lower_bound=lower_bound)
        # note upper_bound and lower_bound variables are expected to be in the namespace of each engine
    # run optimisation of paramters
    popt = dadi_opt_func(p0=p_init, data=sfs, model_func=func_ex, pts=pts_l, \
                                   lower_bound=lower_bound, upper_bound=upper_bound, \
                                   verbose=verbose, maxiter=maxiter, full_output=full_output)
    return p_init, popt


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
11 instead of 40 minutes.
Can you write Python code for it?

get_flag_count(ar_par_NM, NM=True)

# get log likelihood values for all parameters returned
ll = get_ll.map([p[0] for p in ar_par_NM])

# create rows for dataframe, without the parameter values on non-convergent runs
rows = [(p0[0], p0[1], xopt[0][0], xopt[0][1], logL) for p0, xopt, logL in izip(product(p0_nu, p0_T), ar_par_NM, ll) if xopt[4] == 0]

df = pd.DataFrame(data=rows, columns=['nu_0', 'T_0', 'nu_opt', 'T_opt', 'logL'])

df.sort_values(by='logL', ascending=False)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
Compared to a standard neutral model, the two epoch model with the best fitting parameter values does provide a much better fit to the data spectrum. However, it also requires two (instead of zero) parameters and the inferred parameter values cannot be trusted since they are hitting the boundary of sensible values.
Can you write Python code for it?

%%px

dadi_opt_func = dadi.Inference.optimize_log # BFGS

ar_ery_BFGS = lbview.map(run_dadi, product(p0_nu, p0_T), block=False, order=True)

ar_ery_BFGS.progress

get_flag_count(ar_ery_BFGS, NM=False)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
Dadi wants to infer a very drastic bottleneck in the very recent past. It hits the lower bound on T that I set above.
Can you write Python code for it?

%%px

# further reduce lower bound on T (on all engines)

lower_bound = [1e-4, 1e-6]

# check maxiter setting on engines

cl[:]['maxiter']

cl[:]['lower_bound']

cl[:]['dadi_opt_func']


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
Does this model provide a better fit than a standard neutral model?
Can you write Python code for it?

# create link to function that specifies a standard neutral model
snm = dadi.Demographics1D.snm
# make extrapolating function
snm_ex = dadi.Numerics.make_extrap_log_func(snm)

?snm

# calculate the best-fit model SFS given paramter combination
expected_sfs = snm_ex(0, ns, pts_l)
# calculate the log likelihood of the expected SFS given the observed SFS
# no need to fold the model, is done automatically if data is folded
dadi.Inference.ll_multinom(expected_sfs, fs_ery)

# calculate the best-fit model SFS given paramter combination
expected_sfs = func_ex(p_opt, ns, pts_l)
# calculate the log likelihood of the expected SFS given the observed SFS
# no need to fold the model, is done automatically if data is folded
dadi.Inference.ll_multinom(expected_sfs, fs_ery)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
I want to get these starting values out, since they take too long to complete.
Can you write Python code for it?

excl = [ex[0] for ex in y[:2]]
excl[0] == excl[0]

combs = [comb for comb in product(p0_nu, p0_T) if not (comb == excl[0] or comb == excl[1])]
print excl[1] in combs
print excl[0] in combs
print combs[22] in combs


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
It took more than 1 hour to run this on 11 cores!
Can you write Python code for it?

import dill

# save optimsiation results to file
with open("1D_two_epoch_opt_res_ERY.dill", "w") as fh:
    dill.dump(list(ar_ery.get()), fh) # unfortunately, dill cannot save the ar_ery object

get_flag_count(ar_ery)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
None of the runs reached convergence and returned a result.
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
Note, that Nelder-Mead is much slower.
Can you write Python code for it?

%psource dadi.Inference.optimize_log_fmin

warnflag = defaultdict(int)

for res in ar_ery:
    if res[4] == 1:
        warnflag[1] +=1
    elif res[4] == 2:
        warnflag[2] += 1
    elif res[4] == 0:
        warnflag[0] += 1
    else:
        warnflag[999] +=1

print "success", warnflag[0]
print "Maximum number of function evaluations made.", warnflag[1]
print "Maximum number of iterations reached.", warnflag[2]
print "unknown flag", warnflag[999]


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
Of the runs that did return an optimisation result, most just return the initial parameter values.
Can you write Python code for it?

?dadi.Inference.optimize_lbfgsb

?dadi.Inference.optimize_grid

?dadi.Inference.optimize_log

?dadi.Inference.optimize_log_fmin

?dadi.Inference.optimize_log_powell


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
Ok, I think the new parameter array does not include the sets that lead to such extreme running times.
Can you write Python code for it?

ar_par_NM = lbview.map(run_dadi, combs, block=False, order=True)

ar_par_NM.progress

ar_par_NM.elapsed/60


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
Only 21 of the 196 starting parameter combinations lead to a successfull optimisation. I could increase that by 28 if I allowed enough iterations. However, I don't know what to do about the 147 combinations for which the gradient didn't change. Maybe they are in a very flat part of the likelihood surface?
Can you write Python code for it?

from itertools import izip

for p_init, popt in izip(product(p0_nu, p0_T), ar_ery):
    if popt[6] == 0:
        print p_init, popt[0], popt[6]


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
Removing starting parameter combinations that lead to extremely long running times
Can you write Python code for it?

for p0, r in izip(product(p0_nu, p0_T), ar_par_NM):
    if r[4] == 2: # if run hit maxiter limit
        print p0, r[0]

x  = [(p0, xopt[0], m['completed'] - m['started']) for p0, xopt, m in izip(product(p0_nu, p0_T), ar_par_NM, ar_par_NM.metadata)]
y = sorted(x, key=lambda x: x[2], reverse=True)

for i, p in enumerate(y):
    if i > 1: break
    print "nu_0 = %.6f, T_0 = %.6f \t nu = %.6f, T = %.6f" % (p[0][0], p[0][1], p[1][0], p[1][1]), "\t", p[2]


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
See figure 3.4, p. 79, in Wakeley2009 for the distribution of the $T_{MRCA}$. For $n=36$ it has a mode close 1.2 and an expected value of 1.94. Values for $T_{MRCA}$ greater than 4 are very unlikely given a standard coalescent model, but may be more likely under models including population expansion or gene flow from another population.
Can you write Python code for it?

%%px

upper_bound = [1e3, 9]

cl[0]['upper_bound']

cl[0]['lower_bound']

ar_par_NM = lbview.map(run_dadi, product(p0_nu, p0_T), block=False, order=True)

ar_par_NM.progress

get_flag_count(ar_par_NM, NM=True)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
Still hitting the boundaries of $T$. Something does not work here.
I am not sure whether it is worth continuing.
Ryan Gutenkunst:

This indicates that dadi is having trouble fitting your data. One possibility is that the history of the population includes important events that aren’t in your models. Another possibility is that your data is biased in ways that aren’t in your models. For example, maybe your missing calls for rare alleles.

dadi forum thread on parameters hitting the boundary
The data is clearly biased or at least very noisy. However, the models that only include a single event, one event of growth or decline, might also be too simplistic. So, I think before I go on and try to further clean the data, I should try the two remaining more complex 1D models that come with dadi: bottlegrowth and three_epoch. If I cannot reliably estimate parameter values for those models, then I think it is most likely a data issue.
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
The optimisations starting from 51 initial parameter combinations did not converge before the maximum number of iterations (100) was reached.
Can you write Python code for it?

# get log likelihood values for all parameters returned
ll = get_ll.map([p[0] for p in ar_ery])

# create rows for dataframe
rows = [(p0[0], p0[1], xopt[0][0], xopt[0][1], logL) for p0, xopt, logL in izip(product(p0_nu, p0_T), ar_ery, ll) if xopt[4] == 0]

# create dataframe
df = pd.DataFrame(data=rows, columns=['nu_0', 'T_0', 'nu_opt', 'T_opt', 'logL'])

# look and see
df.sort_values(by='logL', ascending=False).head(40)

cl[0]['lower_bound']


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
The optimisations with the highest likelihood are hitting the lower bound of parameter values that I've set. The time parameter $T$ that dadi infers with the highest likelihood is too low to make sense. It needs to be multiplied by $2N_{ref}$ to get the time in generations. With any reasonable effective ancient population size, this implies a very recent event. The event is inferred to be a very strong bottleneck. I don't think that these values can possibly make any sense.
residiual plots
Can you write Python code for it?

p_opt = np.array( df.iloc[0, 2:4] )
p_opt

# calculate the best-fit model SFS given the paramter combination with the highest likelihood
expected_sfs = func_ex(p_opt, ns, pts_l)
expected_sfs.fold()

import matplotlib.pyplot as plt

%matplotlib inline

plt.rcParams['figure.figsize'] = [12.0, 10.0]
print(p_opt)
dadi.Plotting.plot_1d_comp_multinom(expected_sfs.fold()[:19], fs_ery[:19], residual='linear')


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
This has printed out the initial parameter combinations, the final parameter values and the processing for each combination. There were two jobs that have taken a very long time to complete. Note, that these are not the ones which have exceeded maxiter (see above). Neither of the two long lasting jobs is hitting the boundaries of the parameter space that can be searched.
Can you write Python code for it?

cl[:]['upper_bound']

cl[:]['lower_bound']

# get log likelihood values for all parameters returned
ll = get_ll.map([p[0] for p in ar_par_NM])

# create rows for dataframe, without the parameter values on non-convergent runs
rows = [(p0[0], p0[1], xopt[0][0], xopt[0][1], logL) for p0, xopt, logL in izip(product(p0_nu, p0_T), ar_par_NM, ll) if xopt[4] == 0]

df = pd.DataFrame(data=rows, columns=['nu_0', 'T_0', 'nu_opt', 'T_opt', 'logL'])

df.sort_values(by='logL', ascending=False)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
This looks much better. I don't want to increase maxiter even more, due to the already long running time.
Can you write Python code for it?

for p_init, popt in izip(product(p0_nu, p0_T), ar_ery):
    print p_init, popt[0], popt[4]


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
This took 17 minutes to finish.
Can you write Python code for it?

def get_flag_count(ar, NM=True)
    """
    ar: asyncresult object from BFGS or Nelder-Mead optimisation
    """
    if NM: # if ar from Nelder-Mead
        i = 4 # the warnflag is reported at index position 4 in the output array
    else: # ar from BFGS optimisation
        i = 6

    warnflag = defaultdict(int)

    for res in ar:
        if res[i] == 1:
            warnflag[1] +=1
        elif res[i] == 2:
            warnflag[2] += 1
        elif res[i] == 0:
            warnflag[0] += 1
        else:
            warnflag[999] +=1
    if NM:
        print "success", warnflag[0]
        print "Maximum number of function evaluations made.", warnflag[1]
        print "Maximum number of iterations reached.", warnflag[2]
        print "unknown flag", warnflag[999]
    else:
        print "success", warnflag[0]
        print "Maximum number of iterations exceeded.", warnflag[1]
        print "Gradient and/or function calls not changing.", warnflag[2]
        print "unknown flag", warnflag[999]

get_flag_count(ar, NM=True)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
Which initial parameter combination did not converge, i. e. for which did the maximum number of iterations (100) not suffice? These are also likely to be the ones that have run so long.
Can you write Python code for it?

for p0, xopt in izip(product(p0_nu, p0_T), ar_par_NM):
    if xopt[4] == 2:
        print p0

# get log likelihood values for all parameters returned
ll = get_ll.map([p[0] for p in ar_par_NM])

# create rows for dataframe
rows = [(p0[0], p0[1], xopt[0][0], xopt[0][1], logL) for p0, xopt, logL in izip(product(p0_nu, p0_T), ar_par_NM, ll) if xopt[4] == 0]

df = pd.DataFrame(data=rows, columns=['nu_0', 'T_0', 'nu_opt', 'T_opt', 'logL'])

df.sort_values(by='logL', ascending=False)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
check convergence
Can you write Python code for it?

# get optimised parameters (if optimisation was successfull)
popt = [p[0] for p in ar_ery if p[4] == 0]
len(popt)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
define range of initial parameters
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
get log likelihood values
Can you write Python code for it?

# create parallel function with load balancing

@lbview.parallel(block=True)
def get_ll(p):
    """
    p: parameter combination

    First, calculates the best-fit model SFS given paramter combination p.
    Then returns the log likelihood of the expected SFS given the observed SFS.
    """
    expected_sfs = func_ex(p, ns, pts_l)
    return dadi.Inference.ll_multinom(expected_sfs, sfs)
    # expected_sfs does not need to be folded, ll_multinom does that automatically
    # make sure that sfs points to right spectrum

cl[0]['sfs']

ar_ery.get()

# get log likelihood values for all parameters returned

ll = get_ll.map([p[0] for p in ar_ery])

len(ll)

# get all starting parameter combinations, corresponding optimised parameters and log likelihood values
# if the optimisation was successful:

rows = [(p0[0], p0[1], xopt[0][0], xopt[0][1], logL) for p0, xopt, logL in izip(product(p0_nu, p0_T), ar_ery, ll) if xopt[4] == 0]

len(rows)

import pandas as pd

df = pd.DataFrame(data=rows, columns=['nu_0', 'T_0', 'nu_opt', 'T_opt', 'logL'])

df.sort_values(by='logL', ascending=False)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
load 1D folded spectra
Can you write Python code for it?

%%px --local

# import 1D spectrum of ery on all engines:
fs_ery = dadi.Spectrum.from_file('dadiExercises/ERY.FOLDED.sfs.dadi_format')

# import 1D spectrum of ery on all engines:
fs_par = dadi.Spectrum.from_file('dadiExercises/PAR.FOLDED.sfs.dadi_format')

fs_par

%%px --local

ns = fs_ery.sample_sizes # both populations have the same sample size

fs_ery.pop_ids = ['ery']
fs_par.pop_ids = ['par']

# setting the smallest grid size slightly larger than the largest population sample size (36)
pts_l = [50, 60, 70]


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
maxiter still set to 10 on all engines.
Can you write Python code for it?

%%px

# increase maximum number of iterations
maxiter = 100

# run optimisation as above, but now with up to 100 iterations per optimisation

#ar_ery = lbview.map(run_dadi, product(p0_nu, p0_T), block=False, order=True)

ar_ery.progress

ar_ery.wall_time/60


I am working on the file "Data_analysis/SNP-indel-calling/dadi/02_dadi_1D_two_epoch.ipynb".
The first task is:
run optimisation
Can you write Python code for it?

from itertools import product

%%px

# set up global variables on engines required for run_dadi function call

dadi_opt_func = dadi.Inference.optimize_log # uses gradient based BFGS algorithm
sfs = fs_ery # use ERY spectrum
perturb = False
fold = 1
maxiter = 10 # run a maximum of 10 iterations
verbose = 0
full_output = True # need to have full output to get the warnflags (see below)

%px who

# run optimnisations with all combinations of starting values

#ar_ery = lbview.map(run_dadi, product(p0_nu, p0_T), block=False, order=True)

ar_ery.progress


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
13 of 20 optimisations were successful.
Can you write Python code for it?

def flatten(array):
    """
        Returns a list of flattened elements of every inner lists (or tuples)
        ****RECURSIVE****
    """
    import numpy
    res = []
    for el in array:
        if isinstance(el, (list, tuple, numpy.ndarray)):
            res.extend(flatten(el))
            continue
        res.append(el)
    return list( res )

successfull_popt = [flatten(out)[:7] for out in ar_ery if out[1][4] == 0]

import pandas as pd

df = pd.DataFrame(data=successfull_popt, columns=['nuB_0', 'nuF_0', 'T_0', 'nuB_opt', 'nuF_opt', 'T_opt', '-logL'])

df.sort_values(by='-logL', ascending=False)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
Add this type of object to list of objects to flatten in the following function.
Can you write Python code for it?

def flatten(array):
    """
        Returns a list of flattened elements of every inner lists (or tuples)
        ****RECURSIVE****
    """
    import numpy
    res = []
    for el in array:
        if isinstance(el, (list, tuple, numpy.ndarray)):
            res.extend(flatten(el))
            continue
        res.append(el)
    return list( res )

def get_ll(p):
    """
    p: parameter combination

    First, calculates the best-fit model SFS given paramter combination p.
    Then returns the log likelihood of the expected SFS given the observed SFS.
    """
    expected_sfs = func_ex(p, ns, pts_l)
    return dadi.Inference.ll_multinom(expected_sfs, sfs)
    # expected_sfs does not need to be folded, ll_multinom does that automatically
    # make sure that sfs points to right spectrum

# import 1D spectrum of ery on all engines:
fs_ery = dadi.Spectrum.from_file('dadiExercises/ERY.FOLDED.sfs.dadi_format')

# create link to function that specifies the model
func = dadi.Demographics1D.bottlegrowth

# create extrapolating version of the model function
func_ex = dadi.Numerics.make_extrap_log_func(func)

ns = fs_ery.sample_sizes # both populations have the same sample size

fs_ery.pop_ids = ['ery']

# setting the smallest grid size slightly larger than the largest population sample size (36)
pts_l = [50, 60, 70]

sfs = fs_ery

# check get_ll works

ll = get_ll(opt_out[0][1][0])
ll

# check addition of logL to list of parameters

flatten(opt_out[0])[:6] + ['aaaaa']

successfull_popt = [flatten(out)[:6]  + [get_ll(out[1][0])] for out in opt_out if out[1][4] == 0]

successfull_popt[:3]

import pandas as pd

df = pd.DataFrame(data=successfull_popt, columns=['nuB_0', 'nuF_0', 'T_0', 'nuB_opt', 'nuF_opt', 'T_opt', 'logL'])

df.sort_values(by='logL', ascending=False)

p_opt = df.sort_values(by='logL', ascending=False).iloc[0,3:6]
p_opt


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
Almost no optimisation was successfull, mostly because of lacking gradient in the likelihood function. This calls for the other optimsation algorithm that is more robust to noise in the data: Nelder-Mead.
change optimsation algorithm
Can you write Python code for it?

?dadi.Inference.optimize_log_fmin

%%px

dadi_opt_func = dadi.Inference.optimize_log_fmin # uses Nelder-Mead algorithm


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
Check what's in the namespace of the engines:
Can you write Python code for it?

%px %who

from itertools import product, izip

for i, c in enumerate(product(p0_nuB, p0_nuF, p0_T)):
    if i>10: break
    print c


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
During these optimisations, there are usually a few that run much longer than the rest. I would like to be able to interrupt those runs. The following is taken from an issue thread at the ipyparallel github repo:
Can you write Python code for it?

%%px --local

import os

engine_pids = cl[:].apply(os.getpid).get_dict()

engine_pids

import signal
# define function to interrupt an engine
def signal_engine(engine_id, sig=signal.SIGINT):
    """send a signal to a local engine"""
    pid = engine_pids[engine_id]
    os.kill(pid, sig)

# run optimnisations with all combinations of starting values

# DO NOT RUN
ar_ery = lbview.map(run_dadi, product(p0_nuB, p0_nuF, p0_T), block=False, order=True)

ar_ery.progress

for ID in cl.ids:
    signal_engine(ID)

cl.ids


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
I have done the optimisations using the Nelder-Mead optimisation algorithm that should be more robust with noisy data than gradient based algorithms (like BFGS).
check convergence
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
I have run each optimisation allowing for up to 100 iterations. This should be plenty.
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
I have written a python script called run_dadi.py, which can take many command line arguments and in the following parallel framework is the replacement for the run_dadi function from above.
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
I killed the 8 last running processes.
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
I need to extend the maximum number of iterations allowed per optimisation in order to get any results.
increase maxiter
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
I needed to kill 26 optimisation runs because they could not finish with 1 hour.
collect output
Can you write Python code for it?

pickled = glob.glob('OUT_run_dadi/PAR*pickle')
sorted(pickled[:10])

opt_out = []
for filename in glob.glob('OUT_run_dadi/PAR*pickle'):
    opt_out.append( pickle.load(open(filename)))

len(opt_out)

opt_out[0]


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
It is worrying that the optimisation from these starting values does not converge within 100 iterations.
parallelus
Can you write Python code for it?

% ll dadiExercises/

path_to_spectrum_file = "dadiExercises/PAR.FOLDED.sfs.dadi_format"
dadi_model = "dadi.Demographics1D.bottlegrowth"
upper = str(upper_bound)
lower = str(lower_bound)
dadi_opt_func = "dadi.Inference.optimize_log_fmin"
stub = "PAR_bottlegrowth_NM"
maxiter = 100

# check creation of command line

for i, p_init in enumerate(product(p0_nuB, p0_nuF, p0_T)):
    cmd = "./run_dadi.py -p %s -m %s -u '%s' -l '%s' -d %s -s %s --maxiter %s -i '%s'" \
                % (path_to_spectrum_file, dadi_model, upper, lower, dadi_opt_func, stub, str(maxiter), str(p_init))
    print cmd
    if i >= 0: break

E = executor(limit=4) # leave 4 cores unused

# this will block
for i, p_init in enumerate(product(p0_nuB, p0_nuF, p0_T)):
    cmd = "./run_dadi.py -p %s -m %s -u '%s' -l '%s' -d %s -s %s --maxiter %s -i '%s'" \
                % (path_to_spectrum_file, dadi_model, upper, lower, dadi_opt_func, stub, str(maxiter), str(p_init))
    E.err = "stderr"
    E.submit(cmd, p_init)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
Next, I need to define an executor class. This is taken from Tiago Antao's Cookbook:
Can you write Python code for it?

class executor:
    def __init__(self, limit):
        self.limit = limit
        self.ncores = multiprocessing.cpu_count()
        self.running = []
        self.finished = 0

    def submit(self, cmd, p_init):
        #self.progress()
        self.wait()
        if hasattr(self, 'out'):
            out = self.out
        else:
            out = '/dev/null'
        if hasattr(self, 'err'):
            err = self.err
        else:
            err = '/dev/null'
        if err == 'stderr':
            errSt = ''
        else:
            errSt = '2> ' + err
        proc = subprocess.Popen('%s > %s %s' % (cmd, out, errSt), shell=True)
        self.running.append(proc)
        #print "started running optimsation with " + str(p_init)
        #print "started subprocess with process id" + str(proc.pid)
        if hasattr(self, 'out'):
            del self.out
        if hasattr(self, 'err'):
            del self.err

    def wait(self, for_all=False):
        self.clean_done()
        #numWaits = 0
        if self.limit > 0 and type(self.limit) == int:
            cond = 'len(self.running) >= self.ncores - self.limit'
        elif self.limit < 0:
            cond = 'len(self.running) >= - self.limit'
        else:
            cond = 'len(self.running) >= self.ncores * self.limit'
        while eval(cond) or (for_all and len(self.running) > 0):
            time.sleep(1)
            self.clean_done() # updates self.running, removes finished jobs from the running queue
            #numWaits += 1

    def clean_done(self):
        terminated = []
        for i, p in enumerate(self.running):
            if p.poll() is not None: # None means it's still running
                terminated.append(i)
        for idx in reversed(terminated):
            del self.running[idx]
            self.finished += 1

    def progress(self):
        self.clean_done()
        #for p in self.running:
        #    if p.poll() is not None:
        #        print "subprocess with process id %d reports that...


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
Of 216 initial parameter combinations, 208 produced optimisation output file.
This parallel framework, albeit much more laborious to implement than using ipyparallel, has proven useful so far. It allows me to run many optimisations in parallel, save output of individual optimisation runs to file, abort the final long lasting runs from the command line and still be able to retrieve the information from the successful runs. This is something I could not do with ipyparallel, which forced me to always wait until all optimisation jobs had finished or else loose all the output.
collect output
The script run_dadi.py has pickled the optimsation results to individual files for each initial parameter combination. They are located in the directory OUT_run_dadi. I now want to read these output files into a single data structure.
Can you write Python code for it?

import pickle

import glob

pickled = glob.glob('OUT_run_dadi/*pickle')
sorted(pickled[:10])

opt_out = []
for filename in glob.iglob('OUT_run_dadi/*pickle'):
    opt_out.append( pickle.load(open(filename, "r")) )

len(opt_out)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
The built-in bottlegrowth model specifies an instantaneous size change $T \times 2N_{ref}$ generations in the past immediately followed by the start of exponential growth (or decline) toward the contemporary population size. The model has three parameters:
- ratio of population size after instantaneous size change with respect to the ancient population size ($N_{ref}$)
- time of instantaneous size change in $2N_{ref}$ generations in the past
- ratio of contemporary to ancient population size
Can you write Python code for it?

%%px --local

# create link to function that specifies the model
func = dadi.Demographics1D.bottlegrowth

# create extrapolating version of the model function
func_ex = dadi.Numerics.make_extrap_log_func(func)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
The following variables will be passed as string values to run_dadi.py:
Can you write Python code for it?

path_to_spectrum_file = "dadiExercises/ERY.FOLDED.sfs.dadi_format"
dadi_model = "dadi.Demographics1D.bottlegrowth"
upper = str(upper_bound)
lower = str(lower_bound)
dadi_opt_func = "dadi.Inference.optimize_log_fmin"
stub = "bottlegrowth_NM"
maxiter = 100

% ll dadiExercises/

# check creation of command line

for i, p_init in enumerate(product(p0_nuB, p0_nuF, p0_T)):
    cmd = "./run_dadi.py -p %s -m %s -u '%s' -l '%s' -d %s -s %s --maxiter %s -i '%s'" \
                % (path_to_spectrum_file, dadi_model, upper, lower, dadi_opt_func, stub, str(maxiter), str(p_init))
    print cmd
    if i >= 0: break


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
The last 8 optimisations take an exceptionally long time to run.
When I want to kill all processes that are still running, I can execute the following command line in the shell:
sudo pgrep -f "python ./run_dadi" | xargs kill
Can you write Python code for it?

E.progress()

% ll OUT_run_dadi/*pickle | wc -l


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
The vast majority of optimisations could not converge to an optimum within the 100 allowed iterations.
Can you write Python code for it?

# this is very important for the correct calculation of likelihoods

# import 1D spectrum of ery on all engines:
fs_par = dadi.Spectrum.from_file('dadiExercises/PAR.FOLDED.sfs.dadi_format')

sfs = fs_par

# this takes too long
# DO NOT RUN
#successfull_popt = [flatten(out[:6]) + [get_ll(out[1][0])] for out in opt_out if out[1][4] == 0]

# create parallel function with load balancing

@lbview.parallel(block=True)
def get_ll(p):
    """
    p: parameter combination

    First, calculates the best-fit model SFS given paramter combination p.
    Then returns the log likelihood of the expected SFS given the observed SFS.
    """
    expected_sfs = func_ex(p, ns, pts_l)
    return dadi.Inference.ll_multinom(expected_sfs, sfs)
    # expected_sfs does not need to be folded, ll_multinom does that automatically
    # make sure that sfs points to right spectrum

# this takes less than 1 minute running on the virtual 20 cores in parallel
ll = get_ll.map([out[1][0] for out in opt_out if out[1][4] == 0])

successfull_popt = [flatten(out)[:6] for out in opt_out if out[1][4] == 0]

successfull_popt[:3]

print len(successfull_popt)
print len(ll)

from itertools import izip

i=0
for row in izip(successfull_popt, ll):
    print flatten(row)
    i+=1
    if i > 3: break

p_opt = [flatten(row) for row in izip(successfull_popt, ll)]

df = pd.DataFrame(data=p_opt, columns=['nuB_0', 'nuF_0', 'T_0', 'nuB_opt', 'nuF_opt', 'T_opt', 'logL'])

df.sort_values(by='logL', ascending=False).head(20)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
There are only 12 physical cores on huluvu (6 dual-core CPU's').
Can you write Python code for it?

from itertools import product
import numpy as np

# set lower and upper bounds to nuB, nuF and T
upper_bound = [1e4, 1e4, 4]
lower_bound = [1e-4, 1e-4, 0]

str(upper_bound)

# create range of starting values evenly distributed in log space
p0_nuB = np.logspace(-3, 3, base=10.0, num=6)
p0_nuF = np.logspace(-3, 3, base=10.0, num=6)
p0_T = np.logspace(-4, np.log10(4), base=10, num=6)

for i, comb in enumerate(product(p0_nuB, p0_nuF, p0_T)):
    print comb
    if i > 10: break

import sys

sys.path.insert(0, '/home/claudius/Downloads/dadi')

import dadi


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
There is no convergence on a set of optimal parameter values. Fitting the bottlegrowth model to the 1D SFS of parallelus must be considered as failed.
masking outliers
Can you write Python code for it?

import matplotlib.pyplot as plt

%matplotlib inline

plt.rcParams['figure.figsize'] = [12.0, 10.0]

plt.plot(fs_par[:19])


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
This has no effect. Engines keep running. The only way to stop them is by shutting down the whole ipcluster.
Can you write Python code for it?

# total running time in minutes

ar_ery.elapsed/60


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
This runs a very long time!
Several optimisation runs take more than two hours. I have terminated those runs from the terminal with several executions of the following command:
pgrep -of "python ./run_dadi.py" | xargs kill

This will kill the longest running command that has a matching command line and thereby freeing slots for new optimisations.
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
This runs too long! Again, most of the time just on two cores. I cannot wait that long for exploratory analyses. Unfortunately, I cannot interrupt those long lasting jobs on the remote engines once they have started. If I shutdown the whole IPython cluster of engines, I cannot retrieve the information from the jobs that finished earlier (and did not get aborted). I think I need to be able to abort jobs that last too long but still have access to the results of the other jobs.
The only solution I can think of at the moment is to write a python script that takes initial starting values as input (like the function run_dadi), runs dadi optimisation with it and spits out the optimal parameter values to a separate file. Then I start this script in many subprocesses, thereby running in many instances in parallel on many initial parameter values and I would be able to interrupt the parent process (that initiates the subprocesses) without loosing the output from already completed subprocesses.
Also look at these two ipyparallel forum entries: 1 and 2. However, os.kill does not seem to have an effect. Jobs just keep running. I don't know why it does not work and don't have the time (nor the nerves) to find out why.
Can you write Python code for it?

def flatten(array):
    """
        Returns a list of flattened elements of every inner lists (or tuples)
        ****RECURSIVE****
    """
    res = []
    for el in array:
        if isinstance(el, (list, tuple)):
            res.extend(flatten(el))
            continue
        res.append(el)
    return res

# create parallel function with load balancing

@lbview.parallel(block=True)
def get_ll(p):
    """
    p: parameter combination

    First, calculates the best-fit model SFS given paramter combination p.
    Then returns the log likelihood of the expected SFS given the observed SFS.
    """
    expected_sfs = func_ex(p, ns, pts_l)
    return dadi.Inference.ll_multinom(expected_sfs, sfs)
    # expected_sfs does not need to be folded, ll_multinom does that automatically
    # make sure that sfs points to right spectrum


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
Those optimisations that were successful all return the same optimal parameter combination.
Can you write Python code for it?

import dill

dill.dump(list(ar_ery.get()), open("OUT_bottlegrowth/ERY_perturb_ar_ery.dill", "w"))


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
Unfortunately, the first and second frequency classes show counts that are inconsistent with any demographic history. It might therefore be worthwhile to mask them out. I think masking out just one of the first two frequency classes will lead to highly biased inferences. Masking both frequency classes will reduce a lot of the power to infer the demographic history. I therefore think that at least for this SFS, masking will most likely not lead to better estimates.
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
What were the initial parameter values for unsuccessful optimisations?
Can you write Python code for it?

for out in ar_ery:
    if out[1][4] == 0:
        continue
    print out[0]


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
check warnflags
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
rerun optimsation
Can you write Python code for it?

# run optimnisations with all combinations of starting values

ar_ery = lbview.map(run_dadi, product(p0_nuB, p0_nuF, p0_T), block=False, order=True)

ar_ery.progress

ar_ery.elapsed/60


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
run optimisation
The following cell is very inportant for the upper function run_dadi to work. It creates global variables on all engines, which are called by the function (instead of supplied to the function, which is difficult to implement).
Can you write Python code for it?

%%px

# set up global variables on engines required for run_dadi function call

dadi_opt_func = dadi.Inference.optimize_log # uses gradient based BFGS algorithm
sfs = fs_ery # use ERY spectrum
perturb = False
fold = 1
maxiter = 3 # run a maximum of 3 iterations
verbose = 0
full_output = True # need to have full output to get the warnflags (see below)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
run optimisations
Can you write Python code for it?

%%time

E = executor(limit=-12) # leave 4 cores unused

# this will block
for i, p_init in enumerate(product(p0_nuB, p0_nuF, p0_T)):
    cmd = "./run_dadi.py -p %s -m %s -u '%s' -l '%s' -d %s -s %s --maxiter %s -i '%s'" \
                % (path_to_spectrum_file, dadi_model, upper, lower, dadi_opt_func, stub, str(maxiter), str(p_init))
    E.err = "stderr"
    E.submit(cmd, p_init)
    #if i >= 9: break

E.progress()


I am working on the file "Data_analysis/SNP-indel-calling/dadi/03_1D_bottlegrowth.ipynb".
The first task is:
setting boundaries to parameter space
Can you write Python code for it?

%%px

# set lower and upper bounds to nuB, nuF and T
upper_bound = [1e4, 1e4, 4]
lower_bound = [1e-4, 1e-4, 0]

import numpy as np

# create range of starting values evenly distributed in log space
p0_nuB = np.logspace(-3, 3, base=10.0, num=6)
p0_nuF = np.logspace(-3, 3, base=10.0, num=6)
p0_T = np.logspace(-4, np.log10(4), base=10, num=6)

# number of starting parameter combinations:

6**3


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_expGrowth_bottleneck.ipynb".
The first task is:
Again, note that the inferred optimal parameter values only deviate slightly from the initial values and that all optimal parameter combinations have the same likelihood.
Can you write Python code for it?

p0 = [0.01, 0.01, 10, 1] # quick exp. decline, then long period of increased size

ar_par_3 = lbview.map(run_dadi, repeat(p0, 10), block=False, order=False)

get_flag_count(ar_par_3, NM=True)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_expGrowth_bottleneck.ipynb".
The first task is:
I am going to use unfolded spectra from ANGSD that are folded in dadi.
Can you write Python code for it?

%%px --local

# import 1D spectrum of ery on all engines:
fs_ery = dadi.Spectrum.from_file('ERY.unfolded.sfs').fold()

# import 1D spectrum of ery on all engines:
fs_par = dadi.Spectrum.from_file('PAR.unfolded.sfs').fold()


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_expGrowth_bottleneck.ipynb".
The first task is:
In all successful optimisations above, $\nu_F$, the ratio of contemporary population size to ancient population size, converges to a value below 1/3.
Can you write Python code for it?

# perturb previous optimal parameter combination
p0 = success[0][4:8]

ar_ery_4 = lbview.map(run_dadi, repeat(p0, 10), block=False, order=True)

get_flag_count(ar_ery_4, NM=True)

success = [flatten(out)[:9] for out in ar_ery_4 if out[1][4] == 0]

df = pd.DataFrame(data=success, \
                  columns=['nuB_0','TB_0', 'nuF_0', 'TF_0', 'nuB_opt', 'TB_opt', 'nuF_opt', 'TF_opt', '-logL'])

df.sort_values(by='-logL', ascending=True)

# save optimisation results to file

import dill

optout = []

for ar_ery in (ar_ery_1, ar_ery_2, ar_ery_3, ar_ery_4):
    optout.extend(list(ar_ery.get()))

dill.dump(optout, open("OUT_expGrowth_bottleneck/ERY_perturb_ar_ery.dill", "w"))


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_expGrowth_bottleneck.ipynb".
The first task is:
The initial parameter values specified above (and then perturbed) are too far away from an optimum.
Can you write Python code for it?

optout = []

for ar_par in (ar_par_1, list(ar_par_2.get())):
    optout.extend(ar_par)

dill.dump(optout, open("OUT_expGrowth_bottleneck/PAR_perturb_ar_ery.dill", "w"))


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_expGrowth_bottleneck.ipynb".
The first task is:
The optimised parameter values do not deviate very much from the initial parameter values.
Can you write Python code for it?

# specify the initial parameter values, they will be randomly perturbed by up to a factor of 4
p0 = [100, 0.01, 0.1, 1] # quick exp. growth, then long period of small size

ar_ery_2 = lbview.map(run_dadi, repeat(p0, 10), block=False, order=True)

success = [flatten(out)[:9] for out in ar_ery_2 if out[1][4] == 0]

df = pd.DataFrame(data=success, \
                  columns=['nuB_0','TB_0', 'nuF_0', 'TF_0', 'nuB_opt', 'TB_opt', 'nuF_opt', 'TF_opt', '-logL'])

df.sort_values(by='-logL', ascending=True)

# specify the initial parameter values, they will be randomly perturbed by up to a factor of 4
p0 = [0.01, 0.01, 10, 1] # quick exp. decline, then long period of increased size

ar_ery_3 = lbview.map(run_dadi, repeat(p0, 10), block=False, order=True)

success = [flatten(out)[:9] for out in ar_ery_3 if out[1][4] == 0]

df = pd.DataFrame(data=success, \
                  columns=['nuB_0','TB_0', 'nuF_0', 'TF_0', 'nuB_opt', 'TB_opt', 'nuF_opt', 'TF_opt', '-logL'])

df.sort_values(by='-logL', ascending=True)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_expGrowth_bottleneck.ipynb".
The first task is:
This model specifies exponential growth/decline toward $\nu_B$ for some time $TB$, after which the population size undergoes an instantaneous size change to the contemporary size (ratio with $N_{ref}$).
Can you write Python code for it?

%%px --local

# create link to function that specifies the model
func = expGrowth_bottleneck

# create extrapolating version of the model function
func_ex = dadi.Numerics.make_extrap_log_func(func)

%%px

# set up global variables on engines required for run_dadi function call

dadi_opt_func = dadi.Inference.optimize_log_fmin # uses Nelder-Mead algorithm
sfs = fs_ery # use ERY spectrum
perturb = True
fold = 2 # perturb randomly up to 6-fold
maxiter = 100 # run a maximum of 100 iterations
verbose = 0
full_output = True # need to have full output to get the warnflags (see below)
outname = "OUT_expGrowth_bottleneck/ERY_perturb" # set file name stub for opt. result files

# set lower and upper bounds to nu and T
upper_bound = [1e4, 5, 1e4, 5]
lower_bound = [1e-4, 0, 1e-4, 0]

ns = fs_ery.sample_sizes # both populations have the same sample size

fs_ery.pop_ids = ['ery']
fs_par.pop_ids = ['par']

# setting the smallest grid size slightly larger than the largest population sample size (36)
pts_l = [40, 50, 60]

lbview = cl.load_balanced_view()

def run_dadi(p_init): # for the function to be called with map, it needs to have one input variable
    """
    p_init: initial parameter values to run optimisation from
    """
    if perturb == True:
        p_init = dadi.Misc.perturb_params(p_init, fold=fold,
                                      upper_bound=upper_bound, lower_bound=lower_bound)
        # note upper_bound and lower_bound variables are expected to be in the namespace of each engine
    # run optimisation of paramters
    popt = dadi_opt_func(p0=p_init, data=sfs, model_func=func_ex, pts=pts_l, \
                                   lower_bound=lower_bound, upper_bound=upper_bound, \
                                   verbose=verbose, maxiter=maxiter, full_output=full_output)
    # pickle to file
    import dill
    name = outname[:] # make copy of file name stub!
    for p in p_init:
        name += "_%.4f" % (p)
    with open(name + ".dill", "w") as fh:
        dill.dump((p_init, popt), fh)

    return...


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_expGrowth_bottleneck.ipynb".
The first task is:
define exp. growth, then bottleneck model function
Can you write Python code for it?

%psource dadi.Demographics1D.growth

%%px --local

def expGrowth_bottleneck(params, ns, pts):
    """
    exponential growth followed by instantanous size change.

    params = (nuB,TB,nuF,TF)
    ns = (n1,)

    nuB: Ratio of population size after exponential growth to ancient
         population size
    nuF: Ratio of contemporary to ancient population size
    TB: Time during which exponential growth happened
       (in units of 2*Na generations)
    TF: Time in the past at which instantanous size change happened (after exp. growth)
    n1: Number of samples in resulting Spectrum
    pts: Number of grid points to use in integration.
    """
    nuB,TB,nuF,TF = params

    xx = dadi.Numerics.default_grid(pts)
    phi = dadi.PhiManip.phi_1D(xx)

    nu_func = lambda t: numpy.exp(numpy.log(nuB) * t/TB)
    phi = dadi.Integration.one_pop(phi, xx, TB, nu_func)

    phi = dadi.Integration.one_pop(phi, xx, TF, nuF)

    fs = dadi.Spectrum.from_phi(phi, ns, (xx,))
    return fs


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_expGrowth_bottleneck.ipynb".
The first task is:
parallelus
Can you write Python code for it?

%%px

# set up global variables on engines required for run_dadi function call

dadi_opt_func = dadi.Inference.optimize_log_fmin # uses Nelder-Mead algorithm
sfs = fs_par # use PAR spectrum
perturb = True
fold = 2 # perturb randomly up to 6-fold
maxiter = 100 # run a maximum of 100 iterations
verbose = 0
full_output = True # need to have full output to get the warnflags (see below)
outname = "OUT_expGrowth_bottleneck/PAR_perturb" # set file name stub for opt. result files

p0 = [1, 1, 1, 1]

ar_par_1 = lbview.map(run_dadi, repeat(p0, 10), block=False, order=False)

%ll OUT_expGrowth_bottleneck/PAR*

ar_par_1 = []

import glob

for filename in glob.glob("OUT_expGrowth_bottleneck/PAR*"):
    ar_par_1.append(dill.load(open(filename)))

get_flag_count(ar_par_1, NM=True)

success = [flatten(out)[:9] for out in ar_par_1 if out[1][4] == 0]

df = pd.DataFrame(data=success, \
                  columns=['nuB_0','TB_0', 'nuF_0', 'TF_0', 'nuB_opt', 'TB_opt', 'nuF_opt', 'TF_opt', '-logL'])

df.sort_values(by='-logL', ascending=True)

success[1][4:8]

p0 = success[1][4:8]

ar_par_2 = lbview.map(run_dadi, repeat(p0, 10), block=False, order=False)

get_flag_count(ar_par_2, NM=True)

success = [flatten(out)[:9] for out in ar_par_2 if out[1][4] == 0]

df = pd.DataFrame(data=success, \
                  columns=['nuB_0','TB_0', 'nuF_0', 'TF_0', 'nuB_opt', 'TB_opt', 'nuF_opt', 'TF_opt', '-logL'])

df.sort_values(by='-logL', ascending=True)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_three_epoch.ipynb".
The first task is:
Aha!
Can you write Python code for it?

successfull_popt = [flatten(out)[:9] for out in ar_ery_1 if out[1][4] == 0]

df = pd.DataFrame(data=successfull_popt, \
                  columns=['nuB_0', 'nuF_0', 'TB_0', 'TF_0', 'nuB_opt', 'nuF_opt', 'TB_opt', 'TF_opt', '-logL'])

df.sort_values(by='-logL', ascending=True) # smaller is better


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_three_epoch.ipynb".
The first task is:
Fairly different parameter combinations have practically identical likelihood. A reduction of the contemporary population size to 1/4 of the ancient population size is quite conistently inferred ($\nu_F$). The ancient population size change is not inferred consistently.
I am going to try one more p0. In the two epoch model fitting, erythropus had shown an ancient increase in population size.
Can you write Python code for it?

# specify the initial parameter values, they will be randomly perturbed by up to a factor of 2
p0 = [40, 0.4, 0.5, 0.5]

ar_ery_1 = lbview.map(run_dadi, repeat(p0, 40), block=False, order=True)

ar_ery_1.ready()

get_flag_count(ar_ery_1, NM=True)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_three_epoch.ipynb".
The first task is:
I am going to visualise these population size histories.
Can you write Python code for it?

# add time for ancient size change

df['TB+TF'] = pd.Series(df['TB_opt']+df['TF_opt'])

# extract columns from table
nuB = df.loc[:,'nuB_opt']
nuF = df.loc[:, 'nuF_opt']
Tb_Tf = df.loc[:, 'TB+TF']
TF = df.loc[:, 'TF_opt']

# calculate reference (ancient) population size from theta estimate (derived elsewhere by fitting a neutral spectrum)

theta_par = 10632.738922047551
mu = 3e-9
L = fs_par.data.sum() # this sums over all entries in the spectrum, including masked ones, i. e. also contains invariable sites
print "The total sequence length for the ery spectrum is {0:,}.".format(int(L))
N_ref_par = theta_par/L/mu/4

# turn nu into absolute Ne and T into generations

nuB_n = nuB*N_ref_par
nuF_n = nuF*N_ref_par
Tb_Tf_g = Tb_Tf*2*N_ref_par
TF_g = TF*2*N_ref_par

anc = [N_ref_par] * len(nuB)
pres = [1] * len(nuB)
past = [max(Tb_Tf_g)+1000] * len(nuB)

import matplotlib.pyplot as plt

%matplotlib inline

plt.rcParams['figure.figsize'] = [12.0, 10.0]

for run in zip(past, Tb_Tf_g, Tb_Tf_g, TF_g, TF_g, pres, anc, anc, nuB_n, nuB_n, nuF_n, nuF_n):
    plt.semilogy(run[:6], run[6:])
plt.xlabel('generations in the past')
plt.ylabel('effective population size')


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_three_epoch.ipynb".
The first task is:
I would be good to visualise these different scenarios.
Combine both data frames:
Can you write Python code for it?

successfull_popt = [flatten(out)[:9] for out in ar_ery if out[1][4] == 0]

df_1 = pd.DataFrame(data=successfull_popt, \
                  columns=['nuB_0', 'nuF_0', 'TB_0', 'TF_0', 'nuB_opt', 'nuF_opt', 'TB_opt', 'TF_opt', '-logL'])

successfull_popt = [flatten(out)[:9] for out in ar_ery_1 if out[1][4] == 0]

df_2 = pd.DataFrame(data=successfull_popt, \
                  columns=['nuB_0', 'nuF_0', 'TB_0', 'TF_0', 'nuB_opt', 'nuF_opt', 'TB_opt', 'TF_opt', '-logL'])

df = df_1.append(df_2)
df.sort_values(by='-logL')

# remove one parameter combination with slightly lower logL than the others

df = df.sort_values(by='-logL').head(-1)

df['TB+TF'] = pd.Series(df['TB_opt']+df['TF_opt'])

df.head(1)

import matplotlib.pyplot as plt

%matplotlib inline

# extract columns from table
nuB = df.loc[:,'nuB_opt']
nuF = df.loc[:, 'nuF_opt']
Tb_Tf = df.loc[:, 'TB+TF']
TF = df.loc[:, 'TF_opt']

# calculate reference (ancient) population size from theta estimate (derived elsewhere by fitting a neutral spectrum)

theta_ery = 10617.328085456724
mu = 3e-9
L = fs_ery.data.sum() # this sums over all entries in the spectrum, including masked ones, i. e. also contains invariable sites
print "The total sequence length for the ery spectrum is {0:,}.".format(int(L))
N_ref_ery = theta_ery/L/mu/4

# turn nu into absolute Ne and T into generations

nuB_n = nuB*N_ref_ery
nuF_n = nuF*N_ref_ery
Tb_Tf_g = Tb_Tf*2*N_ref_ery
TF_g = TF*2*N_ref_ery

anc = [N_ref_ery] * len(nuB)
pres = [1] * len(nuB)
past = [max(Tb_Tf_g)+1000] * len(nuB)

plt.rcParams['figure.figsize'] = [12.0, 10.0]

for run in zip(past, Tb_Tf_g, Tb_Tf_g, TF_g, TF_g, pres, anc, anc, nuB_n, nuB_n, nuF_n, nuF_n):
    plt.semilogy(run[:6], run[6:])
plt.xlabel('generations in the past')
plt.ylabel('effective population size')


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_three_epoch.ipynb".
The first task is:
It is very hard for me to believe that the log likelihood of all parameter combinations is almost exactly the same AND also the same as the likelihood for all parameter combinations of the three epoch model above!
Can you write Python code for it?

[get_ll(out[1][0]) for out in ar_par_te if out[1][4] == 0]

ar_par_te.get()

from scipy.optimize import fmin

?fmin


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_three_epoch.ipynb".
The first task is:
It would be nice to get a few more replicates for these results. I am going to use these optimal parameter combinations, perturb them randomly and use them as starting values for new optimisations.
Can you write Python code for it?

df.loc[:, ['nuB_opt', 'nuF_opt', 'TB_opt', 'TF_opt']]

# specify new set initial parameter values (the optimal sets repeated 3 times)

p0 = []

for _ in range(3):
    p0.extend(np.array(df.loc[:, ['nuB_opt', 'nuF_opt', 'TB_opt', 'TF_opt']]))

p0

%%px

fold = 2 # perturb up to 4 times

# run optimisations

ar_par_2 = lbview.map(run_dadi, p0, block=False, order=True)

ar_par_2.progress

get_flag_count(ar_par_2, NM=True)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_three_epoch.ipynb".
The first task is:
Next, I need to define an executor class that takes care of submiting jobs as slots become available. This is taken from Tiago Antao's Cookbook:
Can you write Python code for it?

class executor:
    def __init__(self, limit):
        self.limit = limit
        self.ncores = multiprocessing.cpu_count()
        self.running = []
        self.finished = 0

    def submit(self, cmd, p_init):
        self.wait()
        if hasattr(self, 'out'):
            out = self.out
        else:
            out = '/dev/null'
        if hasattr(self, 'err'):
            err = self.err
        else:
            err = '/dev/null'
        if err == 'stderr':
            errSt = ''
        else:
            errSt = '2> ' + err
        # this is where the magic happens:
        proc = subprocess.Popen('%s > %s %s' % (cmd, out, errSt), shell=True)
        self.running.append(proc)
        #
        if hasattr(self, 'out'):
            del self.out
        if hasattr(self, 'err'):
            del self.err

    def wait(self, for_all=False):
        self.clean_done()
        #numWaits = 0
        if self.limit > 0 and type(self.limit) == int:
            cond = 'len(self.running) >= self.ncores - self.limit'
        elif self.limit < 0:
            cond = 'len(self.running) >= - self.limit'
        else:
            cond = 'len(self.running) >= self.ncores * self.limit'
        while eval(cond) or (for_all and len(self.running) > 0):
            time.sleep(1)
            self.clean_done() # updates self.running, removes finished jobs from the running queue
            #numWaits += 1

    def clean_done(self):
        terminated = []
        for i, p in enumerate(self.running):
            if p.poll() is not None: # None means it's still running
                terminated.append(i)
        for idx in reversed(terminated):
            del self.running[idx]
            self.finished += 1

    def progress(self):
        self.clean_done()
        print self.finished

E = executor(limit=-18) # use up to 18 cores

# this will block
for i, p_init in enumerate(product(p0_nuB, p0_nuF, p0_TF, p0_TB)):
    cmd = "./run_dadi.py -p %s -m %s -u '%s' -l '%s' -d %s -s %s --maxiter %s -i...


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_three_epoch.ipynb".
The first task is:
Oho!
Can you write Python code for it?

successfull_popt_ery_te = [flatten(out)[:5] for out in ar_ery_te if out[1][4] == 0]
df = pd.DataFrame(data=successfull_popt_ery_te, \
                  columns=['nuB_0', 'T_0', 'nuB_opt', 'T_opt', 'logL'])
df.sort_values(by='logL', ascending=False)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_three_epoch.ipynb".
The first task is:
Ok, this looks like several quite different population histories are equally likely. These parameter values seem to suggest that erythropus first underwent a population increase to a $\nu_B$ fold of the ancient population size (ranging from 23$\times$ to $\gt$7000$\times$!) and stayed at this size for $TB$ (x2N) generations upon which it dramatically reduced its population size to a fraction $\nu_F$ of the ancient size and stayed at this size the remaining TF (x2N) generations.
Can you write Python code for it?


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_three_epoch.ipynb".
The first task is:
The ancient size change is generally inferred to be more than 1 million generations ago (even up to 10 million)?!
Are the strength of the first population size change, $(\frac{1}{\nu_B})^{TB}$, and the strength of the second population size change, $(\frac{1}{\nu_F})^{TF}$, correlated with each other?
Can you write Python code for it?

plt.rcParams['font.size'] = 12.0

plt.plot((1.0/df['nuB_opt'])**df['TB_opt'], (1.0/df['nuF_opt'])**df['TF_opt'], 'bo')
plt.xlabel('strength of first size change')
plt.ylabel('strength of second size change')


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_three_epoch.ipynb".
The first task is:
The built-in three_epoch model specifies a piecewise history (with only instantaneous population size changes instead of gradual changes). At time $TF+TB$ the ancient population underwent a size change, stayed at this size for $TB \times 2N_{ref}$ generations and then underwent a second size size change $TF \times 2N_{ref}$ generations in the past to the contemporary population size. The model has therefore two population size parameters, $\nu_B$ and $\nu_F$ as well as two time parameters, $TB$ and $TF$.
Can you write Python code for it?

# set lower and upper bounds to nuB, nuF and T
upper_bound = [1e4, 1e4, 4, 4]
lower_bound = [1e-4, 1e-4, 0, 0]

# create range of starting values evenly distributed in log space
p0_nuB = np.logspace(-3, 3, base=10.0, num=6)
p0_nuF = np.logspace(-3, 3, base=10.0, num=6)
p0_TF = np.logspace(-4, np.log10(4), base=10, num=6)
p0_TB = np.logspace(-4, np.log10(4), base=10, num=6)

print "There are %d starting parameter values" % (6**4)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_three_epoch.ipynb".
The first task is:
There is no correlation between the strength of the two size changes.
Can you write Python code for it?

import dill

dump_this = ar_par
dump_this.extend(list(ar_par_2.get()))

dill.dump(dump_this, open("OUT_three_epoch/PAR_perturb_ar_par.dill", "w"))

p0 = [100, 1e-2, 1, 2] # start from more extreme initial parameter values

ar_par_3 = lbview.map(run_dadi, repeat(p0, 10), block = False, order=False)

ar_par_3.progress

get_flag_count(ar_par_3, NM=True)

dill.dump(list(ar_par_3.get()), open("OUT_three_epoch/PAR_perturb_extreme_ar_par.dill", "w"))

# add new output to previous output
successfull_popt_3 =  [flatten(out)[:9] for out in ar_par_3 if out[1][4] == 0]

# create data frame
df_3 = pd.DataFrame(data=successfull_popt_3, \
                  columns=['nuB_0', 'nuF_0', 'TB_0', 'TF_0', 'nuB_opt', 'nuF_opt', 'TB_opt', 'TF_opt', 'logL'])

# sort data frame by negative log likelihood
df_3.sort_values(by='logL', ascending=True)

plt.plot(df_3['nuB_opt']**df_3['TB_opt'], (1.0/df_3['nuF_opt'])**df_3['TF_opt'], 'bo')


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_three_epoch.ipynb".
The first task is:
These are already an enormous amount of parameter combinations. I think I will need to keep maxiter low so that optimisations that do not converge early end more quickly when they hit the maximum number of iterations. I will also frequently have to prune the available cores of optimisation runs that have already taken a considerable amount of time. This can be done with the following command:
pgrep -of "python ./run_dadi.py" | xargs kill

The following variables will be passed as string values to run_dadi.py:
Can you write Python code for it?

path_to_spectrum_file = "dadiExercises/ERY.FOLDED.sfs.dadi_format"
dadi_model = "dadi.Demographics1D.three_epoch"
upper = str(upper_bound)
lower = str(lower_bound)
dadi_opt_func = "dadi.Inference.optimize_log_fmin" # Nelder-Mead
stub = "ERY_three_epoch_NM"
maxiter = 10 # keep maxiter low, but not too low to allow convergence of some optimisations

%run run_dadi.py -h

from itertools import product

# check creation of command line

for i, p_init in enumerate(product(p0_nuB, p0_nuF, p0_TF, p0_TB)):
    cmd = "./run_dadi.py -p %s -m %s -u '%s' -l '%s' -d %s -s %s --maxiter %s -i '%s'" \
                % (path_to_spectrum_file, dadi_model, upper, lower, dadi_opt_func, stub, str(maxiter), str(p_init))
    print cmd
    if i >= 0: break


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_three_epoch.ipynb".
The first task is:
fopt is the likelihood of the otimal parameter combination. I therefore don't need to calculate the likelihood with get_ll. It's already returned by the optimisation function.
OK. it may be that the likelihood values are correct and that there is a ridge in the likelihood surface.
retry two_epoch model with ery spectrum
I previously also wasn't able to find reasonable optimal parameter values for the two epoch model with the erythropus spectrum. I want to make another attempt here, given the apparently successful optimisation for the three epoch model above.
Can you write Python code for it?

%%px

# set up global variables on engines required for run_dadi function call

dadi_opt_func = dadi.Inference.optimize_log_fmin # uses Nelder-Mead algorithm
sfs = fs_ery # use ERY spectrum
perturb = True
fold = 1
maxiter = 100 # run a maximum of 100 iterations
verbose = 0
full_output = True # need to have full output to get the warnflags (see below)

# specify the initial parameter values, they will be randomly perturbed by up to a factor of 2
p0 = [1, 1]

ar_ery_te = lbview.map(run_dadi, repeat(p0, 20), block=False, order=True)

ar_ery_te.elapsed

get_flag_count(ar_ery_te, NM=True)

ar_ery_te.get()[0]

successfull_popt_ery_te = [flatten(out)[:5] for out in ar_ery_te if out[1][4] == 0]
df = pd.DataFrame(data=successfull_popt_ery_te, \
                  columns=['nuB_0', 'T_0', 'nuB_opt', 'T_opt', 'logL'])
df.sort_values(by='logL', ascending=False)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/04_1D_three_epoch.ipynb".
The first task is:
parallelus
Can you write Python code for it?

%%px

# set up global variables on engines required for run_dadi function call

dadi_opt_func = dadi.Inference.optimize_log_fmin # uses Nelder-Mead algorithm
sfs = fs_par # use PAR spectrum
perturb = True
fold = 3 # perturb randomly up to 6-fold
maxiter = 100 # run a maximum of 100 iterations
verbose = 0
full_output = True # need to have full output to get the warnflags (see below)
outname = "OUT_three_epoch/PAR_perturb" # set file name stub for opt. result files

def run_dadi(p_init): # for the function to be called with map, it needs to have one input variable
    """
    p_init: initial parameter values to run optimisation from
    """
    if perturb == True:
        p_init = dadi.Misc.perturb_params(p_init, fold=fold,
                                      upper_bound=upper_bound, lower_bound=lower_bound)
        # note upper_bound and lower_bound variables are expected to be in the namespace of each engine
    # run optimisation of paramters
    popt = dadi_opt_func(p0=p_init, data=sfs, model_func=func_ex, pts=pts_l, \
                                   lower_bound=lower_bound, upper_bound=upper_bound, \
                                   verbose=verbose, maxiter=maxiter, full_output=full_output)
    # pickle to file
    import dill
    name = outname[:] # make copy of file name stub!
    for p in p_init:
        name += "_%.4f" % (p)
    with open(name + ".dill", "w") as fh:
        dill.dump((p_init, popt), fh)

    return p_init, popt

# specify the initial parameter values, they will be randomly perturbed by up to a factor of 2 times `fold`
p0 = [1, 1, 1, 1]

ar_par = lbview.map(run_dadi, repeat(p0, 20), block=False, order=True)

ar_par.progress

ar_par.elapsed/60


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_1D_model_synthesis.ipynb".
The first task is:
$\theta$ and implied $N_{ref}$
standard neutral model
Can you write Python code for it?

# create link to built-in model function

func = dadi.Demographics1D.snm

# make the extrapolating version of the demographic model function

func_ex = dadi.Numerics.make_extrap_log_func(func)

# setting the smallest grid size slightly larger than the largest population sample size

pts_l = [40, 50, 60]

ns = fs_ery.sample_sizes
ns

# calculate unfolded AFS under standard neutral model (up to a scaling factor theta)

neutral_model = func_ex(0, ns, pts_l)
neutral_model

theta_ery = dadi.Inference.optimal_sfs_scaling(neutral_model, fs_ery)
theta_ery

theta_par = dadi.Inference.optimal_sfs_scaling(neutral_model, fs_par)
theta_par


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_1D_model_synthesis.ipynb".
The first task is:
All parameter combinations consistently show a reduction in the contemporary population size with respect to the ancient population size ($\nu_F$). The ancient population size change is much less clear and ranges from a 300-fold increase to 100-fold reduction. Also this event is inferred to have occurred quite distant in the past, generally much more than 1 million generations ago.
exponential growth then bottleneck model
This model specifies exponential growth/decline toward $\nu_B \times N_{ref}$ during a time period of $TB \times 2N_{ref}$ generations, after which (at time $TF \times 2N_{ref}$) the population size undergoes an instantaneous size change to the contemporary size ($\nu_F \times N_{ref}$).
erythropus
Can you write Python code for it?

ar_ery = dill.load(open("OUT_expGrowth_bottleneck/ERY_perturb_ar_ery.dill"))

success = [flatten(out)[:9] for out in ar_ery if out[1][4] == 0]

df = pd.DataFrame(data=success, \
                  columns=['nuB_0','TB_0', 'nuF_0', 'TF_0', 'nuB_opt', 'TB_opt', 'nuF_opt', 'TF_opt', '-logL'])

df.sort_values(by='-logL', ascending=True)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_1D_model_synthesis.ipynb".
The first task is:
Can a shortterm effective population size of 21 million reasonably be assumed?
Can you write Python code for it?

print "This was followed by exponential decline towards a current population size of {0:,}.".format(int(0.4576*N_ref_ery))


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_1D_model_synthesis.ipynb".
The first task is:
Except for the last two, all optimal parameter combinations have the same likelihood and they do not deviate much from their respective initial values. In all successful optimisations above, $\nu_F$, the ratio of contemporary population size to ancient population size, converges to a value below 1/3. I think an ancient size increase or decrease cannot be inferred.
Can you write Python code for it?

F = df[df['-logL'] < 2172].loc[:,['nuF_opt', 'TF_opt']]

pylab.plot(F['TF_opt']*2*N_ref_ery, F['nuF_opt']*N_ref_ery, 'bo')
pylab.xlabel('generations')
pylab.ylabel('contemporary pop. size')


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_1D_model_synthesis.ipynb".
The first task is:
For parallelus, the spectrum folded by dadi looks better than the spectrum folded by ANGSD. I am therefore going to use spectra folded in dadi.
Can you write Python code for it?

fs_ery = fs_ery_unfolded.fold()
fs_par = fs_par_unfolded.fold()


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_1D_model_synthesis.ipynb".
The first task is:
If a long period of increased population size would be correlated with a long period of decreased population size, this plot should show a positive correlation. This does not seem to be the case.
parallelus
Can you write Python code for it?

import dill

ar_par = dill.load(open("OUT_three_epoch/PAR_perturb_ar_par.dill"))
ar_par_extreme = dill.load(open("OUT_three_epoch/PAR_perturb_extreme_ar_par.dill"))

# add new output to previous output
successfull_popt = [flatten(out)[:9] for out in ar_par if out[1][4] == 0]
successfull_popt.extend([flatten(out)[:9] for out in ar_par_extreme if out[1][4] == 0])

# create data frame
df = pd.DataFrame(data=successfull_popt, \
                  columns=['nuB_0', 'nuF_0', 'TB_0', 'TF_0', 'nuB_opt', 'nuF_opt', 'TB_opt', 'TF_opt', '-logL'])

# sort data frame by negative log likelihood
df.sort_values(by='nuB_opt', ascending=True)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_1D_model_synthesis.ipynb".
The first task is:
Note, all but the last parameter combination have the same likelihood. Fairly different parameter combinations have practically identical likelihood. A reduction of the contemporary population size to 1/4 of the ancient population size is quite conistently inferred ($\nu_F$). The ancient population size change ($\nu_B$) is not inferred consistently. It ranges from a 186-fold increase to a reduction to 1/3 of the ancient population size.
Can you write Python code for it?

# remove one parameter combination with slightly lower logL than the others

df = df.sort_values(by='-logL').head(-1)

# the time of the ancient pop size change is TB+TF

df['TB+TF'] = pd.Series(df['TB_opt']+df['TF_opt'])

# extract columns from table
nuB = df.loc[:,'nuB_opt']
nuF = df.loc[:, 'nuF_opt']
Tb_Tf = df.loc[:, 'TB+TF']
TF = df.loc[:, 'TF_opt']

# turn nu (a ratio) into absolute Ne and T into generations

nuB_n = nuB*N_ref_ery
nuF_n = nuF*N_ref_ery
Tb_Tf_g = Tb_Tf*2*N_ref_ery
TF_g = TF*2*N_ref_ery

anc = [N_ref_ery] * len(nuB) # ancestral pop size
pres = [1] * len(nuB) # 1 generation in the past
past = [max(Tb_Tf_g)+1000] * len(nuB) # furthest time point in the past

pylab.rcParams['figure.figsize'] = [12.0, 10.0]
pylab.rcParams['font.size'] = 12.0

# plot the best 21 parameter combinations

for x, y in zip(zip(past, Tb_Tf_g, Tb_Tf_g, TF_g, TF_g, pres), zip(anc, anc, nuB_n, nuB_n, nuF_n, nuF_n)):
    pylab.semilogy(x, y)
pylab.xlabel('generations in the past')
pylab.ylabel('effective population size')


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_1D_model_synthesis.ipynb".
The first task is:
Often, but not always are optimal parameter values close to their respective initial values. All optimal parameter combinations have the same likelihood. Only a recent (stepwise) population decrease is consistently inferred ($\nu_F$).
Can you write Python code for it?

F = df[df['TF_opt'] < 3].loc[:,['nuF_opt', 'TF_opt']]

pylab.plot(F['TF_opt']*2*N_ref_par, F['nuF_opt']*N_ref_par, 'bo')
pylab.xlabel('TF generations')
pylab.ylabel('contemporary pop. size')


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_1D_model_synthesis.ipynb".
The first task is:
The lower plot (green line) is for the scaled Poisson residuals.
$$
residuals = (model - data)/\sqrt{model}
$$
The model is the expected counts in each frequency class. If these counts are Poisson distributed, then their variance is equal to their expectation. The differences between model and data are therefore scaled by the expected standard deviation of the model counts.
Although a standard neutral model is unrealistic for either population, the observed counts deviate by up to 90 standard deviations from this model!
What could be done about this?
The greatest deviations are seen for the first two frequency classes, the ones that should provide the greatest amount of information for theta (Fu1994) and therefore probably also other parameters. I think masking out just one of the first two frequency classes will lead to highly biased inferences. Masking both frequency classes will reduce a lot of the power to infer the demographic history. I therefore think that masking will most likely not lead to better estimates.
Toni has suggested that the doubleton class is inflated due to "miscalling" heterozygotes as homozygotes. When they contain a singleton they will be "called" as homozygote and therefore contribute to the doubleton count. This is aggravated by the fact that the sequenced individuals are all male which only possess one X chromosome. The X chromosome is the fourth largest of the 9 chromosomes of these grasshoppers (8 autosomes + X) (see Gosalvez1988, fig. 2). That is, about 1/9th of the sequenced RAD loci are haploid but ANGSD assumes all loci to be diploid. The genotype likelihoods it calculates are all referring to diploid genotypes.
I think one potential reason for the extreme deviations is that the genotype likelihoods are generally biased toward homozygote genotypes (i. e. also for autosomal loci) due to PCR duplicates (see eq. 1 in Nielsen2012). So, one potential improvement would be to remove PCR duplicates.
Another potential improvement could be found by subsampling 8/9th to 8/10th of the...
Can you write Python code for it?

import dill

# loading optimisation results from previous analysis

growth_ery = dill.load(open('OUT_exp_growth_model/ERY_perturb_ar_ery.dill'))

def flatten(array):
    """
        Returns a list of flattened elements of every inner lists (or tuples)
        ****RECURSIVE****
    """
    import numpy
    res = []
    for el in array:
        if isinstance(el, (list, tuple, numpy.ndarray)):
            res.extend(flatten(el))
            continue
        res.append(el)
    return list( res )

import pandas as pd

i = 4 # where to find flag, 6 for BFGS, 4 for Nelder-Mead

successfull_popt_ery = [flatten(out)[:5] for out in growth_ery if out[1][i] == 0]

df = pd.DataFrame(data=successfull_popt_ery, \
                  columns=['nu_0', 'T_0', 'nu_opt', 'T_opt', '-logL'])

df.sort_values(by='-logL', ascending=True) # smaller is better

print "The most likely parameter combination for this model suggests that the ery population size started to exponentially shrink at {0:,} generations ago to a current population size of {1:,}.".format(int(0.00769*2*N_ref_ery), int(0.14929*N_ref_ery))


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_1D_model_synthesis.ipynb".
The first task is:
There could be a linear correlation between the duration of population size reduction and the size of the reduction, but it's weaker than for erythropus.
Conclusion
Not much can be inferred from the 1D spectra of both erythropus and parallelus. However, the dadi optimisations have consistently suggested a population size reduction more than 400,000 generations ago to less than 1/3 of the ancient population size. The reduction seems to have been much stronger in parallelus than in erythropus.
Can gene flow explain the difficulty of fitting 1D models?
The fitting of 2D models was substantially easier than fitting 1D models and led to conistent and robust parameter estimates.
Could it be that the inability to model gene flow from an external source is responsible for the issues with fitting 1D models?
I have saved to files the results of each optimisation run (i. e. for each combination of starting parameter values).
Can you write Python code for it?

# load results of fitting isolation with migration model to 2D sepctrum

import dill, glob

split_mig_res = []

for filename in glob.glob("OUT_2D_models/split_mig*dill"):
    split_mig_res.append(dill.load(open(filename)))

from utility_functions import *

import pandas as pd

success = [flatten(out)[:9] for out in split_mig_res if out[1][4] == 0]

df = pd.DataFrame(data=success, \
                  columns=['nu1_0','nu2_0', 'T_0', 'm_0', 'nu1_opt', 'nu2_opt', 'T_opt', 'm_opt', '-logL'])

df.sort_values(by='-logL', ascending=True)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_1D_model_synthesis.ipynb".
The first task is:
These are the successful parameter optimisations I have for the model that specifies a split at time $T$ in the past into two daughter populations with population size ratios (relative to $N_{ref}$) of $\nu_1$ for ery and $\nu_2$ for par. $m$ is the rate of migration, which is assumed to be symmetrical. The migration rate is the fraction of individuals from the other population times $2N_{ref}$, i. e. number of immigrant alleles.
The best paramter combination, that I got from these optimisation runs is:
Can you write Python code for it?

popt = df.sort_values(by='-logL', ascending=True).iloc[0, 4:8]
popt

import numpy as np

popt = np.array(popt)

func = dadi.Demographics2D.split_mig # divergence-with-gene-flow model function (built-in)

func_ex = dadi.Numerics.make_extrap_log_func(func)

ns = np.array([36, 36])

pts_l = [40, 50, 60]

# get optimal model spectrum from 2D model 'divergence-with-migration' model

split_mig_2D_best_fit_model_spectrum = func_ex(popt, ns, pts_l)

split_mig_2D_best_fit_model_spectrum

# import 2D unfolded spectrum
sfs2d_unfolded = dadi.Spectrum.from_file('dadiExercises/EryPar.unfolded.2dsfs.dadi_format')

# add population labels
sfs2d_unfolded.pop_ids = ["ery", "par"]

# fold the spectrum
sfs2d = sfs2d_unfolded.fold()

# need to scale model spectrum by optimal theta, which depends on the number of sites in the data

model_spectrum = dadi.Inference.optimally_scaled_sfs(split_mig_2D_best_fit_model_spectrum, sfs2d)
model_spectrum

import pylab

%matplotlib inline

pylab.rcParams['figure.figsize'] = [12, 10]
pylab.rcParams['font.size'] = 14

model_spectrum.pop_ids = sfs2d.pop_ids

dadi.Plotting.plot_single_2d_sfs(model_spectrum.fold(), vmin=1, cmap=pylab.cm.jet)

# get the marginal model spectra for ery and par from the optimal 2D model spectrum

fs_ery_model = model_spectrum.marginalize([1])

fs_par_model = model_spectrum.marginalize([0])

pylab.plot(fs_par_model.fold(), 'go-', label='par')
pylab.plot(fs_ery_model.fold(), 'rs-', label='ery')
pylab.xlabel('minor allele frequency')
pylab.ylabel('SNP count')
pylab.legend()

# import 1D unfolded spectrum of ery
fs_ery_unfolded = dadi.Spectrum.from_file('ERY.unfolded.sfs')

# import 1D unfolded spectrum of par
fs_par_unfolded = dadi.Spectrum.from_file('PAR.unfolded.sfs')

fs_ery = fs_ery_unfolded.fold()
fs_par = fs_par_unfolded.fold()

dadi.Plotting.plot_1d_comp_multinom(fs_ery_model.fold()[:19], fs_ery[:19])

dadi.Plotting.plot_1d_comp_multinom(fs_par_model.fold()[:19], fs_par[:19])

# log likelihoods

ll_model_ery = dadi.Inference.ll_multinom(fs_ery_model.fold(),...


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_1D_model_synthesis.ipynb".
The first task is:
These effective population sizes are consistent with those reported for other insect species (Lynch2016, fig. 3b).
In dadi, times are given in units of $2N_{ref}$ generations.
reducing noise and bias
Can you write Python code for it?

# compare neutral model prediction with ery spectrum

dadi.Plotting.plot_1d_comp_multinom(neutral_model.fold()[:19], fs_ery[:19], residual='linear')

# compare neutral model prediction with par spectrum

dadi.Plotting.plot_1d_comp_multinom(neutral_model.fold()[:19], fs_par[:19], residual='linear')


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_1D_model_synthesis.ipynb".
The first task is:
This optimal parameter set was difficult to find. It can only be found from parameter values that are already quite close to the optimal values. The model enforces exponential population decline. This may not be very realistic.
parallelus
A sweep through the parameter space did not lead to convergence of parameters for the parallelus spectrum. Optimisations starting from up to 6-fold randomly perturbed neutral parameter values (1, 1) also did not lead to successful optimisations. The exponential growth/decline model could therefore not be fit to the par spectrum.
two epoch model
erythropus
The built-in two epoch model defines a piecewise history, where at some time $T$ in the past the ancestral population instantaneously changed in size to the contemporary population size, which has ratio of $\nu$ to the ancestral population size.
The parameters values with the highest likelihood ($\nu=0.0001$ and $T=0.000002$) are hitting the lower bound of parameter space that I've set. The time parameter $T$ that dadi infers with the highest likelihood corresponds to 3 generations and therefore does not make sense. The two epoch model could therefore not be fit to the erythropus spectrum.
parallelus
Can you write Python code for it?

ar_par_te = dill.load(open("OUT_two_epoch/PAR_perturb_ar_par.dill"))

i = 4 # index of flag with NM algorithm

successfull_popt = [flatten(out)[:5] for out in ar_par_te if out[1][i] == 0]

df = pd.DataFrame(data=successfull_popt, \
                  columns=['nu_0','T_0', 'nu_opt', 'T_opt', '-logL'])

df.sort_values(by='-logL', ascending=True)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_1D_model_synthesis.ipynb".
The first task is:
Very different population size histories have the same likelihood. The most likely parameter combinations are hitting the upper parameter bound for the time parameter (6).
Can you write Python code for it?

print "A T of 6 corresponds to {0:,} generations.".format(6*2*N_ref_par)


I am working on the file "Data_analysis/SNP-indel-calling/dadi/05_1D_model_synthesis.ipynb".
The first task is:
What effective ancestral population size would that imply?
According to section 5.2 in the dadi manual:
$$
\theta = 4 N_{ref} \mu_{L} \qquad \text{L: sequence length}
$$
Then
$$
\mu_{L} = \mu_{site} \times L
$$
So
$$
\theta = 4 N_{ref} \mu_{site} \times L
$$
and
$$
N_{ref} = \frac{\theta}{4 \mu_{site} L}
$$
Let's assume the mutation rate per nucleotide site per generation is $3\times 10^{-9}$ (see e. g. Liu2017).
Can you write Python code for it?

mu = 3e-9
L = fs_ery.data.sum() # this sums over all entries in the spectrum, including masked ones, i. e. also contains invariable sites
print "The total sequence length for the ery spectrum is {0:,}.".format(int(L))
N_ref_ery = theta_ery/L/mu/4
print "The effective ancestral population size of ery (in number of diploid individuals) implied by this theta is: {0:,}.".format(int(N_ref_ery))

mu = 3e-9
L = fs_par.data.sum() # this sums over all entries in the spectrum, including masked ones, i. e. also contains invariable sites
print "The total sequence length for par spectrum is {0:,}.".format(int(L))
N_ref_par = theta_par/L/mu/4
print "The effective ancestral population size of par (in number of diploid individuals) implied by this theta is: {0:,}.".format(int(N_ref_par))