I am working on the file "Notebooks/ODE.ipynb".
The first task is:
Example 2 - Better coding practice
and also comparison between Forward Euler and 4th-order Runge-Kutta

Implement integrator to take f as an argument
Use consistent call signature for f

Make a function that will take f and an integrator, and step forward in time until Tmax

Also: Try modifying the timestep, and note how much faster the 4th-order Runge-Kutta converges compared to the 1st-order Euler method
Can you write Python code for it?

#####################
#### Integrators ####
#####################

# The forward Euler method
def euler(x, t, h, f):
    # x is coordinates (as a vector)
    # h is timestep
    # f(x) is a function that returns the derivative
    # "Slopes"
    k1  = f(x,           t)
    # Update time and position
    x_  = x + h*k1
    return x_

# Heun's Method
def rk2(x, t, h, f):
    # x is coordinates (as a vector)
    # h is timestep
    # f(x) is a function that returns the derivative
    # "Slopes"
    k1  = f(x,          t)
    k2  = f(x + k1*h,   t + h)
    # Update time and position
    x_ = x + h*(k1 + k2)/2
    return x_

# Kutta's method
def rk3(x, t, h, f):
    # x is coordinates (as a vector)
    # h is timestep
    # f(x) is a function that returns the derivative
    # "Slopes"
    k1  = f(x,                 t)
    k2  = f(x + k1*h/2,        t + h/2)
    k3  = f(x - k1*h + 2*k2*h, t + h)
    # Update time and position
    x_  = x + h*(k1 + 4*k2 + k3)/6
    return x_

# 4th-order Runge-Kutta
def rk4(x, t, h, f):
    # x is coordinates (as a vector)
    # h is timestep
    # f(x) is a function that returns the derivative
    # "Slopes"
    k1  = f(x,          t)
    k2  = f(x + k1*h/2, t + h/2)
    k3  = f(x + k2*h/2, t + h/2)
    k4  = f(x + k3*h,   t + h)
    # Update time and position
    x_  = x + h*(k1 + 2*k2 + 2*k3 + k4)/6
    return x_

###########################
#### Utility functions ####
###########################

def trajectory(X0, T0, Tmax, h, f, integrator):
    # Number of timesteps
    Nt   = int((Tmax) / h)
    # Add 2 for initial position and fractional step at end
    # X0.size lets X hold Nt+2 arrays of the same size as X0
    X0   = np.array(X0)
    X    = np.zeros((Nt+2, X0.size))
    T    = np.zeros(Nt+2)
    # Initialise
    X[0,:] = X0
    T[0]   = T0
    # Loop over timesteps
    t = T0
    for i in range(Nt+1):
        # Make sure the last step stops exactly at Tmax
        h  = min(h, Tmax - t)
        # Calculate next position
        X[i+1,:] = integrator(X[i,:], t, h, f)
     ...


I am working on the file "Notebooks/ODE.ipynb".
The first task is:
Example 3 - Error as function of timestep
To look at error as a function of timestep:
* Calculate trajectory for many different timesteps
* Calculate error as $x(t_N) - x_N$, where $N$ is the total number of timesteps and $x(t_N)$ is the exact solution at $t_N$.
* Plot on log-log scale, and add straight lines proportiona to $h$ and $h^4$ to compare slope
* The reason the error doesn't get smaller than about $10^{-12}$ in this case is due to limited precision in floating point numbers, and the large number of steps for the shorter timesteps
Can you write Python code for it?

def f(x, t):
    return x

# Initial conditions
x0   = 1/2
t0   = 0
Tmax = 10

# Define Nh logarithmically spaced timesteps from 0.0001 to 1
Nh = 50
timesteps = np.logspace(-4, 0, Nh)

# For each of those timesteps, calculate the endpoint of the trajectory
Xe_euler = np.zeros(Nh)
Xe_rk4   = np.zeros(Nh)
for i, h in enumerate(timesteps):
    Xs, Ts = trajectory(x0, t0, Tmax, h, f, euler )
    Xe_euler[i] = Xs[-1]
    Xs, Ts = trajectory(x0, t0, Tmax, h, f, rk4   )
    Xe_rk4[i]   = Xs[-1]

# Analytic solution
Xact = x0*np.exp(Tmax)
# Calculate error, normalised by exact solution
err_euler = np.abs(Xe_euler - Xact) / Xact
err_rk4   = np.abs(Xe_rk4   - Xact) / Xact

fig = plt.figure(figsize = (12,6))
# Plot error as function of timesteps
# And plot straight lines to show slope
l, = plt.plot(timesteps, err_euler, marker = '.', label = 'Euler')
plt.plot(timesteps, 4.0*timesteps**1, '--', c = l.get_color(), label = '$\sim h^1$')
l, = plt.plot(timesteps, err_rk4,   marker = '.', label = 'RK4')
plt.plot(timesteps, 0.15*timesteps**4, '--', c = l.get_color(), label = '$\sim h^4$')

plt.legend(loc = 'lower right', fontsize = 20)
plt.xlabel('Timestep')
plt.ylabel('Normalised error')

# Logscale
plt.xscale('log')
plt.yscale('log')


I am working on the file "Notebooks/ODE.ipynb".
The first task is:
Example 6 - Dormand and Prince 5(4)
This example includes an implementation of the Dormand-Prince method, which is a method that makes one 5th-order and one 4th-order prediction of the next step, using only 7 evaluations of $f(x,t)$, and uses the 5th-order prediction as the estimate of the next point.
Stepsize control is based on a recipe from Solving Ordinary Differential Equations, vol I by Hairer, Nørsett and Wanner, page 167-168 (see link further up). It uses the error and the current timestep to calculate the optimal timestep, i.e., the timestep that would produce an error exactly matching the tolerance. It then sets the sets the timestep a little smaller than this. The result is a quite efficient method where fewer steps are rejected than the simpler approach above.
Can you write Python code for it?


I am working on the file "Notebooks/ODE.ipynb".
The first task is:
Example 8 - Implicit solvers
For stiff equations, the best solution is often to use an implicit solver. In practice, the most common approach is to use a numerical library (see the scipy example below), but we illustrate the principle here, using the implicit Euler method.
Recall that the forward Euler method was
$$ x_{n+1} = x_n + h\cdot f(x_n, t_n).$$
The implicit Euler method (also known as backward Euler) is instead
$$ x_{n+1} = x_n + h\cdot f(x_{n+1}, t_{n+1}).$$
The problem then is of course that $x_{n+1}$ is unknown, and we have to solve an equation to find it. Methods of this kind are known as implicit methods, since $x_{n+1}$ is given implicitly, instead of explicitly. In general, some kind of numerical equation solver is used to find $x_{n+1}$.
In simple cases, such as the above example, we can solve the equation analytically. We have the second order ODE
$$\ddot{x} = \alpha (v(x) - \dot{x}).$$
We use the substitutions $x = x_1$, $\dot{x} = x_2$ to rewrite into a system of two first-order ODEs:
$$\dot{x_1} = x_2,$$
$$\dot{x_2} = \alpha \big(v(x_1) - x_2\big).$$
Inserting into the implicit Euler scheme, we find
$$ x_{1_{n+1}}  = x_{1_n} + h \cdot x_{2_{n+1}}, $$
$$ x_{2_{n+1}}  = x_{2_n} + h \cdot \alpha \big(v(x_{1_{n+1}}) - x_{2_{n+1}}\big).$$
This is a system of equations with two unknowns. Using that $v(x) = -x$, we can solve this system to find
$$ x_{1_{n+1}}  = x_{1_n} + h \cdot \frac{x_{2_n} - \alpha h x_{1_n}}{1 + \alpha h + \alpha h^2}, $$
$$ x_{2_{n+1}}  = \frac{x_{2_n} - \alpha h x_{1_n}}{1 + \alpha h + \alpha h^2}.$$
Can you write Python code for it?

# Implementing the above equations in an implicit scheme
# Note that we are writing this method specifically for
# the problem we are looking at here. It does not work
# for other problems.

def implicit_euler(X, t, h, f):
    # The f isn't used for anything, but keeping
    # the call signature consistent allows us to
    # re-use the earlier trajectory function.
    alpha = 1e3
    x1 = X[0] + h*(X[1]-alpha*h*X[0]) / (1 + alpha*h + alpha*h**2)
    x2 = (X[1]-alpha*h*X[0]) / (1 + alpha*h + alpha*h**2)
    return np.array([x1, x2])

# Initial conditions for f2
x02 = np.array([2, -2])
# Timestep
# Note that the timestep can be orders of magnitude larger
# than what we found above for the explicit schemes.
h2 = 0.2
# Calculate trajectories
X2s, T2s = trajectory(x02, t0, Tmax, h2, f2, implicit_euler)

# Make a figure with two subplots
fig, ax = plt.subplots(nrows = 2, sharex = True, figsize = (12,8))

# Plot numerical solutions of position
ax[0].plot(T2s, X2s[:,0], label = 'f2 Implicit Euler')

# Plot numerical solutions of velocity
ax[1].plot(T2s, X2s[:,1], label = 'f2 Implicit Euler')

# Labels
ax[0].set_ylabel('$x$', fontsize = 24)
ax[1].set_ylabel('$\dot{x}$', fontsize = 24)
ax[1].set_xlabel('$t$', fontsize = 24)
# Add legend
ax[0].legend(fontsize = 16)
ax[1].legend(fontsize = 16)
# Plot limits
ax[0].set_ylim(-3, 3)
ax[1].set_ylim(-3, 3)
ax[1].set_xlim(-0.05, 5)
# Reduce whitespace
plt.tight_layout()


I am working on the file "Notebooks/OSM_Results/OSM Prelim Results.ipynb".
The first task is:
ION_ACTIVITY [ACTIVE] Exemplar molecules that were added to the training set when moving from EC50_200 to EC50_500
Commentary
These molecules have the same Triazole arm as we noticed in the previous notebook when trying to classifiy the molecular ION_ACTIVITY using D840_ACTIVE (DRAGON). This structure is also well represented in the test molecules.
ION_ACTIVITY [ACTIVE] Exemplar molecules that were added to the training set when moving from EC50_500 to EC50_1000
The results of the EC50_500 classification of the test molecules.
Can you write Python code for it?

sorted = test_results.sort_values("M5_500_250", ascending=False)
mols, labels = mol_label_list(sorted)
Draw.MolsToGridImage(mols,legends=labels,molsPerRow=4)


I am working on the file "Notebooks/OSM_Results/OSM Prelim Results.ipynb".
The first task is:
ION_ACTIVITY [ACTIVE] in EC50_200
Can you write Python code for it?

ion_active = ec50_200_active.loc[train_results["ION_ACTIVITY"] == "ACTIVE"].sort_values("EC50")
mols, labels = mol_label_list(ion_active)
Draw.MolsToGridImage(mols,legends=labels,molsPerRow=4)


I am working on the file "Notebooks/OSM_Results/OSM Prelim Results.ipynb".
The first task is:
Where the META MODELs are as follows:

DNN - A Deep Neural Network classifier [16, 32, 32, 16, 2] from the Keras toolkit. Cross-entropy loss function.

NBC - A Naive Bayes Classifier

SVMC - A support vector machine classifier.

LOGC - A Logistic classifier.

Modelling.
The Meta Models run on Linux and Windows under Python 2.7 and 3.5 (Mac untested):

Download (follow the readme setup) the entire directory tree from google drive here: https://github.com/kellerberrin/OSM-QSAR. Detailed instructions will be also posted so that the withheld molecules can be tested against the optimal model with minimum hassle. The pre-trained DRAGON classifier "ION_DRAGON_625.krs" must be in the model directories. In addition, for the "osm" model, the pre-trained meta model "ION_META_40.krs" should be in the "osm" model directory. The software should give sensible error messages if they are missing.

Make sure you have setup and activated the python anaconda environment as described in "readme.md".

For the optimal OSM meta model (--help for flag descriptions) the following cmd was used (the clean flag is optional it removes previous results from the model directory):
$python OSM_QSAR.py --classify osm --load ION_META --epoch 40 --train 0 [--clean]
For the svmc SKLearn meta model (--help for flag descriptions) the following cmd was used (the clean flag is optional it removes previous results from the model directory):
$python OSM_QSAR.py --classify osm_sk [--clean]
We visualize the training set probability maps by normalizing them to the unit interval [0, 1] and sorting them in descending order.
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
from pylab import *
from sklearn.preprocessing import minmax_scale

def sort_map(column):
    array = minmax_scale(train_results[column])
    return array[np.argsort(-array)]

scale = 1.0
fig = plt.figure(num=None, figsize=(8 * scale, 6 * scale), dpi=80, facecolor='w', edgecolor='k')
for map in all_active: plt.plot(sort_map(map), label=map)
xlabel("molecules")
ylabel("normalized probability")
title(" Training Set [ACTIVE] Probability Maps")
legend(loc=1); # upper right corner

def mol_label_list(data_frame):  # Function to produce rdkit mols and associated molecular labels
    id = data_frame["ID"].tolist()
    klass =  data_frame["ACTUAL_500"].tolist()
    potency = data_frame["EC50"].tolist()
    ion_activity = data_frame["ION_ACTIVITY"].tolist()
    map_prob = data_frame["M5_500_250"].tolist()
    labels = []
    for idx in range(len(id)):
        labels.append("{} {} {} {} {:5.0f} ({:5.4f})".format(idx+1, id[idx],
                                                           klass[idx][0], ion_activity[idx][0],
                                                           potency[idx]*1000, map_prob[idx]))
    smiles = data_frame["SMILE"].tolist()
    mols = [Chem.MolFromSmiles(smile) for smile in smiles]
    return mols, labels

from rdkit import Chem
from rdkit.Chem import Draw
from rdkit.Chem.Draw import IPythonConsole
from rdkit import rdBase
IPythonConsole.ipython_useSVG=True


I am working on the file "Notebooks/Old/phacking.ipynb".
The first task is:
Large sample on small effect sizes
When you have a large sample size, you can find small effects even when there is a null effect.
The effect size Cohen's d is calculated by:
$ d = \frac{\text{mean difference}}{\text{standard deviation}} = \frac{\text{M2 - M1}}
{\frac{\sqrt{Var1+Var2}}{2} }$
Interpretation of Cohen's d:
.2 = small effect
.5 = moderate effect
.8 and above = large effect
Can you write Python code for it?

np.random.seed(12345678)
maxrep = 6000
n = 2
stop,ps = 0,[]
while stop==0:
    rvs1 = stats.norm.rvs(loc=5,scale=5,size=n)
    rvs2 = stats.norm.rvs(loc=4.5,scale=5,size=n)
    t,p = stats.ttest_ind(rvs1,rvs2) # ttest1.
    ps.append(p)
    # print "n = %i, mdiff = %.2f, t = %.2f, p = %.5f" % (n,rvs1.mean()-rvs2.mean(), t,p)
    n += 1
    if n == maxrep:
        stop = 1
ps= np.array(ps)
x = np.array(range(2,maxrep))
plt.figure(figsize=(12,6))
plt.title("Progression of p-values")
plt.xlabel("Sample size: number of participants in each condition")
plt.ylabel("p-values")
plt.plot(x,ps,'o-')
plt.plot(x,np.repeat(0.05,len(x)),'r--')
plt.show()

cohens_d = (np.mean(rvs1) - np.mean(rvs2)) / (np.sqrt((np.std(rvs1) ** 2. + np.std(rvs2) ** 2.) / 2.))
print 'Effect size : %.2f' %cohens_d


I am working on the file "Notebooks/Old/phacking.ipynb".
The first task is:
Once again, if the means are truly different, you will see a more consistent p-value across experiments.
Can you write Python code for it?

np.random.seed(12345678)
maxrep = 101
n = 32
trial,ps = 1,[]
while trial < 101:
    rvs1 = stats.norm.rvs(loc=5,scale=10,size=n)
    rvs2 = stats.norm.rvs(loc=12,scale=10,size=n)
    t,p = stats.ttest_ind(rvs1,rvs2) # ttest1.
    ps.append(p)
    # print "n = %i, mdiff = %.2f, t = %.2f, p = %.5f" % (n,rvs1.mean()-rvs2.mean(), t,p)
    trial +=1
ps= np.array(ps)
x = np.array(range(1,maxrep))
plt.figure(figsize=(12,6))
plt.title("Progression of p-values")
plt.xlabel("Experiment number")
plt.ylabel("p-values")
plt.plot(x,ps,'o-')
plt.plot(x,np.repeat(0.05,len(x)),'r--')
plt.show()


I am working on the file "Notebooks/Old/phacking.ipynb".
The first task is:
We can see that if you have a sufficiently large sample size, you are able to detect even the smallest of effects.
Can you write Python code for it?

from scipy.stats.stats import pearsonr

d1 = np.random.rand(20)
d2 = d1 * .3
pearsonr(d1,d2)

# The desired covariance matrix.
r = np.array([
        [  3.0, 1],
        [ 1,  3.0]
    ])
y = np.random.multivariate_normal([0,0], r, size=10000)

pearsonr(y[:,1],y[:,0])

?np.random.multivariate_normal

np.random.seed(12345678)
maxrep = 100000
ns = [20,40,60,80,100]
rs = [.3, .5, .7]

for n in ns:
    print n
    for rep in range(0,maxrep):

#         rvs1 = stats.norm.rvs(loc=5,scale=10,size=n)
#         rvs2 = stats.norm.rvs(loc=5,scale=10,size=n)
#         t,p = stats.ttest_ind(rvs1,rvs2) # ttest1.
#         ps.append(p)
#         # print "n = %i, mdiff = %.2f, t = %.2f, p = %.5f" % (n,rvs1.mean()-rvs2.mean(), t,p)
#         n += 1
#         if n == maxrep:
#             stop = 1


I am working on the file "Notebooks/OneOffs/Quyen_proteases.ipynb".
The first task is:
TPA protease
Can you write Python code for it?


I am working on the file "Notebooks/OneOffs/Quyen_proteases.ipynb".
The first task is:
elastases
Can you write Python code for it?

g = ['CELA1','CELA2A','CELA2B','CELA3A','CELA3B','CTRC','ELANE','MMP12']

paired_bp_tn_split?

fig, axs = subplots(8, 1, figsize=(15,20), sharex=True)
for i,gene in enumerate(g):
    paired_bp_tn_split(matched_rna.ix[gene], codes, ax=axs[i],
                      data_type='')


I am working on the file "Notebooks/PCAonSyntheticData.ipynb".
The first task is:
4. Sklearn PCA
Can you write Python code for it?

#define number of principal components
sklearn_pca = sklearnPCA(n_components=9)

#matrix with each sample in terms of the PCs
SkPC = sklearn_pca.fit_transform(featurematrix_std)

#covariance matrix
Skcov = sklearn_pca.get_covariance()

#score matrix
#Skscore = sklearn_pca.score_samples(featurematrix_std)

#explained variance
Skvariance = sklearn_pca.explained_variance_
Skvarianceratio = sklearn_pca.explained_variance_ratio_

Skvarianceratio

Skvariance


I am working on the file "Notebooks/PCAonSyntheticData.ipynb".
The first task is:
5. Matrix decomposition (see A User's Guide to Principal Components by Jackson, 1991.)
U’SU  = L
U = orthonormal matrix, characteristic vectors
S = covariance matrix
L = diagonal matrix, characteristic roots
Get characteristic roots: |𝑺−𝒍𝑰|=𝟎
Get characteristic vector: |𝑺−𝒍𝑰|𝒕𝒊=𝟎
The projection of sample n onto principal component i: z$i$ = u$^{’}{i}$[x${n}$-x${avg}$]
Can you write Python code for it?

mean_vec = np.mean(featurematrix_std, axis=0)

#need to take transpose, since rowvar = true by default
cov_mat = np.cov(featurematrix_std.T)

#solve for characteristic roots and vectors
eig_vals, eig_vecs = np.linalg.eig(cov_mat)

#check that the loadings squared sum to 1:
Lsquared = sum(eig_vecs**2)


I am working on the file "Notebooks/PCAonSyntheticData.ipynb".
The first task is:
6. Matlab's PCA
Can you write Python code for it?

mlPCA = PCA(featurematrix_std)
#get projections of samples into PCA space
mltrans = mlPCA.Y
#reshape
mltransreshape = mltrans.reshape((256,256,9))
mlloadings = mlPCA.Wt
#mltrans[513,:] should be the same as mltransreshape[2,1,:]
mlloadings.shape


I am working on the file "Notebooks/PCAonSyntheticData.ipynb".
The first task is:
Check that all three give similar results
Can you write Python code for it?

#projection of first sample, on to the first PC
P11 = np.dot(eig_vecs[:,0], featurematrix_std[0,:]-mean_vec)
mlP11 = mlPCA.Y[0,0]
SkP11 = SkPC[0,0]

P12 = np.dot(eig_vecs[:,1], featurematrix_std[0,:]-mean_vec)
mlP12 = mlPCA.Y[0,1]
SkP12 = SkPC[0,1]

P152 = np.dot(eig_vecs[:,1], featurematrix_std[15,:]-mean_vec)
mlP152 = mlPCA.Y[15,1]
SkP152 = SkPC[15,1]

print(P11, mlP11, SkP11)
print(P12, mlP12, SkP12)
print(P152, mlP152, SkP152)

print(mlloadings[0,7])
print(eig_vecs[0,7])


I am working on the file "Notebooks/Paragraph2Mem.ipynb".
The first task is:
Define some constant variables

Chagne path to your books!
all_books : Path to the 7 seven books used to mine meomories
coolors & bins : Defining some hisotrgram constants
docs : List of pensieve.Doc's, one for each book
Can you write Python code for it?

all_books = ['../../clusterpot/book1.txt']
#               '../../clusterpot/book2.txt',
#               '../../clusterpot/book3.txt',
#               '../../clusterpot/book4.txt',
#               '../../clusterpot/book5.txt',
#               '../../clusterpot/book6.txt',
#               '../../clusterpot/book7.txt']
#all_books = ['../../clusterpot/book1.txt']

colors = ['black', 'red', 'green', 'blue', 'cyan', 'yellow','magenta']
bins = [ 5*i for i in range(0,201) ]
docs = []
for book in all_books:
    docs.append(pens.Doc(book))


I am working on the file "Notebooks/Paragraph2Mem.ipynb".
The first task is:
Generating memory text
With the previous information we can define meaningful paragraphs and sentences. We can iterate over paragraphs, selecting those with nHarry's/sentence > 0.4 and construct memories:
* Find textacy main verbs and select meaningful verbs (documenta wise freq. < 50) and find their subjects and objects
* Get verb auxiliaries and check if there is necesary negation
Can you write Python code for it?

max = 100
def para_2_mem(p,counter):
    main_verbs = textacy.spacy_utils.get_main_verbs_of_sent(p.doc)
    results = []
    for verb in main_verbs:
        my_string = ''
        if p.words['verbs'][verb.text] < 50:
            # --- Getting aux verbs
            span = textacy.spacy_utils.get_span_for_verb_auxiliaries(verb)
            complex_verb = p.doc[span[0]].text
            span_end = 1
            if textacy.spacy_utils.is_negated_verb(verb) is True:
                complex_verb = complex_verb+' not '
                span_end = 0
            for a in range(span[0]+1,span[1]+span_end):
                complex_verb +=" "+p.doc[span[1]].text
            # ---
            subjects = textacy.spacy_utils.get_subjects_of_verb(verb)
            objects = textacy.spacy_utils.get_objects_of_verb(verb)
            if len(subjects)>0 and len(objects)>0:
                results.append([subjects,complex_verb,objects])

                #if counter < max:
                    #print(subjects,complex_verb,objects)

    #print(" ------------------------------- ")
    return results


I am working on the file "Notebooks/Paragraph2Mem.ipynb".
The first task is:
Harry paragraphs
We define the harryness of a paragraph as the number of instances of 'Harry' and 'Harry'/sentence in a paragraph
Can you write Python code for it?

def harry_paragraphs( d ):
    x = []
    y= []
    par = []
    par_weight = {}
    for i, p in enumerate(d.paragraphs):
        count = 1
        for sent in p.doc.sents:
            #print(sent.text)
            count+=1
        #lines.append(count)
    #print(sent)
        harryness = p.words['names']['Harry']
        harryness_per_sent = p.words['names']['Harry']/count #+ p.words['names']['Potter']
        #+ p.words['names']['Potter'] + p.words['names']['Harry Potter']
        if len(p.text) > 0:
            #print(harryness/len(p.text))
            if harryness >=1:
                x.append(harryness)
                y.append(harryness_per_sent)
            if harryness >= 3:
                par.append(i)
                par_weight[i] = harryness
    return x,y


I am working on the file "Notebooks/Paragraph2Mem.ipynb".
The first task is:
Next we plot frequency and frequency per sentence per paragraph
Can you write Python code for it?

fig2 = plt.figure()
ax = fig2.add_subplot(111)
ax.set_xlabel('Harry appearance frequency')
plt.hist(histogram)#, color=colors)
# for i, h in enumerate(histogram):
#     lab = "book"+str(i)
#     plt.hist(h,color = colors[i],label = lab)
plt.yscale('log')
ax.legend(bbox_to_anchor=(0, 0, 1, 1))
plt.show()
#print(len(par))

fig3 = plt.figure()
ax = fig3.add_subplot(111)
ax.set_xlabel('Harry appearance frequency per sentence in paragraph')
# for i, h in enumerate(hist_per_sent):
#     lab = "book"+str(i)
#     plt.hist(h,color = colors[i],label = lab)
plt.hist(hist_per_sent)
plt.yscale('log')
ax.legend(bbox_to_anchor=(0, 0, 1, 1))
plt.show()


I am working on the file "Notebooks/Paragraph2Mem.ipynb".
The first task is:
QA Histograms
We process the loaded pensieve docs. Some basic QA histograms are filled and we plot the # of instances of the different 'verbs' document wise. This will be used to select the significant verbs in each paragraph (appearance freq. < 50)
Can you write Python code for it?

histogram = []
hist_per_sent = []
words_hist = []

for i, d in enumerate(docs):
    print('Reading ',d)
    h, h_per_line = harry_paragraphs(d)
    histogram.append(h)
    hist_per_sent.append(h_per_line)
    for verb in d.words['verbs']:
        words_hist.append(d.words['verbs'][verb])

fig = plt.figure()
ax = fig.add_subplot(111)
ax.set_xlabel('Appearance frequency')
ax.set_title('All names')
#
plt.hist(words_hist,bins)
plt.yscale('log')
#
plt.show()


I am working on the file "Notebooks/Paragraph2Mem.ipynb".
The first task is:
Quick test on documents as proof of principal
Can you write Python code for it?

# Will only print out the first processed paragraphs for QA

def doc2mem(docs):
    counter = 0
    passed = 0
    for p in docs.paragraphs:
        #
        count = 1
        for sent in p.doc.sents:
            count+=1
        #
        if p.words['names']['Harry']/count >= 0.9 :
            #
            passed +=1
            print(" Sentences ",counter," \n",p.text)
            print(" ------------------------------- ")
            log = ''
            for key in p.words['places']:
                log+=key+' '
            #print("Places: ", log)
            #print(" ------------------------------- ")
            log = ''
            for key in p.words['names']:
                log+=key+' '
            print("People: ", log)
            print(" ------------------------------- ")
            print('Actions \n ')
            #
            log = para_2_mem(p,counter)

            for i_log in log:
                print(i_log)
                #print(log)
            counter+=1
            print(" ------------------------------- ")
    return passed

counter = 0
passed = 0
for d in docs:
    passed+=doc2mem(d)
#    print(len(d.paragraphs))
    #passed+=doc2mem(d)

print(passed)


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Are these 0 values possible?
How many rows have have unexpected 0 values?
Can you write Python code for it?

print("# rows in dataframe {0}".format(len(df)))
print("# rows missing glucose_conc: {0}".format(len(df.loc[df['glucose_conc'] == 0])))
print("# rows missing diastolic_bp: {0}".format(len(df.loc[df['diastolic_bp'] == 0])))
print("# rows missing thickness: {0}".format(len(df.loc[df['thickness'] == 0])))
print("# rows missing insulin: {0}".format(len(df.loc[df['insulin'] == 0])))
print("# rows missing bmi: {0}".format(len(df.loc[df['bmi'] == 0])))
print("# rows missing diab_pred: {0}".format(len(df.loc[df['diab_pred'] == 0])))
print("# rows missing age: {0}".format(len(df.loc[df['age'] == 0])))


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Change diabetes from boolean to integer, True=1, False=0
Can you write Python code for it?

diabetes_map = {True : 1, False : 0}
df['diabetes'] = df['diabetes'].map(diabetes_map)


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Check for additional correlations
Can you write Python code for it?


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Check for null values
Can you write Python code for it?


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Correlated Feature Check
Helper function that displays correlation by color.  Red is most correlated, Blue least.
Can you write Python code for it?

def plot_corr(df, size=11):
    """
    Function plots a graphical correlation matrix for each pair of columns in the dataframe.

    Input:
        df: pandas DataFrame
        size: vertical and horizontal size of the plot

    Displays:
        matrix of correlation between columns.  Blue-cyan-yellow-red-darkred => less to more correlated
                                                0 ------------------>  1
                                                Expect a darkred line running from top left to bottom right
    """

    corr = df.corr()    # data frame correlation function
    fig, ax = plt.subplots(figsize=(size, size))
    ax.matshow(corr)   # color code the rectangles by correlation value
    plt.xticks(range(len(corr.columns)), corr.columns)  # draw x tick marks
    plt.yticks(range(len(corr.columns)), corr.columns)  # draw y tick marks

plot_corr(df)

df.corr()

df.head(5)


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Data has 0 in places it should not.
Just like test or test datasets we will use imputation to fix this.
Can you write Python code for it?

#Impute with mean all 0 readings
from sklearn.impute import SimpleImputer
fill_0 = SimpleImputer(missing_values=0, strategy="mean")
pd.options.mode.chained_assignment = None
X_predict_cols_not_num_preg = X_predict.columns.difference(['num_preg']) # do not impute num_preg column
X_predict[X_predict_cols_not_num_preg] = fill_0.fit_transform(X_predict[X_predict_cols_not_num_preg])
X_predict


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Definition of features
From the metadata on the data source we have the following definition of the features.
| Feature  | Description | Comments |
|--------------|-------------|--------|
| num_preg     | number of pregnancies         | 0 is valid
| glucose_conc | Plasma glucose concentration a 2 hours in an oral glucose tolerance test         |
| diastolic_bp | Diastolic blood pressure (mm Hg) |
| thickness | Triceps skin fold thickness (mm) |
|insulin | 2-Hour serum insulin (mu U/ml) |
| bmi |  Body mass index (weight in kg/(height in m)^2) |
| diab_pred |  Diabetes pedigree function |
| Age (years) | Age (years)|
| skin | ???? | What is this? |
| diabetes | Class variable (1=True, 0=False) |  Why is our data boolean (True/False)? |
Check for null values
Can you write Python code for it?


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Good distribution of true and false cases.  No special work needed.
Spliting the data
70% for training, 30% for testing
Can you write Python code for it?

#from sklearn.cross_validation import train_test_split
from sklearn.model_selection import train_test_split
feature_col_names = ['num_preg', 'glucose_conc', 'diastolic_bp', 'thickness', 'insulin', 'bmi', 'diab_pred', 'age']
predicted_class_names = ['diabetes']

X =  df[feature_col_names]   # predictor feature columns (8 X m)
y = df[predicted_class_names] # predicted class (1=true, 0=false) column (1 X m)

split_test_size = 0.30

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=split_test_size, random_state=42)
                            # test_size = 0.3 is 30%, 42 is the answer to everything


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Loading and Reviewing the Data
Can you write Python code for it?

# COLAB VERSION ONLY - load file from user's computer
#from google.colab import files
#uploaded = files.upload()
# once uploaded, read the files contents into the dataframe
#import io
#df = pd.read_csv(io.BytesIO(uploaded['pima-data.csv']))

# regular version pointing to file on users system
df = pd.read_csv("./data/pima-data.csv")

df.shape

df.head(5)

df.tail(5)


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Logisitic regression with class_weight='balanced'
Can you write Python code for it?

C_start = 0.1
C_end = 5
C_inc = 0.1

C_values, recall_scores = [], []

C_val = C_start
best_recall_score = 0
while (C_val < C_end):
    C_values.append(C_val)
    lr_model_loop = LogisticRegression(C=C_val, class_weight="balanced", random_state=42, solver='liblinear', max_iter=10000)
    lr_model_loop.fit(X_train, y_train.values.flatten())
    lr_predict_loop_test = lr_model_loop.predict(X_test)
    recall_score = metrics.recall_score(y_test, lr_predict_loop_test)
    recall_scores.append(recall_score)
    if (recall_score > best_recall_score):
        best_recall_score = recall_score
        best_lr_predict_test = lr_predict_loop_test

    C_val = C_val + C_inc

best_score_C_val = C_values[recall_scores.index(best_recall_score)]
print("1st max value of {0:.3f} occured at C={1:.3f}".format(best_recall_score, best_score_C_val))

%matplotlib inline
plt.plot(C_values, recall_scores, "-")
plt.xlabel("C value")
plt.ylabel("recall score")

from sklearn.linear_model import LogisticRegression
lr_model =LogisticRegression( class_weight="balanced", C=best_score_C_val, random_state=42, solver='liblinear')
lr_model.fit(X_train, y_train.values.flatten())
lr_predict_test = lr_model.predict(X_test)

# training metrics
print("Accuracy: {0:.4f}".format(metrics.accuracy_score(y_test, lr_predict_test)))
print(metrics.confusion_matrix(y_test, lr_predict_test) )
print("")
print("Classification Report")
print(metrics.classification_report(y_test, lr_predict_test))
print(metrics.recall_score(y_test, lr_predict_test))


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Logistic Regression
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression

lr_model =LogisticRegression(C=0.7, random_state=42, solver='liblinear', max_iter=10000)
lr_model.fit(X_train, y_train.values.flatten()) #.ravel())
lr_predict_test = lr_model.predict(X_test)

# training metrics
print("Accuracy: {0:.4f}".format(metrics.accuracy_score(y_test, lr_predict_test)))
print(metrics.confusion_matrix(y_test, lr_predict_test) )
print("")
print("Classification Report")
print(metrics.classification_report(y_test, lr_predict_test))


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
No obvious null values.
Check class distribution
Rare events are hard to predict
Can you write Python code for it?

num_obs = len(df)
num_true = len(df.loc[df['diabetes'] == 1])
num_false = len(df.loc[df['diabetes'] == 0])
print("Number of True cases:  {0} ({1:2.2f}%)".format(num_true, (num_true/num_obs) * 100))
print("Number of False cases: {0} ({1:2.2f}%)".format(num_false, (num_false/num_obs) * 100))


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Performance on Testing Data
Can you write Python code for it?

# predict values using the testing data
nb_predict_test = nb_model.predict(X_test)

from sklearn import metrics

# training metrics
print("nb_predict_test", nb_predict_test)
print ("y_test", y_test)
print("Accuracy: {0:.4f}".format(metrics.accuracy_score(y_test, nb_predict_test)))


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Performance on Training Data
Can you write Python code for it?

# predict values using the training data
nb_predict_train = nb_model.predict(X_train)

# import the performance metrics library
from sklearn import metrics

# Accuracy
print("Accuracy: {0:.4f}".format(metrics.accuracy_score(y_train, nb_predict_train)))
print()


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Post-split Data Preparation
Hidden Missing Values
Can you write Python code for it?


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Predict Test Data
Can you write Python code for it?

rf_predict_test = rf_model.predict(X_test)

# training metrics
print("Accuracy: {0:.4f}".format(metrics.accuracy_score(y_test, rf_predict_test)))

print(metrics.confusion_matrix(y_test, rf_predict_test) )
print("")
print("Classification Report")
print(metrics.classification_report(y_test, rf_predict_test))


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Predict Training Data
Can you write Python code for it?

rf_predict_train = rf_model.predict(X_train)
# training metrics
print("Accuracy: {0:.4f}".format(metrics.accuracy_score(y_train, rf_predict_train)))


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Predict on Test data
Can you write Python code for it?

lr_cv_predict_test = lr_cv_model.predict(X_test)

# training metrics
print("Accuracy: {0:.4f}".format(metrics.accuracy_score(y_test, lr_cv_predict_test)))
print(metrics.confusion_matrix(y_test, lr_cv_predict_test) )
print("")
print("Classification Report")
print(metrics.classification_report(y_test, lr_cv_predict_test))


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
The correlations look good.  There appear to be no coorelated columns.
Mold Data
Data Types
Inspect data types to see if there are any issues.  Data should be numeric.
Can you write Python code for it?


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
The skin and thickness columns are correlated 1 to 1.  Dropping the skin column
Can you write Python code for it?


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
The truncated file contained 4 rows from the original CSV.
Data is the same is in same format as the original CSV file's data.  Therefore, just like the original data, we need to transform it before we can make predictions on the data.
Note: If the data had been previously "cleaned up" this would not be necessary.
We do this by executed the same transformations as we did to the original data
Start by dropping the "skin" which is the same as thickness, with different units.
Can you write Python code for it?


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Training Initial Algorithm - Naive Bayes
Can you write Python code for it?

from sklearn.naive_bayes import GaussianNB

# create Gaussian Naive Bayes model object and train it with the data
nb_model = GaussianNB()

nb_model.fit(X_train, y_train.values.flatten())


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
Verifying predicted value was split correctly
Can you write Python code for it?

print("Original True  : {0} ({1:0.2f}%)".format(len(df.loc[df['diabetes'] == 1]), (len(df.loc[df['diabetes'] == 1])/len(df.index)) * 100.0))
print("Original False : {0} ({1:0.2f}%)".format(len(df.loc[df['diabetes'] == 0]), (len(df.loc[df['diabetes'] == 0])/len(df.index)) * 100.0))
print("")
print("Training True  : {0} ({1:0.2f}%)".format(len(y_train[y_train[:] == 1]), (len(y_train[y_train[:] == 1])/len(y_train) * 100.0)))
print("Training False : {0} ({1:0.2f}%)".format(len(y_train[y_train[:] == 0]), (len(y_train[y_train[:] == 0])/len(y_train) * 100.0)))
print("")
print("Test True      : {0} ({1:0.2f}%)".format(len(y_test[y_test[:] == 1]), (len(y_test[y_test[:] == 1])/len(y_test) * 100.0)))
print("Test False     : {0} ({1:0.2f}%)".format(len(y_test[y_test[:] == 0]), (len(y_test[y_test[:] == 0])/len(y_test) * 100.0)))


I am working on the file "Notebooks/Pima-Prediction-with-reload.ipynb".
The first task is:
We need to drop the diabetes column since that is what we are predicting.
Store data without the column with the prefix X as we did with the X_train and X_test to indicate that it contains only the columns we are prediction.
Can you write Python code for it?

X_predict = df_predict
del X_predict['diabetes']


I am working on the file "Notebooks/Placing convergent events in phylogenies.ipynb".
The first task is:
Entire function to place convergent events in a tree and output the parameters for bppsuite.
Can you write Python code for it?

def placeTransitionsAndGetBppSeqGenCommand(numTransitions, tree, C1, C2, Nch):
    reNumberNodes(tree)
    nodesWithTransitions,convergentModel = randomTransitionsWithHypergeometricDistribution(numTransitions, tree)
    return( getBppSeqGenCommandFromNodesWithTransitions(convergentModel, C1, C2, Nch) )

# We try the function on the large tree simulated at the beginning
placeTransitionsAndGetBppSeqGenCommand(10, tl, 1, 2, 1000)


I am working on the file "Notebooks/Placing convergent events in phylogenies.ipynb".
The first task is:
Now we create a function to place the n events of transition.
We don't care about branch lengths, meaning that we decide that a transition is not more likely on a long branch than on a short one.
Can you write Python code for it?

# We could use some dynamic programming to be able to generate paths that yield n transitions exactly.
# INstead we randomly generate transitions on the tree until we get the desired number.
# We have two states: ancestral (0) and convergent (1).
# We count the numbers of transitions
def randomTransitions(numTransitions, tree):
    numberOfNodes = len(tree.get_tree_root().get_descendants()) + 1
    rate = float(numTransitions)/float(numberOfNodes)
    ancestralTransition=dict()
    totalNumberOfTransitions = 0
    nodesWithTransitions = list()
    for node in tree.traverse("levelorder"):
        if node.is_root() :
            ancestralTransition[node] = False
        elif ( ancestralTransition[node.up] == True):
            ancestralTransition[node] = True
        else :
            sisterHasAlreadyTransitioned=False
            if ancestralTransition.__contains__(node.get_sisters()[0]): #Here we assume binary trees!
                sisterHasAlreadyTransitioned=True
            #randomly draw whether we do a transition or not
            transitionBool = stats.bernoulli.rvs(rate, size=1)[0] == 1
            if (transitionBool and not sisterHasAlreadyTransitioned):
                ancestralTransition[node] = True
                nodesWithTransitions.append(node)
                totalNumberOfTransitions = totalNumberOfTransitions + 1
            else:
                ancestralTransition[node] = False
    return nodesWithTransitions, totalNumberOfTransitions, ancestralTransition

def placeNTransitionsInTree(numTransitions, tree):
    observedNumTransitions = 2*numTransitions
    nodesWithTransitions = list()
    numTries = 0
    convergentModel = dict()
    while observedNumTransitions != numTransitions and numTries < 100:
        observedNumTransitions = 0
        nodesWithTransitions, observedNumTransitions, convergentModel = randomTransitions(numTransitions, tree)
        print ("Observed Number of Transitions: "+ str(observedNumTransitions ) + " compared to "+ str(numTransitions) +...


I am working on the file "Notebooks/Placing convergent events in phylogenies.ipynb".
The first task is:
Now we want to output the tree and the command line for bppseqgen and bppml.
Can you write Python code for it?

# convergentModel is the ouput from thefunction that places transitions
# C1 and C2 are two profile numbers
# Nch is the number of characters
def getBppSeqGenCommandFromNodesWithTransitions(convergentModel, C1, C2, Nch):
    #First, get the nodes with the convergent model and the nodes with the ancestral model.
    nodesWithConvergentModel = list()
    nodesWithAncestralModel = list()
    for k,v in convergentModel.items():
        if v == True:
            nodesWithConvergentModel.append(k.ND)
        if v == False:
            nodesWithAncestralModel.append(k.ND)

    n1="\""+ str(nodesWithAncestralModel[0])
    for n in nodesWithAncestralModel[1:len(nodesWithAncestralModel)-1]:
        n1 += "," + str(n)
    n1 += "\""
    n2="\""+ str(nodesWithConvergentModel[0])
    for n in nodesWithConvergentModel[1:len(nodesWithConvergentModel)-1]:
        n2 += "," + str(n)
    n2 += "\""
    #Dummy values:
    command="bppseqgen param=CATseq.bpp mod1Nodes=%s mod2Nodes=%s Nch=%d Ns1=%d Ns2=%d Ne1=%d Ne2=%d"%(n1,n2,Nch,C1,C2,C1,C2)
    return (command)

getBppSeqGenCommandFromNodesWithTransitions(convergentModel, 1, 2, 1000)


I am working on the file "Notebooks/Placing convergent events in phylogenies.ipynb".
The first task is:
Now, let's number the nodes of the tree.
Can you write Python code for it?

def reNumberNodes (tl):
    nodeId = 0
    for n in tl.traverse():
        n.add_features(ND=nodeId)
        if n.name=="":
            n.name = str(nodeId)
        nodeId = nodeId + 1
    return

reNumberNodes(tl)

#Writing in NHX format
tl.write(features=['ND'])
print(tl.get_ascii(show_internal=True))


I am working on the file "Notebooks/Placing convergent events in phylogenies.ipynb".
The first task is:
We have a problem: the random algorithm above does not work well for large numbers of convergent events: it needs to do a large number of trials and errors to get something that works, and often fails.
Therefore we need to use another algorithm.
Improved algorithm
Can you write Python code for it?

# we load a small tree, for testing purpose
t = Tree('((((H,K)D,(F,I)G)B,E)A,((L,(N,Q)O)J,(P,S)M)C);', format=1)

print(t.write(format=1))
print(t.get_ascii(show_internal=True))

#for node in t.traverse("levelorder"):
  # Do some analysis on node
  #print (node.name)

def getCherries(leaves, tree):
    length = len(leaves)
    dist=list()
    leaveslist = list(leaves)
    for i in range(length-1):
        for j in range(i+1,length):
            di = tree.get_distance(leaveslist[i].name, leaveslist[j].name, topology_only=True)
            dist.append([di, leaveslist[i].name, leaveslist[j].name])
    #print(dist)
    cherries = list()
    for d in dist:
        if (d[0] == 1.0):
            cherries.append(d)
    return (cherries)

def placeTransitionOnBranchIfSisterHasNotTransitioned (node, ancestralTransition, nodesWithTransitions):
    if node.is_leaf():
        if ancestralTransition.__contains__(node.get_sisters()[0]) and ancestralTransition[node.get_sisters()[0]]==True:
            print("Problem: sister of node "+node.name+" has transitioned!")
            return
        ancestralTransition[ node ] = True
        nodesWithTransitions.append(node)
        print ("Adding transition on leaf node "+node.name)
        return
    descen = node.get_descendants()
    SisterHasTransitioned = True
    converg = 0
    while (not SisterHasTransitioned):
        converg = np.random.randint( 0, high=len(descen) )
        if ancestralTransition.__contains__(descen[converg].get_sisters()[0] and ancestralTransition[descen[converg].get_sisters()[0]]==True):
            pass
        else:
            SisterHasTransitioned = False
    ancestralTransition[ descen[converg] ] = True
    nodesWithTransitions.append(descen[converg])
    print ("Adding transition on node "+node.name)
    return

def randomTransitionsWithHypergeometricDistribution(numTransitions, tree):
    numberOfNodes = len(tree.get_tree_root().get_descendants()) + 1
    rate = float(numTransitions)/float(numberOfNodes)
    node2leaves =...


I am working on the file "Notebooks/Publish/Custom Filter Banks with OpenCV.ipynb".
The first task is:
And to call this helper:
Can you write Python code for it?

def fdog_filter_kernel(L, sigma, t = 3):
    '''
    K = - (x/(sqrt(2 * pi) * sigma ^3)) * exp(-x^2/2sigma^2), |y| <= L/2, |x| < s * t
    '''
    return _filter_kernel_mf_fdog(L, sigma, t, False)

def gaussian_matched_filter_kernel(L, sigma, t = 3):
    '''
    K =  1/(sqrt(2 * pi) * sigma ) * exp(-x^2/2sigma^2), |y| <= L/2, |x| < s * t
    '''
    return _filter_kernel_mf_fdog(L, sigma, t, True)

gf = gaussian_matched_filter_kernel(15, 5)
fdog = fdog_filter_kernel(15, 5)

# visualize:
show_images([gf, fdog], ["Gaussian", "FDOG"])


I am working on the file "Notebooks/PyDSDExamples.ipynb".
The first task is:
And we can dig down further and see what is in one of these objects. Similar to the defaults read in, this will store the values in the data member, and metadata attached to the object.
Can you write Python code for it?


I am working on the file "Notebooks/PyDSDExamples.ipynb".
The first task is:
As before we can compose these by passing in an ax argument.
Can you write Python code for it?

fig = plt.figure(figsize=(12,6))
ax = plt.subplot(1,2,1)
pydsd.plot.plot_dsd(dsd, ax=ax)

ax = plt.subplot(1,2,2)
pydsd.plot.plot_NwD0(dsd,  ax=ax)
plt.tight_layout()


I am working on the file "Notebooks/PyDSDExamples.ipynb".
The first task is:
As expected, estimators that use polarimetry tend to do much better.
Convective Stratiform Partitioning
Finally we have several algorithms that exist for stratiform partitioning in a variety of situations. Let's look at an applicable ground based one due to Bringi
Can you write Python code for it?

cs = pydsd.partition.cs_partition.cs_partition_bringi_2010(dsd.fields['Nw']['data'], dsd.fields['D0']['data'])


I am working on the file "Notebooks/PyDSDExamples.ipynb".
The first task is:
Finally let's visualize a few more of the calculated fields. We can also look at what all new fields have appeared.
Can you write Python code for it?

dsd.fields.keys()

plt.figure(figsize=(12,12))

plt.subplot(2,2,1)
pydsd.plot.plot_ts(dsd, 'D0', x_min_tick_format='hour')

plt.xlabel('Time(hrs)')
plt.ylabel('$D_0$')
# plt.xlim(5,24)

plt.subplot(2,2,2)
pydsd.plot.plot_ts(dsd, 'Nw', x_min_tick_format='hour')

plt.xlabel('Time(hrs)')
plt.ylabel('$log_{10}(N_w)$')
# plt.xlim(5,24)

plt.subplot(2,2,3)
pydsd.plot.plot_ts(dsd, 'Dmax', x_min_tick_format='hour')

plt.xlabel('Time(hrs)')
plt.ylabel('Maximum Drop Size')
# plt.xlim(5,24)

plt.subplot(2,2,4)
pydsd.plot.plot_ts(dsd, 'mu', x_min_tick_format='hour')

plt.xlabel('Time(hrs)')
plt.ylabel('$\mu$')

plt.tight_layout()

plt.show()


I am working on the file "Notebooks/PyDSDExamples.ipynb".
The first task is:
Note the fit submodule has alternative algorithms for calculating various DSD parameter fits.
Radar Equivalent Scattering
We can calculate radar equivalent parameters as well. We use the PyTMatrix library under the hood for this. Let's look at what these measurements would look like if we did T-Matrix scatttering at X-band, which is the default.
Can you write Python code for it?


I am working on the file "Notebooks/PyDSDExamples.ipynb".
The first task is:
Note this updates the parameters, but does not re-scatter the fields until you ask it to for computational reasons. Let's do that now while also changing the DSR we are using, and the maximum diameter we will scatter for.
Can you write Python code for it?

dsd.calculate_radar_parameters(dsr_func = pydsd.DSR.bc, max_diameter=7)


I am working on the file "Notebooks/PyDSDExamples.ipynb".
The first task is:
Now PyDSD will calculate various parameters and store these on the dsd object in the fields dictionary.
Can you write Python code for it?


I am working on the file "Notebooks/PyDSDExamples.ipynb".
The first task is:
Now we can plot these variables up using our pydsd.plot.plot_ts function as before.
Can you write Python code for it?

plt.figure(figsize=(12,12))

plt.subplot(2,2,1)
pydsd.plot.plot_ts(dsd, 'Zh', x_min_tick_format='hour')

plt.xlabel('Time(hrs)')
plt.ylabel('Reflectivity(dBZ)')
# plt.xlim(5,24)

plt.subplot(2,2,2)
pydsd.plot.plot_ts(dsd, 'Zdr', x_min_tick_format='hour')

plt.xlabel('Time(minutes)')
plt.ylabel('Differential Reflectivity(dB)')
# plt.xlim(5,24)

plt.subplot(2,2,3)
pydsd.plot.plot_ts(dsd, 'Kdp', x_min_tick_format='hour')

plt.xlabel('Time(hrs)')
plt.ylabel('Specific Differential Phase(deg/km)')
# plt.xlim(5,24)

plt.subplot(2,2,4)
pydsd.plot.plot_ts(dsd, 'Ai', x_min_tick_format='hour')

plt.xlabel('Time(hrs)')
plt.ylabel('Specific Attenuation')
# plt.xlim(5,24)

plt.tight_layout()

plt.show()


I am working on the file "Notebooks/PyDSDExamples.ipynb".
The first task is:
Now we can use some built in plotting functions to examine these. The first plot type is a time series plot of arbitrary 1D parameters. For example we can plot the D0 variable corresponding to median drop diameter. The plots know how to handle time and various pieces of metadata. Note that you can pass through arguments accepted by plt.plot and we will pass these on so you can customize the plot.
Can you write Python code for it?


I am working on the file "Notebooks/PyDSDExamples.ipynb".
The first task is:
Plotting routines usually take in extra arguments to help customize the plot. Please check the documentation for the full list of arguments. We'll revisit plotting in a bit once we've calculated a few more interesting parameters.
Depending on the type of Disdrometer, the drop spectra may be stored. PyDSD has some capabilities for working with spectra data including filtering, and reducing this back down to a DSD measurement.
TODO: Examples of spectra processing
DSD Estimation
PyDSD has routines implemented for calculating various parameterizations of the Drop Size Distribution. A default set of these can be called using the DropSizeDistribution.calculate_dsd_parameterization class function. Let's do this and see what we get.
Can you write Python code for it?


I am working on the file "Notebooks/PyDSDExamples.ipynb".
The first task is:
Rain Rate estimators
PyDSD has support for some fairly simple built in rain rate estimators for each of the polarimetric variables. Let's calculate a few of these and see how well they work out.
TODO: Add support for storing these on the object.
TODO: Add better built in plotting support for these.
Can you write Python code for it?

(r_z_a, r_z_b), opt = dsd.calculate_R_Zh_relationship()
print(f'RR(Zh) = {r_z_a} Zh **{r_z_b}')

(r_kdp_a, r_kdp_b), opt = dsd.calculate_R_Kdp_relationship()
print(f'RR(KDP) = {r_kdp_a} KDP **{r_kdp_b}')

(r_zk_a, r_zk_b1, r_zk_b2), opt = dsd.calculate_R_Zh_Kdp_relationship()
print(f'RR(Zh, KDP) = {r_zk_a} Zh **{r_zk_b1} * KDP ** {r_zk_b2}')


I am working on the file "Notebooks/PyDSDExamples.ipynb".
The first task is:
Similar to before these new fields will be added to the DropSizeDistribution object in the fields dictionary.
Can you write Python code for it?


I am working on the file "Notebooks/PyDSDExamples.ipynb".
The first task is:
So at this point we have the drop size distribution read in. Let's start by looking at the format of the dsd object. A full listing of the features can be found at http://josephhardinee.github.io/PyDSD/pydsd.html#module-pydsd.DropSizeDistribution .
Generally data is stored in the fields dictionary. This is a dictionary of dictionaries where each key corresponds to a variable. Inside each of these dictionaries the data is stored in data while other metadata is attached at the top level.
Can you write Python code for it?


I am working on the file "Notebooks/PyDSDExamples.ipynb".
The first task is:
This assumes BC shape relationship, X band, 10C. All of the scattering options are fully configurable.
Can you write Python code for it?


I am working on the file "Notebooks/PyDSDExamples.ipynb".
The first task is:
We can also color code the D0,Nw points to get a better visual understanding of this algorithm.
Can you write Python code for it?


I am working on the file "Notebooks/PyDSDExamples.ipynb".
The first task is:
We can now start to plot and visualize some of this data. PyDSD has several built in plotting functions, and you can always pull the data out yourself for more custom plotting routines. Many built-in plotting routines are available in the pydsd.plot module.
Can you write Python code for it?

pydsd.plot.plot_dsd(dsd)
plt.title('Drop Size Distribution - November 12, 2018')


I am working on the file "Notebooks/PyDSDExamples.ipynb".
The first task is:
We have other standard types of plots built in to make life easier. For instance a normal thing to do is compare the relationship of the median drop diameter, with the normalized intercept parameter (Nw).
Can you write Python code for it?


I am working on the file "Notebooks/PyDisdrometer Examples.ipynb".
The first task is:
Next let's estimate some microphysical parameterizations.
Can you write Python code for it?

dsd.calculate_dsd_parameterization()

plt.figure(figsize=(12,12))

plt.subplot(2,2,1)
plt.plot(dsd.time/60.0, dsd.fields['D0']['data'])
plt.xlabel('Time(hrs)')
plt.ylabel('$D_0$')
plt.xlim(5,24)

plt.subplot(2,2,2)
plt.plot(dsd.time/60.0, np.log10(dsd.fields['Nw']['data']))
plt.xlabel('Time(hrs)')
plt.ylabel('$log_{10}(N_w)$')
plt.xlim(5,24)

plt.subplot(2,2,3)
plt.plot(dsd.time/60.0, dsd.fields['Dmax']['data'])
plt.xlabel('Time(hrs)')
plt.ylabel('Shape parameters')
plt.xlim(5,24)

plt.subplot(2,2,4)
plt.pcolor(dsd.time/60.0, dsd.diameter, np.log10(dsd.Nd.T), vmin=0, vmax=6)
plt.xlabel('Time(hrs)')
plt.ylabel('Dmax')
plt.colorbar()
plt.ylim(0,5) #Zoom in some
plt.xlim(5,24)

plt.tight_layout()

plt.show()


I am working on the file "Notebooks/PyDisdrometer Examples.ipynb".
The first task is:
So at this point we have the drop size distribution read in. NASA strips out rainfall information though.  Let's do some T-Matrix scattering though. This should take a little bit(up to a minute or so on my laptop depending on the file).
Can you write Python code for it?


I am working on the file "Notebooks/Python BootCamp 2017 - A example of a notebook.ipynb".
The first task is:
There is a bug on MatPlotLib that does not allow it to run with Jupyter and some backends. When one tries to do some plot, the following error may appear:
Can you write Python code for it?

import matplotlib.pyplot as plt
plt.plot(x, y)


I am working on the file "Notebooks/Python BootCamp 2017 - Your code pretty and healthy.ipynb".
The first task is:
And if I need to get some more information about this function, I can write:
Can you write Python code for it?


I am working on the file "Notebooks/Python BootCamp 2017 - Your code pretty and healthy.ipynb".
The first task is:
Do a proper name
Look the following method definition:

Muti-word variables and methods should be lower-case and separated by underscore. Ex:
Can you write Python code for it?

number_of_states = 5
def sum_integers(x, y):
    return (x, y)


I am working on the file "Notebooks/Python BootCamp 2017 - Your code pretty and healthy.ipynb".
The first task is:
Here is how the code is supposed to be:
Can you write Python code for it?

for i in range(10):
    x = i * 2
    print x


I am working on the file "Notebooks/Python BootCamp 2017 - Your code pretty and healthy.ipynb".
The first task is:
If the identation size is not respected as well, Python will raise an error. So watch out!
Can you write Python code for it?

for i in range(10):
    x = i * 2
     print x


I am working on the file "Notebooks/Python BootCamp 2017 - Your code pretty and healthy.ipynb".
The first task is:
Leave some space / Self Explanatory Names
Try to avoid nested functions. They save number of lines but the are hard to read.
Also, try to use self-explanatory names for parameters. For example, avoid this:
Can you write Python code for it?

import numpy as np

def SnellLaw(t,n):
    return np.deg(np.asin(np.sin(np.rad(t))/n))


I am working on the file "Notebooks/Python BootCamp 2017 - Your code pretty and healthy.ipynb".
The first task is:
Line length
Avoid code lines with more than 80 character. That's the terminal lenght.
Can you write Python code for it?

import numpy as np

def snells_law(incident_angle, ref_index):

    theta = np.radian(incident_angle)
    n = ref_index

    temp = np.sin(theta) / n
    temp = np.asin(temp)
    out_angle = np.deg(temp)

    return out_angle


I am working on the file "Notebooks/Python BootCamp 2017 - Your code pretty and healthy.ipynb".
The first task is:
Ok. That is easy. But what if my code has hundreds of lines and what if I imported four files instead of one?
Can you write Python code for it?

from file_1 import *
from file_2 import *
print a + x


I am working on the file "Notebooks/Python BootCamp 2017 - Your code pretty and healthy.ipynb".
The first task is:
There are different documentation standards that you or your group may choose to use. The important thing here is to make sure that Sphinx can read it and process it so you are saving your time. Really.
Here are two examples of documentation standards that I already used:
PEP252 DocString Standard
Can you write Python code for it?

def say_hello(name):
    """
    This method simply says hello to the person
    whose name is given as parameter.

    :param name: the name of the person that will be greeted.
    :type name: string
    """
    print "Hello, {0}!".format(name)


I am working on the file "Notebooks/Python BootCamp 2017 - Your code pretty and healthy.ipynb".
The first task is:
This is almost like poetry but it is how we should be writing our codes. And you know what? It is EASY! This notebook will have a quick overview of how to add docstrings and comments to your code and how to write a clear code.
Do the docstring for your life
It is absurdely simple to add docstrings to your python code. That can be done in the beggining of a file or after the definition of a method/function or class. Let me show you some examples.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
My Sample Code

This is such a simple documentation! The only thing I have to do
is to add three single quotes (') or double quotes (") together to
start a docstring block! Then just write down in a few words what
you want to do in your file.

by J. Bond - Feb 31, 202X
"""

And there you go! You added documentation! Yay!
You can do that also if you have the definition of a method.
Can you write Python code for it?

def say_hello(name):
    """
    This method simply says hello to the person whose name is given
    as parameter.
    """
    print "Hello, {0}!".format(name)

say_hello("Bruno")


I am working on the file "Notebooks/Python BootCamp 2017 - Your code pretty and healthy.ipynb".
The first task is:
This is how Google has been used to document their codes and it is also read by Sphinx. You can find more about this style here. The final choice is yours to do but you want to make sure you are the clearest possible.
Why write if others can’t read?
People that work with Python are worried about code readability. That is so true that a document called PEP-8 was written in order to help mortals to get their code clearer and easier to read. The full document is quite short and easy to read and it can be found here. Here are some examples.
Language
It may be very tempting to write comments and documentation in your mother language. But if you are not 10000000% sure that your code will be used by people that speak it, write them in English.
Identation
According to PEP-8, one should always use space instead of usig tabs. The recommended identation size is 4 spaces. This avoid errors when messing up with both identation methods. Just search over the Internet how to setup your favorite editor to keep the identation method always the same (hopefully 4 spaces).
Can you write Python code for it?

for i in range(10):
	x = i * 2
    print x


I am working on the file "Notebooks/Python BootCamp 2017 - Your code pretty and healthy.ipynb".
The first task is:
Three lines. Two files. And we are already screwed. Please, be a nice person and just do a proper import so the reader know what is going on.
Can you write Python code for it?

# file_2.py
import file_1
import file_2
print file_1.x + file_2.a + file_1.y


I am working on the file "Notebooks/Quick-Reference-Guide-NLTK/Quick-Reference-Guide-NLTK.ipynb".
The first task is:
Ex.11. Plot 2 Conditional Frequency Distribution
Can you write Python code for it?

languages = ['Chickasaw', 'English', 'German_Deutsch',
             'Greenlandic_Inuktikut', 'Hungarian_Magyar', 'Ibibio_Efik']

cfd = nltk.ConditionalFreqDist(
              (lang, len(word))
              for lang in languages
              for word in udhr.words(lang + '-Latin1'))

cfd.plot(cumulative=True)


I am working on the file "Notebooks/Quick-Reference-Guide-NLTK/Quick-Reference-Guide-NLTK.ipynb".
The first task is:
Ex.12. Plot 3 Conditional Frequency Distribution
Can you write Python code for it?

names = nltk.corpus.names
names.fileids()
['female.txt', 'male.txt']
male_names = names.words('male.txt')
female_names = names.words('female.txt')
[w for w in male_names if w in female_names]

cfd = nltk.ConditionalFreqDist(
              (fileid, name[-1])
              for fileid in names.fileids()
              for name in names.words(fileid))
cfd.plot()


I am working on the file "Notebooks/Quick-Reference-Guide-NLTK/Quick-Reference-Guide-NLTK.ipynb".
The first task is:
Ex.13. Generate Random Text with Bigram
Can you write Python code for it?

sent = ['In', 'the', 'beginning', 'God', 'created', 'the', 'heaven',
...   'and', 'the', 'earth', '.']
list(nltk.bigrams(sent))

def generate_model(cfdist, word, num=15):
    result = ''
    for i in range(num):
        result += word + ' '
        word = cfdist[word].max()

    print result.strip()

text = nltk.corpus.genesis.words('english-kjv.txt')
bigrams = nltk.bigrams(text)
cfd = nltk.ConditionalFreqDist(bigrams)

cfd['living']
nltk.FreqDist({'creature': 7, 'thing': 4, 'substance': 2, ',': 1, '.': 1, 'soul': 1})
generate_model(cfd, 'living')


I am working on the file "Notebooks/Quick-Reference-Guide-NLTK/Quick-Reference-Guide-NLTK.ipynb".
The first task is:
Ex.15. Words as a Graph
Can you write Python code for it?

def traverse(graph, start, node):
    graph.depth[node.name] = node.shortest_path_distance(start)
    for child in node.hyponyms():
        graph.add_edge(node.name, child.name)
        traverse(graph, start, child)

def hyponym_graph(start):
    G = nx.Graph()
    G.depth = {}
    traverse(G, start, start)
    return G

def graph_draw(graph):
    nx.draw_graphviz(graph,
         node_size = [16 * graph.degree(n) for n in graph],
         node_color = [graph.depth[n] for n in graph],
         with_labels = False)
    plt.show()

dog = wn.synset('dog.n.01')
graph = hyponym_graph(dog)
graph_draw(graph)


I am working on the file "Notebooks/Quick-Reference-Guide-NLTK/Quick-Reference-Guide-NLTK.ipynb".
The first task is:
Ex.2. Common Context
Can you write Python code for it?


I am working on the file "Notebooks/Quick-Reference-Guide-NLTK/Quick-Reference-Guide-NLTK.ipynb".
The first task is:
Ex.3. Dispersion Plot
Can you write Python code for it?

text4.dispersion_plot(["citizens", :"democracy", "freedom", "duties", "America"])


I am working on the file "Notebooks/Quick-Reference-Guide-NLTK/Quick-Reference-Guide-NLTK.ipynb".
The first task is:
Ex.3. Similar tokens
Can you write Python code for it?


I am working on the file "Notebooks/Quick-Reference-Guide-NLTK/Quick-Reference-Guide-NLTK.ipynb".
The first task is:
Ex.7. Filter for Long Tokens
Can you write Python code for it?


I am working on the file "Notebooks/Quick-Reference-Guide-NLTK/Quick-Reference-Guide-NLTK.ipynb".
The first task is:
Ex.8. Collocations
Can you write Python code for it?


I am working on the file "Notebooks/Quick-Reference-Guide-NLTK/Quick-Reference-Guide-NLTK.ipynb".
The first task is:
Ex.9. Conditional Frequency Distribution
Can you write Python code for it?

cfd = nltk.ConditionalFreqDist(
            (genre, word)
            for genre in brown.categories()
            for word in brown.words(categories=genre))
genres = ['news', 'religion', 'hobbies', 'science_fiction', 'romance', 'humor']
modals = ['can', 'could', 'may', 'might', 'must', 'will']
cfd.tabulate(conditions=genres, samples=modals)


I am working on the file "Notebooks/Regression-With-Multiple-Features.ipynb".
The first task is:
Learning Rate - The Intuition
<img src="../Images/gradient-descent-intuition.png" alt="Under- and Overshoot in Learning Size" style="width: 600px;"/>
Can you write Python code for it?

# Run gradient descent and capture the progression of cost values and the ultimate optimal W values
%time W_opt, final_penalty, running_w, running_penalty = gradientDescent(X, y, W_init, num_iters, learning_rate)
# Get the optimal W values and the last few W values and cost values
W_opt, final_penalty, running_w[-5:], running_penalty[-5:]

# How the cost changes as the number of iterations increase
fig, ax = plt.subplots(figsize=(8,5))
ax.plot(np.arange(num_iters), running_penalty, 'g')
ax.set_xlabel('Number of Iterations')
ax.set_ylabel('Cost')
plt.xlim(0,num_iters)
ax.set_title('Cost vs. Iterations Over the Dataset');

# Run gradient descent for a few different values of the learning rate
learning_rates = [0.0001, 0.003, 0.005, 0.01, 0.03, 0.1]
gdResults = [gradientDescent(X, y, W_init, num_iters, learning_rates[i]) for i in range(len(learning_rates))]
#gdResults

# For each learning rate, get the progression of costs
#  for each iteration
penalty_list = [gdResults[i][3] for i in range(len(gdResults))]
penalty_list[0]

# How the cost of the transformation varies with the learning rate
plot_color_list = ['g', 'r', 'c', 'b', 'm', 'y']
fig, ax = plt.subplots(figsize=(12,8))
[ax.plot(np.arange(num_iters), penalty_list[i], plot_color_list[i], label=learning_rates[i]) for i in range(len(learning_rates))]
ax.set_xlabel('Iterations')
ax.set_ylabel('Cost')
ax.legend()
plt.xlim(0,1000)
ax.set_title('Cost vs. Learning Rate');


I am working on the file "Notebooks/Regression-With-Multiple-Features.ipynb".
The first task is:
Let's see how the prediction is sensitive to the number of iterations and the learning rate.
Can you write Python code for it?

# Try different values
n_iters = [10, 100, 1000, 10000]
l_rate = [0.0001, 0.0005, 0.01]

# Keep these the same
sq_ft = 3000
rooms = 5

# get it into the right format to plug into the gradient descent function
combos_list = []
for i in range(len(n_iters)):
    for j in range(len(l_rate)):
        combos_list.append([n_iters[i], l_rate[j]])

# Run gradient descent on all the different combinations
gdResults = [gradientDescent(X, y, \
                             W_init, combos_list[i][0], combos_list[i][1]) for i in range(len(combos_list))]

# Here are the optimal parameter values for these settings
W_values = [gdResults[i][0] for i in range(len(gdResults))]

# Price for each set of optimal values
# Remember we've run the model using rescaled house sizes and number of bedrooms
# So we should scale the inputs down and then scale the prediction up when we're done
size_scaled = (sq_ft - data2.mean()[0])/data2.std()[0]
beds_scaled = (rooms - data2.mean()[1])/data2.std()[1]

# This is our model -- we're just using it here to make a calculation
pred_prices = [(W_values[i][0] * 1) + (W_values[i][1] * size_scaled) + (W_values[i][2] * beds_scaled) \
              for i in range(len(W_values))]

# Convert pred_prices from a list of matrices into a simple list
# This makes it suitable for plotting
prices_out = np.array(pred_prices).reshape(1,-1).squeeze()

#print(combos_list)

# Format and print the result
#[print("Predicted Price: ", '${:6,.0f}'.format(math.ceil(pred_price))) for pred_price in pred_prices]

# Plot the results
fig,ax = plt.subplots(figsize=(12,8))
# Define the number of the bars
x_pos = list(range(len(combos_list)))
# Set the bar labels
bar_labels = combos_list
ax.bar(x_pos, prices_out, width=0.8, color='#FFC222')
# set axes labels and title
plt.ylabel('Predicted Price')
plt.xticks(x_pos, bar_labels, rotation=90)
plt.title('Predicted Price for Different Values of [Iterations, Learning Rate]')
# add a grid
plt.grid()
plt.show()


I am working on the file "Notebooks/Regression-With-Multiple-Features.ipynb".
The first task is:
Rescale/Normalize the Data
Notice that the size of a house in square feet is about 1,000 times the number of bedrooms it has. Similarly, the price of a house is about 100 times the size of the house. This is quite common in many datasets, but when it happens, the iterative method of gradient descent (which we'll use again) becomes inefficient. This is simply a matter of making the computations efficient. To do so, we'll do something called feature normalization.
Can you write Python code for it?

# Normalize the scales in the dataset
# NOTE: the output doesn't have to be rescaled but we've done that
# here anyway
# If you know statistics: What we're doing is rewriting each value
#  in terms of standard deviation units
data2Norm = (data2 - data2.mean()) / data2.std()
data2Norm.head()

# In Orange use the Preprocessor widget

# Let's keep track of the mean and standard deviation of the house sizes,
#  number of bedrooms, and prices in the dataset.
# We'll need these values when we make predictions

# We can get them easily for size, bedrooms and price by using data2.mean()[0], ..., [2]
data2.mean()[0], data2.mean()[1], data2.mean()[2], data2.std()[0], data2.std()[1], data2.std()[2]


I am working on the file "Notebooks/Regression-With-Multiple-Features.ipynb".
The first task is:
Step 1: Visualize the Data
Can you write Python code for it?

# Scatter plot of just the size and price
data2.plot.scatter(x='Size', y='Price', figsize=(12,8));

# Scatter plot of just the number of bedrooms and price
data2.plot.scatter(x='Bedrooms', y='Price', figsize=(12,8));

# Scatter plot of bedrooms and size (the 2 features)
data2.plot.scatter(x='Bedrooms', y='Size', figsize=(12,8));

# From https://matplotlib.org/mpl_toolkits/mplot3d/tutorial.html
# 3D scatter plot of size, bedrooms and price
from mpl_toolkits.mplot3d import Axes3D
threeD = plt.figure(figsize=(12,8)).gca(projection='3d')
threeD.scatter(data2['Bedrooms'], data2['Size'], data2['Price'])
threeD.set_xlabel('Bedrooms', fontsize=14)
threeD.set_ylabel('Size', fontsize=14)
threeD.set_zlabel('Price', fontsize=14)
plt.show()


I am working on the file "Notebooks/Regression-With-Multiple-Features.ipynb".
The first task is:
Step 2b: Identify the Output
The output is the house price. Let's set it up as a variable y.
Can you write Python code for it?

# The output -- the price of a house
# Don't need to normalize the output
#y = data2['Price']
y = data2.iloc[:, cols-1:cols]
# First few house prices in the dataset
y.head()


I am working on the file "Notebooks/Regression-With-Multiple-Features.ipynb".
The first task is:
Step 3: Define the Model
Step 3a: Define the Features
In this case our features are exactly the same as our inputs. We have 2 features: the size of a house in square feet and the number of bedrooms a house has. These are the features encoded in the variables $x_{1}$ and $x_{2}$.
Later we'll see models where the features and the inputs are not one and the same. In these cases the features are constructed by combining inputs in various ways.
Step 3b: Transform the Inputs Into an Output
We're going to stick to a simple model -- exactly the same one as before but now with an additional feature.
$$\hat{y} = w_{0} * x_{0}\ +\ w_{1} * x_{1} +\ w_{2} * x_{2}$$
The scheme for transforming inputs to an output has not changed -- it has more than 1 variable, hence the $x_{2}$ along with the $x_{1}$.
$x_{0}$ is still always 1.
Step 3c: Clarify the Parameters of the Model
$w_{0}$, $w_{1}$, and $w_{2}$ are the parameters of the model. These parameters can each take on an infinite number of values. In other words they are continuous variables. $w_{0}$ is called the intercept or bias value.
With this model, we know exactly how to transform an input into an output -- that is, once the values of the parameters are given.
Let's pick a value of X from the dataset, fix a specific value for $w_{0}$ and $w_{1}$, and see what we get for the value of y.
Specifically, let
$\begin{bmatrix}
w_{0} \
w_{1} \
w_{2}
\end{bmatrix} =
\begin{bmatrix}
-10 \
1 \
0
\end{bmatrix}$
This means $w_{0}$ is -10, $w_{1}$ is 1, and $w_{2}$ is 0.
Step 4: Define the Penalty for Getting it Wrong
Our cost function is exactly the same as it was before for the single variable case.
The only difference from what we had before is the $w_{2}$ and $x_{2}$ are now added because we have 2 features rather than just one. We'll always have the same number of $w$ and $x$ values if you count the feature $x_{0}$ that we "manufacture" and set to always be equal to 1.
We're going to take $\hat{y}$ -- the predicted price -- for every row of the dataset as...
Can you write Python code for it?

# Compute the cost for a given set of W values over the entire dataset
# Get X and y in to matrix form
penalty(X, y, W_init, squaredPenalty)


I am working on the file "Notebooks/Regression-With-Multiple-Features.ipynb".
The first task is:
Step 6: Use the Model and Optimal Parameter Values to Make Predictions
It looks like a learning rate greater than 0.003 is good enough to get our iterative gradient descent to plunge down to arrive at the lowest cost value and stay there.
Let's make some predictions...What is our prediction for a house that is 5,000 square feet in size with 4 bedrooms? Let's plug these in to our model and use the optimal $W$ values we've calculated above.
Can you write Python code for it?

# Change size and num_bedrooms to make distinct predictions
size = 3000
num_bedrooms = 3

# Remember we've run the model using rescaled house sizes and number of bedrooms
# So we should scale the inputs down and then scale the prediction up when we're done
size_scaled = (size - data2.mean()[0])/data2.std()[0]
beds_scaled = (num_bedrooms - data2.mean()[1])/data2.std()[1]

# This is our model -- we're just using it here to make a calculation
pred_price = (W_opt[0] * 1) + (W_opt[1] * size_scaled) + (W_opt[2] * beds_scaled)

# Format and print the result
print("Predicted Price: ", '${:20,.0f}'.format(math.ceil(pred_price)))
print("Optimal Parameter Values: {}".format(W_opt))


I am working on the file "Notebooks/Regression-With-Multiple-Features.ipynb".
The first task is:
The Business Problem: Predicting Housing Prices
What's the market value of a house? One way to determine the market value is to collect up the prices of houses based on a few characteristics such as size in square feet and number of bedrooms. Then apply machine learning to "learn" what the price should be based on this data.
The data we have is a well know machine learning dataset of housing prices in Portland, Oregon. Let's refresh our memories of what this dataset looks like.
Load the Data
Can you write Python code for it?

import os
# OS-independent way to navigate the file system
# Data directory is one directory up in relation to directory of this notebook
data_dir_root = os.path.normpath(os.getcwd() + os.sep + os.pardir)
# Where the file is
file_url = data_dir_root + os.sep + "Data" + os.sep + "portland-house-prices.txt"
# Load the data into a dataframe
data2 = pd.read_csv(file_url, header=None, names=['Size', 'Bedrooms', 'Price'])

# The number of (rows, columns) in the dataset
data2.shape


I am working on the file "Notebooks/Regression-with-a-Single-Feature.ipynb".
The first task is:
ACKNOWLEDGEMENT
The dataset used in this notebook is from Andrew Ng's course on Machine Learning on Coursera.
Linear regression has been in use for hundreds of years. What place does it have in the shiny (relatively) new field of machine learning? It's the same end result you've learned in statsistics classes, but in a way much simpler. The method for getting there is via the steps of machine learning outlined in the Nuts and Bolts notebook. We'll go through these steps again in this notebook.
The Business Problem: Predicting Restaurant Profits
You're the CEO of a restaurant franchise. Your restaurants operate in a number of small towns. You're thinking of how to grow the business. Where's the best place to put the next restaurant?
For each restaurant your company owns and operates, you have the population (in 10,000s) of the town the restaurant is located in and the most recent year's profit (in \$10,000s) generated by the restaurant. You'd like to use this data to make some profit predictions and use these to prioritize locations for new restaurants.
Let's have a look at this data.
Load the Data
Can you write Python code for it?

# Load up the packages to investigate the data
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.cm as cm
%matplotlib inline
import seaborn as sns

import os
# OS-independent way to navigate the file system
# Data directory is one directory up in relation to directory of this notebook
data_dir_root = os.path.normpath(os.getcwd() + os.sep + os.pardir)
# Where the file is
file_url = data_dir_root + os.sep + "Data" + os.sep + "food-truck-profits.txt"
# Load the data into a dataframe
data = pd.read_csv(file_url, header=None, names=['Population', 'Profit'])

# Quick check on what we have
data.shape


I am working on the file "Notebooks/Regression-with-a-Single-Feature.ipynb".
The first task is:
Exercise 3
Does the penalty we've chosen make sense? Convince yourself of this and write a paragraph explaining why it makes sense.
Can you write Python code for it?

penalty(X,y,[-10, 1], VPenalty)

penalty(X,y,[-10, 1], invertedVPenalty)


I am working on the file "Notebooks/Regression-with-a-Single-Feature.ipynb".
The first task is:
Exercise 4
Experiment with different values of alpha, W, and iters. Write down your observations.
Step 6: Use the Model and Optimal Parameter Values to Make Predictions
Let's see how our optimal parameter values can be used to make predictions.
Can you write Python code for it?

W_opt[0,0], W_opt[1,0]

# Create 100 equally spaced values going from the minimum value of population
# to the maximum value of the population in the dataset.
x = np.linspace(data.Population.min(), data.Population.max(), 100)
f = (W_opt[0, 0] * 1) + (W_opt[1, 0] * x)

fig, ax = plt.subplots(figsize=(8,5))
ax.plot(x, f, 'g', label='Prediction')
ax.scatter(data.Population, data.Profit, label='Training Data')
ax.legend(loc='upper left')
ax.set_xlabel('Population')
ax.set_ylabel('Profit')
ax.set_title('Predicted Profit vs. Population Size');

# First 5 population values in the dataset
X[0:5].values.flatten()

# Prediction of profit for the first 5 populations in the dataset
#populations = [5, 6, 12, 14, 15]
populations = X[0:5].values.flatten()
profits = [W_opt[0, 0] + (W_opt[1, 0] * pop * 10000) for pop in populations]
#print(profits)
print(['${:5,.0f}'.format(profit) for profit in profits])


I am working on the file "Notebooks/Regression-with-a-Single-Feature.ipynb".
The first task is:
Exercise 5
In the plot above what do you observe about the way in which the learning rate and the number of interactions determine the "prediction line" that is learned?
Exercise 6
Now you can make predictions of profit based on your data. What are the predicted profits for populations of 50,000, 100,000, 160,000, and 180,000? Are the predictions reasonable? Explain why or why not?
Can you write Python code for it?

# We're using the optimal W values obtained when the learning rate = 0.001
# and the number of iterations = 500
predictions = [(W_values[3][0] * 1) + (W_values[3][1] * pop) for pop in [50000, 100000, 160000, 180000]]

# Get into the right form form printing
preds = np.array(predictions).squeeze()
print(['${:5,.0f}'.format(pred) for pred in preds])


I am working on the file "Notebooks/Regression-with-a-Single-Feature.ipynb".
The first task is:
Learning from Experience
Exercise 7
What happens to the optimal values of W if we use just a quarter of the dataset? What happens if we now use half of the training set? How does this relate to Tom Mitchell's definition of machine learning?
Can you write Python code for it?

# We'll use the values of num_iters and learning_rate defined above
print("num_iters: {}".format(num_iters))
print("learning rate: {}".format(learning_rate))

# Vary the size of the dataset
dataset_sizes = [2, 5, 10, 25, 50, len(X)]
gdResults = [gradientDescent(X[0:dataset_sizes[i]], y[0:dataset_sizes[i]], \
                             W_init, num_iters, learning_rate) for i in range(len(dataset_sizes))]

# This allows us to color our plot lines differently without explicitly specifying colors
cmap2 = plt.get_cmap('jet')
plot_colors2 = cmap2(np.linspace(0, 1, len(dataset_sizes)))

W_values = [gdResults[i][0] for i in range(len(gdResults))]
W_values[0]

# Create 100 equally spaced values going from the minimum value of population
# to the maximum value of the population in the dataset.
x = np.linspace(data.Population.min(), data.Population.max(), 100)
f_list = [(W_values[i][0] * 1) + (W_values[i][1] * x).T for i in range(len(W_values))]
fig, ax = plt.subplots(figsize=(12,8))
#[ax.plot(x, f_list[i], 'y', label=dataset_sizes[i]) for i in range(len(f_list))]
for i, (dataset, color) in enumerate(zip(dataset_sizes, plot_colors2), 1):
    ax.plot(x, f_list[i-1], label=dataset, c=color)
ax.scatter(data.Population, data.Profit, label='Dataset')
ax.legend(loc='upper left')
ax.set_xlabel('Population')
ax.set_ylabel('Profit')
ax.set_title('Predicted Profit as Dataset Size Changes')


I am working on the file "Notebooks/Regression-with-a-Single-Feature.ipynb".
The first task is:
Once we've identified the inputs and the output, the task is easy to define: given the inputs, predict the output. So in this case, given a town's population, predict the profit a restarurant would generate.
Step 3: Define the Model
As we saw in the Nuts and Bolts session, a model is a way of transforming the inputs into the output.
For every row of the dataset, we use the same weights $w_{0}$ and $w_{1}$ to multiply each corresponding feature value in that row and summing like so:
$$(w_{0} * x_{0}) + (w_{1} * x_{1})$$
Depending on the values of the $w$ values and the features, this gives us a value for the output -- $\hat{y}$ -- in that row of the dataset.
Step 3a: Define the Features
In this case the single feature is the population size (also our only input). Here the features and the inputs are one and the same. We'll find later that this doesn't always have to be so -- in some cases there can be fewer or greater features than the number of inputs.
Step 3b: Transform the Features Into An Output
Once again, as we saw in the ML-Nuts-and-Bolts notebook, a model is a way of transforming the inputs into the output.
For every row of the dataset, we use the same weights $w_{0}$ and $w_{1}$ to multiply each corresponding feature value in that row and summing like so:
$$(w_{0} * x_{0}) + (w_{1} * x_{1})$$
Remember from the Nuts and Bolts notebook that $w_{0}$ is called the intercept and the value of $x_{0}$ is 1 for all rows.
Depending on the values of the $w$s and the features, this gives us a value for the output -- $\hat{y}$ -- in that row of the dataset.
SIDE BAR - Matrix Notation
Because our datasets typically have lots of rows, applying a model to a dataset means hundreds, if not millions of equations like the one above -- in fact, one for each row of the dataset. The beautiful thing is matrix notation can express these millions of equations in a compact way using just a single line.
To see how matrix notation does this neat trick, first separate the dataset into the input features matrix $X$, the output...
Can you write Python code for it?

# A Handful of Penalty Functions
# Generate the error range
x = np.linspace(-10,10,100)
[penaltyPlot(x, pen) for pen in penaltyFunctions.keys()];


I am working on the file "Notebooks/Regression-with-a-Single-Feature.ipynb".
The first task is:
Regression With a Single Feature
Using a single feature to make a numerical prediction
TO DO
- nothing for the moment
Can you write Python code for it?

# Share functions used in multiple notebooks
%run Shared-Functions.ipynb


I am working on the file "Notebooks/Regression-with-a-Single-Feature.ipynb".
The first task is:
Run the iterative gradient descent method to determine the optimal parameter values.
Can you write Python code for it?

# Set hyper-parameters
num_iters = 50 # number of iterations
learning_rate = 0.0005 # the learning rate

# Run gradient descent and capture the progression
#  of cost values and the ultimate optimal W values
%time W_opt, final_penalty, running_w, running_penalty = gradientDescent(X, y, W_init, num_iters, learning_rate)
# Get the optimal W values and the last few cost values
W_opt, final_penalty, running_w[-5:], running_penalty[-5:]


I am working on the file "Notebooks/Regression-with-a-Single-Feature.ipynb".
The first task is:
SIDEBAR -  How the Penalty is Usually Written
The cost of getting it wrong is defined as a function $J(W)$:
$$J(W) = \frac{1}{2m} \sum_{i=1}^{m} (h_{W}x^{(i)}) - y^{(i)})^2$$
What we're saying here: For each input, transform it using $w_{0}$ and $w_{1}$. This will give us a number. Subtract from this number the actual value of the output for that input. This gives us another number. Take this number and square it. This gives us our final result for that particular input. Add each of these final results -- one for each input we have in our dataset -- and divide it by $2m$ -- that is, twice the number of data points in our data set. This last division step is to make the cost of getting it wrong relative to the size of the dataset -- think of it simply as a mathematical convenience.
This way of writing things expresses exactly the same thing as writing out a series of equations would. That's why matrix notation is so powerful and widely used -- the matrix notation helps keep things short and sweet.
How the Penalty Varies as the $w_{0}$ and $w_{1}$ Values Change
Can you write Python code for it?

# Visualize what np.meshgrid does when used with plot
w0 = np.linspace(1,5,5)
w1 = np.linspace(1,5,5)
W0, W1 = np.meshgrid(w0,w1)
plt.plot(W0,W1, marker='*', color='g', linestyle='none');

# Plot the cost surface
# From https://stackoverflow.com/questions/9170838
# See Also: Helpful matplotlib tutorial at
#   http://jeffskinnerbox.me/notebooks/matplotlib-2d-and-3d-plotting-in-ipython.html

# Set up a grid over w0,w1 values
w0 = np.linspace(-10,10,50)
w1 = np.linspace(-10,10,50)
W0, W1 = np.meshgrid(w0,w1)

# Get the penalty value for each point on the grid
# See the Shared-Functions.ipynb notebook for the list of defined penalty functions
# List of penalty functions in dict penaltyFunctions
penalties = np.array([penalty(X,y,[w_0,w_1], squaredPenalty) for w_0,w_1 in zip(np.ravel(W0), np.ravel(W1))])
Z = penalties.reshape(W0.shape)

# Create the plot
from mpl_toolkits.mplot3d import Axes3D
fig, ax = plt.subplots(figsize=(12,8))
ax = fig.add_subplot(1,1,1, projection='3d')
ax.set_title("Cost Surface for a " + penaltyFunctions[squaredPenalty] + " Function")
ax.set_xlabel('w0')
ax.set_ylabel('w1')
ax.set_zlabel('Cost')
p = ax.plot_surface(W0, W1, Z, rstride=4, cstride=4)

# Contour Lines
fig, ax = plt.subplots(figsize=(12,8))
plt.contour(Z, cmap=cm.RdBu,vmin=abs(Z).min(), vmax=abs(Z).max(), extent=[-10,10,-10,10])

# Heatmap or Colormap
fig, ax = plt.subplots(figsize=(12,8))
p = ax.pcolor(W0, W1, Z, cmap=cm.RdBu, vmin=abs(Z).min(), vmax=abs(Z).max())
cb = fig.colorbar(p)


I am working on the file "Notebooks/Regression-with-a-Single-Feature.ipynb".
The first task is:
Step 1: Visualize the Data
Can you write Python code for it?

# Visualize the data
data.plot.scatter(x='Population', y='Profit', figsize=(8,6));


I am working on the file "Notebooks/Regression-with-a-Single-Feature.ipynb".
The first task is:
Step 5: Find the Parameter Values that Minimize the Cost
The cost function might have a minimum but how can we possibly find it? We can't use the brute force method of choosing every possible combination of values for $w_{0}$ and $w_{1}$ -- there are an infinite number of combinations and we'll never finish our task.
This is where the concept of gradient descent comes in. Imagine starting anywhere on the surface, say at $w_{0}$ at -10 and $w_{1}$ at -10. That's at the left front edge of the surface plot you've just seen. If we took a step in the direction where the slope under our feet is steepest, then we would be one step closer to the "bottom" of the surface. So let's take that step and then take the next step in the direction where the slope under our feet is steepest. That gets us even lower and in the right direction to the bottom. Eventually, after a number of these steps, you'll get to the bottom.
Let's look at this a bit more systematically.
That's the idea. To make it work, we have to write out an expression for the next set of parameter values to try. And it turns out that for the cost function $J(W)$, there is a well-worked out way to write these values for $w_{0}$ and $w_{1}$ based on the direction of the steepest slope.
How To Choose the Next Set of Values for $W$
$$w_{j} := w_{j} - \frac{\alpha}{m} \ [\sum_{i=1}^{m} (h_{W}(x^{(i)}) - y^{(i)}) x_{j}^{(i)}]$$
Implement the Iterative Method of Gradient Descent
Here's what we'll do.
- First pick initial values for $w_0$ and $w_1$. Pick anything within reason -- let's say 1 and -1 respectively.
- Choose a penalty function. We'll choose the squaredPenalty.
- For each row of the dataset find $\hat{y}$ for the values of $w_0$ and $w_1$. Find the value of the penalty for $y - \hat{y}$ using the penalty function.
- Add up the penalty values for each row. This total is the cost.
- Use gradient descent to find the next set of $w_0$ and $w_1$ values that lower the cost. Repeat until the number of iterations specified are complete.
- The final values of...
Can you write Python code for it?

# Initialize the parameter values W and pick the penalty function
W_init = [1,-1.0]
penalty_function = squaredPenalty

# Test out the penalty function in the Shared-Functions notebook
penalty(X, y, W_init, penalty_function)

# Test out the gradientDescent function in the Shared-Functions notebook
gradientDescent(X, y, W_init, num_iterations=5)


I am working on the file "Notebooks/Regression-with-a-Single-Feature.ipynb".
The first task is:
The complete data set can be described using the traditional statistical descriptors:
Can you write Python code for it?

# Calculate some useful statistics showing how the data is distributed
data.describe()


I am working on the file "Notebooks/Regression-with-a-Single-Feature.ipynb".
The first task is:
This means that the dataset has 97 rows and 2 columns. Let's see what the data looks like.
The first few rows of our data look like this:
Can you write Python code for it?


I am working on the file "Notebooks/Regression-with-a-Single-Feature.ipynb".
The first task is:
We can see that the Ws are changing even after 5000 interations...but at the 4th decimal place. Similarly, the penalty is changing (decreasing) in the 100s place.
How cost changes as the number of iterations increase
Can you write Python code for it?

# How the penalty changes as the number of iterations increase
fig, ax = plt.subplots(figsize=(8,5))
ax.plot(np.arange(num_iters), running_penalty, 'g')
ax.set_xlabel('Number of Iterations')
ax.set_ylabel('Cost')
ax.set_title('Cost vs. Iterations Over the Dataset for a Specific Learning Rate');

np.array(running_w).flatten()

w0 = np.array([param[0].flatten() for param in running_w][1:]).flatten()
w1 = np.array([param[1] for param in running_w][1:]).flatten()
len(w0), len(w1), len(np.arange(num_iters))

# How the Ws change as the number of iterations increase
fig, (ax1,ax2) = plt.subplots(figsize=(14,6), nrows=1, ncols=2, sharey=False)
ax1.plot(np.arange(num_iters), w0, 'g')
ax1.set_xlabel('Number of Iterations')
ax1.set_ylabel(r'$w_{0}$')
ax1.set_title(r'$w_{0}$ vs. Iterations Over the Dataset')
ax2.plot(np.arange(num_iters), w1, 'y')
ax2.set_xlabel('Number of Iterations')
ax2.set_ylabel(r'$w_{1}$')
ax2.set_title(r'$w_{1}$ vs. Iterations Over the Dataset')

fig, ax = plt.subplots(figsize=(14,6))
ax.plot(np.arange(num_iters), w0, 'g', label="w0")
ax.plot(np.arange(num_iters), w1, 'y', label="w1")
plt.legend()


I am working on the file "Notebooks/Reweighting a fastdate chronogram distribution that was generated with high variance.ipynb".
The first task is:
Credibility intervals without and with reweighting. First, the first scores:
Can you write Python code for it?

%matplotlib inline

def plotSingleCorrelation(unweightedIntervals,weightedIntervals,xlabel,ylabel):
    fit = np.polyfit(unweightedIntervals,weightedIntervals,1)
    fit_fn = np.poly1d(fit)

    #Plotting the first score-reweighted trees:
    plt.plot(unweightedIntervals,weightedIntervals, 'bo')
    plt.plot(unweightedIntervals, fit_fn(unweightedIntervals), 'b-')
    plt.plot( unweightedIntervals, unweightedIntervals, color="black")
    plt.xlabel(xlabel, fontsize=15)
    plt.ylabel(ylabel, fontsize=15)
    #plt.axis([-0.01, 0.08, -0.01, 0.06])
    regString = "y="+"{:10.2f}".format(fit[0])+"*x"+"{:10.2f}".format(fit[1])
    plt.legend(['data', regString, 'y = x'], loc='upper left')

plotSingleCorrelation(unweightedIntervals, weightedIntervals,"Unweighted intervals","Weighted intervals")


I am working on the file "Notebooks/Reweighting a fastdate chronogram distribution that was generated with high variance.ipynb".
The first task is:
Edition following a commentary by Gergely: I used to normalize by the sum of all scores. Here I will try normalizing by the maximum score.
Can you write Python code for it?

print(np.max(scores))
for i in range(len(scores)):
    normalizedScores.append( float(scores[i])/np.max(scores) )
    normalizedScores2.append( float(scores2[i])/np.max(scores2) )
    normalizedScores3.append( float(scores3[i])/np.max(scores3) )

scoresDF = pd.DataFrame({'kItem': scores,'scores':normalizedScores})
scores2DF = pd.DataFrame({'kItem': scores,'scores':normalizedScores2})
scores3DF = pd.DataFrame({'kItem': scores,'scores':normalizedScores3})

mea, med = computeWeightedMeanAndMedian (scoresDF)
mea2, med2 = computeWeightedMeanAndMedian (scores2DF)
mea3, med3 = computeWeightedMeanAndMedian (scores3DF)

print ("Mean and median for score 1: "+str(mea)+" ; "+str(med))
print ("Mean and median for score 2: "+str(mea2)+" ; "+str(med2))
print ("Mean and median for score 3: "+str(mea3)+" ; "+str(med3))

print("Scores 1:")
print(scoresDF.describe())
print("Scores 2:")
print(scores2DF.describe())
print("Scores 3:")
print(scores3DF.describe())


I am working on the file "Notebooks/Reweighting a fastdate chronogram distribution that was generated with high variance.ipynb".
The first task is:
Function to count the number of constraints met by a chronogram:
Can you write Python code for it?

def checkConstraints(id2Height, constraints):
    numberOfMetConstraints = 0
    for c in constraints:
        try:
            if (id2Height[str(float(c[0]))] > id2Height[str(float(c[1]))]):
                numberOfMetConstraints = numberOfMetConstraints + 1
        except KeyError:
            print ("Constraint involving nodes " + str(float(c[0])) + " and " + str(float(c[1])) + " cannot be checked.")
    return numberOfMetConstraints


I am working on the file "Notebooks/Reweighting a fastdate chronogram distribution that was generated with high variance.ipynb".
The first task is:
Function to renumber nodes as in a reference tree:
Can you write Python code for it?

def renumberNodes( treeToAnnotate, leafList2NodeId ):
    #print treeToAnnotate.get_ascii(attributes=[ "name"], show_internal=False)
    node2leaves = treeToAnnotate.get_cached_content()
    for k in node2leaves.keys():
        if len(node2leaves[k]) == 1: #leaf node
            pass
        else:
            nodelist = list()
            for n in node2leaves[k]:
                nodelist.append( n.name )
            nodelist.sort()
            k.support = leafList2NodeId[tuple(nodelist)]


I am working on the file "Notebooks/Reweighting a fastdate chronogram distribution that was generated with high variance.ipynb".
The first task is:
Getting node ages in a chronogram:
Can you write Python code for it?

def getNodeHeights( t ):
    node2Height = dict()
    id2Height = dict()
    for node in t.traverse("postorder"):
        if node not in node2Height:
            node2Height[node] = 0.0
            id2Height[node.name] = 0.0
        if node.up:
            node2Height[node.up] = node2Height[node] + node.dist
            id2Height[str(node.up.support)] = node2Height[node] + node.dist
      # print node.name + " : " + str(node2Height[node])
    return node2Height,id2Height


I am working on the file "Notebooks/Reweighting a fastdate chronogram distribution that was generated with high variance.ipynb".
The first task is:
Let's compare the node ages between these consensus chronograms:
Can you write Python code for it?

%matplotlib inline

#Plotting the score-reweighted node ages vs the unweighted node ages:
plt.plot(unweightedAges,weightedAges, 'ro', markerfacecolor='none', markeredgecolor="red")
plt.plot(unweightedAges,weightedAges2, 'bs', markerfacecolor='none', markeredgecolor="blue")
plt.plot(unweightedAges,weightedAges3, 'gd', markerfacecolor='none', markeredgecolor="green")

plt.plot( unweightedAges, unweightedAges, color="black") # add x=y
plt.xlabel('Unweighted Ages', fontsize=15)
plt.ylabel('Weighted Ages', fontsize=15)
plt.legend(['Score 1', 'Score 2', 'Score 3', 'y = x'], loc='upper left')