I am working on the file "2015-10_Lecture/Lecture2/code/3_Intro_Lasagne_Solution.ipynb".
The first task is:
Create the Train Function
After loading the data and defining the MLP, we can now create the train function.
Can you write Python code for it?

# Parameters
n_in = 28*28
n_hidden = 50
n_out = 10

# Create the network
x = T.dmatrix('x')  # the data, one image per row
y = T.lvector('y')  # the labels are presented as 1D vector of [int] labels

network = build_mlp(n_in, n_hidden, n_out, x)

# Create a loss expression for training, i.e., a scalar objective we want
# to minimize (for our multi-class problem, it is the cross-entropy loss):
prediction = lasagne.layers.get_output(network)
loss = lasagne.objectives.categorical_crossentropy(prediction, y)
loss = loss.mean()

# Create update expressions for training, i.e., how to modify the
# parameters at each training step. Here, we'll use Stochastic Gradient
# Descent (SGD) with Nesterov momentum, but Lasagne offers plenty more.
params = lasagne.layers.get_all_params(network, trainable=True)
updates = lasagne.updates.nesterov_momentum(loss, params, learning_rate=0.01, momentum=0.9)

# Predict the labels
network_predict_label = T.argmax(lasagne.layers.get_output(network, deterministic=True), axis=1)

# Compile a function performing a training step on a mini-batch (by giving
# the updates dictionary) and returning the corresponding training loss:
train_fn = theano.function(inputs=[x, y], outputs=loss, updates=updates)

# Create the predict_labels function
predict_labels = theano.function(inputs=[x], outputs=network_predict_label)


I am working on the file "2015-10_Lecture/Lecture4/code/20Newsgroups/Autoencoder.ipynb".
The first task is:
Baseline
We use Scikit-learn to derive some baselines for the above setting
Can you write Python code for it?

#Naive Bayes
from sklearn.naive_bayes import BernoulliNB
clf = BernoulliNB(alpha=.01)
clf.fit(train_subset_x, train_subset_y)
pred = clf.predict(test_x)
acc = metrics.accuracy_score(test_y, pred)
print "Naive Bayes: %f%%" % (acc*100)

#Gaussian Naive Bayes
from sklearn.naive_bayes import GaussianNB
clf = GaussianNB()
clf.fit(train_subset_x, train_subset_y)
pred = clf.predict(test_x)
acc = metrics.accuracy_score(test_y, pred)
print "Gaussian Naive Bayes: %f%%" % (acc*100)

#MultinomialNB
from sklearn.naive_bayes import MultinomialNB
clf = MultinomialNB()
clf.fit(train_subset_x, train_subset_y)
pred = clf.predict(test_x)
acc = metrics.accuracy_score(test_y, pred)
print "Multinomial Naive Bayes: %f%%" % (acc*100)

#LinearSVM
from sklearn import svm
clf = svm.LinearSVC()
clf.fit(train_subset_x, train_subset_y)
pred = clf.predict(test_x)
acc = metrics.accuracy_score(test_y, pred)
print "LinearSVM: %f%%" % (acc*100)


I am working on the file "2015-10_Lecture/Lecture4/code/20Newsgroups/Autoencoder.ipynb".
The first task is:
Deep Feed Forward Network
Can you write Python code for it?

from keras.layers import containers
import keras
from keras.models import Sequential
from keras.layers.core import Dense, Flatten, AutoEncoder, Dropout
from keras.optimizers import SGD
from keras.utils import np_utils
from keras.callbacks import EarlyStopping

random.seed(2)
np.random.seed(2)

nb_epoch = 30
batch_size = 200

model = Sequential()
model.add(Dense(500, input_dim=max_words, activation='relu'))
model.add(Dropout(0.5))
#model.add(Dense(500, activation='relu'))
#model.add(Dropout(0.5))
model.add(Dense(nb_labels, activation='softmax'))

train_subset_y_cat = np_utils.to_categorical(train_subset_y, nb_labels)
test_y_cat = np_utils.to_categorical(test_y, nb_labels)

model.compile(loss='categorical_crossentropy', optimizer='adam')

print('Start training')
model.fit(train_subset_x, train_subset_y_cat, batch_size=batch_size, nb_epoch=nb_epoch,
          show_accuracy=True, verbose=True, validation_data=(test_x, test_y_cat))

score = model.evaluate(test_x, test_y_cat, show_accuracy=True, verbose=False)
print('Test accuracy:', score[1])


I am working on the file "2015-10_Lecture/Lecture4/code/BrownCorpus/GenreClassification.ipynb".
The first task is:
Neural Network
Given the Training and Test sets, we now define a feed forward network. We use a 500 dimensional hidden layer with dropout of 0.5.
Feel free to try different hidden layer sizes and number of hidden layers.
Can you write Python code for it?

from keras.layers import containers
import keras
from keras.models import Sequential
from keras.layers.core import Dense, Flatten, AutoEncoder, Dropout
from keras.optimizers import SGD
from keras.utils import np_utils

batch_size = 30
nb_epoch = 50
nb_classes = len(category2Idx)

model = Sequential()
model.add(Dense(500, input_dim=num_max_words, activation='tanh'))
model.add(Dropout(0.5))
model.add(Dense(nb_classes, activation='softmax'))

train_y_cat = np_utils.to_categorical(train_y, nb_classes)
test_y_cat = np_utils.to_categorical(test_y, nb_classes)

model.compile(loss='categorical_crossentropy', optimizer='Adam')
score = model.evaluate(test_x, test_y_cat, show_accuracy=True, verbose=0)
print('Test score before fine turning:', score[0])
print('Test accuracy before fine turning:', score[1])
model.fit(train_x, train_y_cat, batch_size=batch_size, nb_epoch=nb_epoch,
          show_accuracy=True, validation_data=(test_x, test_y_cat))
score = model.evaluate(test_x, test_y_cat, show_accuracy=True, verbose=0)
print('Test score after fine turning:', score[0])
print('Test accuracy after fine turning:', score[1])


I am working on the file "2015-10_Lecture/Lecture4/code/MNIST/Autoencoder.ipynb".
The first task is:
PCA
In comparison we are going to plot also an PCA image.
Can you write Python code for it?

from sklearn.decomposition import PCA
pca = PCA(n_components=2)
pca.fit(train_x)

pca_test = pca.transform(test_x)

colors = {0: 'b', 1: 'g', 2: 'r', 3:'c', 4:'m',
          5:'y', 6:'k', 7:'orange', 8:'darkgreen', 9:'maroon'}

markers = {0: 'o', 1: '+', 2: 'v', 3:'<', 4:'>',
          5:'^', 6:'s', 7:'p', 8:'*', 9:'x'}

plt.figure(figsize=(10, 10))
patches = []
for idx in xrange(0,300):
    point = pca_test[idx]
    label = test_y[idx]
    if label in [2,5,8,9]:
        continue
    color = colors[label]
    marker = markers[label]
    line = plt.plot(point[0], point[1], color=color, marker=marker, markersize=8)

#plt.axis([-1.1, 1.1, -1.1, +1.1])
plt.show()


I am working on the file "2015-10_Lecture/Lecture4/code/MNIST/Autoencoder.ipynb".
The first task is:
Plot Autoencoder
Here we are going to plot the output of the autoencoder (dimension of the last hidden layer should be 2).
Can you write Python code for it?

############
# Plot it
############
%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches

model = Sequential()
for encoder in encoders:
    model.add(encoder)
model.compile(loss='categorical_crossentropy', optimizer='Adam')

ae_test = model.predict(test_x)

colors = {0: 'b', 1: 'g', 2: 'r', 3:'c', 4:'m',
          5:'y', 6:'k', 7:'orange', 8:'darkgreen', 9:'maroon'}

markers = {0: 'o', 1: '+', 2: 'v', 3:'<', 4:'>',
          5:'^', 6:'s', 7:'p', 8:'*', 9:'x'}

plt.figure(figsize=(10, 10))
patches = []
for idx in xrange(0,300):
    point = ae_test[idx]
    label = test_y[idx]

    if label in [2,5,8,9]: #We skip these labels to make the plot clearer
        continue

    color = colors[label]
    marker = markers[label]
    line = plt.plot(point[0], point[1], color=color, marker=marker, markersize=8)

#plt.axis([-1.1, 1.1, -1.1, +1.1])


I am working on the file "2015-10_Lecture/Lecture4/code/MNIST/Autoencoder.ipynb".
The first task is:
k-Means clustering
We run a k-means clustering on the AutoEncoder representations and the PCA representations and then do a majority voting to get the label per cluster. We then compute the accurarcy of the clustering. This gives us some impression how good the 2-dim representations are. This is not perfect, as AutoEncoder and PCA might create non-linear cluster boundaries.
Can you write Python code for it?

from sklearn.cluster import KMeans
import operator

def clusterAccurarcy(predictions, n_clusters=10):
    km = KMeans(n_clusters=n_clusters)

    clusters = km.fit_predict(predictions)

    #Count labels per cluster
    labelCount = {}

    for idx in xrange(len(test_y)):
        cluster = clusters[idx]
        label = test_y[idx]

        if cluster not in labelCount:
            labelCount[cluster] = {}

        if label not in labelCount[cluster]:
            labelCount[cluster][label] = 0

        labelCount[cluster][label] += 1

    #Majority Voting
    clusterLabels = {}
    for num in xrange(n_clusters):
        maxLabel = max(labelCount[num].iteritems(), key=operator.itemgetter(1))[0]
        clusterLabels[num] = maxLabel
    #print clusterLabels
    #Number of errors
    errCount = 0
    for idx in xrange(len(test_y)):
        cluster = clusters[idx]
        clusterLabel = clusterLabels[cluster]
        label = test_y[idx]

        if label != clusterLabel:
            errCount += 1

    return errCount/float(len(test_y))

print "PCA Accurarcy: %f%%" % (clusterAccurarcy(pca_test)*100)
print "AE Accurarcy: %f%%" % (clusterAccurarcy(ae_test)*100)


I am working on the file "2015-11-18 - TD12 - Introduction aux tableaux.ipynb".
The first task is:
Exercice 1 :  Recherchez si un élément est présent dans un tableau.
Can you write Python code for it?

def cherche(tab, elt):
    i = 0
    while i < len(tab):
        if tab[i] == elt:
            print("J'ai trouvé !")
        i = i+1

tableau =  ["pommes", "tomates", "fromage", "lait", "sucre"]
cherche(tableau, "tomates")


I am working on the file "2015-11-18 - TD12 - Introduction aux tableaux.ipynb".
The first task is:
Nous pouvons parcourir les éléments d'un tableau :
Can you write Python code for it?

tab = ["pommes", "tomates", "fromage", "lait", "sucre"]
i = 0
while i < len(tab):
    print(tab[i])
    i = i+1

# Attention, notez la différence avec :
j = 0
while j < len(tab):
    print(j)
    j = j+1


I am working on the file "2015-11-26 - TD14 - Rappels de cours sur les entrées, les sorties, et les entrées-sorties.ipynb".
The first task is:
Notons cependant que les entrées-sorties d'une fonction ne peuvent pas avoir pour mode de transmission "AFFECTÉE précédemment" ou "AFFECTÉE pour la suite", puisque les variables d'une fonction n'existent qu'à l'intérieur de cette fonction.
Fonction et procédure
On a appelé "fonction" tout algorithme défini par le mot-clé def. C'est effectivement le terme général employé dans la plupart des langages de programmation.
Cela dit, il est parfois nécessaire de distinguer plusieurs types de fonction (au sens large).

Dans son sens strict, le terme fonction désigne un algorithme dont toutes les entrées sont PASSÉES en paramètre, et toutes les sorties sont RETOURNÉES (par exemple, la fonction fact ci-dessus).

On appelle procédure un algorithme dont aucune sortie n'est RETOURNÉE (par exemple, la fonction print de Python).

La dénomination des algorithmes ne rentrant dans aucune de ces deux catégorie varie d'une personne à l'autre ; par exemple, la fonction input de Python sera appelée "fonction" par certains (parce qu'elle retourne une valeur), et "procédure" par d'autres (parce que ce n'est pas une fonction au sens strict, son entrée n'est pas passée en paramètre).
 Exercice 1  Écrire un algorithme qui recherche si un entier est contenu dans un tableau. Vous écrirez également le contrat et ferez bien attention à expliquer si vous manipulez des entrées, des sorties ou les deux.
Can you write Python code for it?

def rechercheTab(tab, nb):
    '''
    :entree tab: un tableau d'entiers
    :entree nb: le nombre recherché
    :sortie present: un booleen qui indique si le nombre est présent dans le tableau ou non
    :pré-condition: le tableau peut être vide
    '''
    present = False
    i = 0
    while (i < len(tab)) and (present == False):
        if tab[i] == nb:
            present = True
        i = i + 1
    return present

print(rechercheTab([3, 6, 4, 5, 9], 6))
print(rechercheTab([3, 6, 4, 5, 9], 1))
print(rechercheTab([], 2))

# Même solution (et même contrat), mais avec un for à la place du while
# Attention, on va parcourir tout le tableau même si on n'en a pas besoin
def rechercheTabBis(tab, nb):
    present = False
    for i in range(len(tab)):
        if tab[i] == nb:
            present = True

    return present

print(rechercheTabBis([3, 6, 4, 5, 9], 6))
print(rechercheTabBis([3, 6, 4, 5, 9], 1))
print(rechercheTabBis([], 2))


I am working on the file "2015-11-26 - TD14 - Rappels de cours sur les entrées, les sorties, et les entrées-sorties.ipynb".
The first task is:
On admettra que, dans une fonction, lorsque qu'on ne spécifie pas le mode de transmission des entrées-sorties, il est forcément "PASSÉE en paramètre" pour les entrées, et "RETOURÉE" pour les sorties.
Mais certaines fonctions utilisent d'autres modes de transmission, par exemple :
Can you write Python code for it?

# la fonction input fournie par Python (version sans paramètre)
def input():
    """
    :entrée txt1: str, SAISIE au clavier
    :sortie txt2: str
    :post-cond: txt2 = txt1
    """

# la fonction print fournie par Python
def print(val1):
    """
    :entrée val1: une valeur quelconque
    :sortie val2: une valeur quelconque, AFFICHÉE à l'écran
    :post-cond: val2 = val1
    """


I am working on the file "2015-12-02 - TD15 - Révisions sur la récursivité.ipynb".
The first task is:
Exercice Robozzle : nous avons ensuite résolu l'exercice Robozzle n°656, l'objectif était de vous faire comprendre le fonctionnement de la pile d'appels récursifs. http://robozzle.com/js/play.aspx?puzzle=656
Can you write Python code for it?

def fact(n):
    """
    :entrée n: int
    :pré-cond: n > 0
    :sortie f: int
    :post-cond: f = n * (n-1) * ... * 1
    """
    if n == 1:
        f = 1
    else:
        f = fact(n-1)*n
    print("--- fact({}) = {}".format(n,f))
    return f

print(fact(6))


I am working on the file "2015-12-03 - TD16 - Récursivité et tableaux.ipynb".
The first task is:
Encore une fois, la solution ci-dessus est correcte mais elle est loin d'être "simple" et facile à lire pour quelqu'un d'autre que celui ou celle qui a écrit l'algorithme. On va donc (ci-dessous) proposer une ré-écriture plus simple.
Can you write Python code for it?

def sommeRecBis(tab):
    '''
    :entree l: une liste de nombres (entiers ou flottants)
    :sortie somme: la somme des éléments de la liste
    :pre-conditions: la liste peut être vide
    :post-condition: somme contient la somme des éléments de la liste, et est donc du même type que les éléments.
    >>> sommeRecBis([1, 2, 3])
    6
    >>> sommeRecBis([])
    0
    >>> sommeRecBis([6, 42.2, 34])
    82.2
    '''

    if len(tab) == 0:
        somme = 0
    else:
        somme = tab[0]+sommeRecBis(tab[1:])
    return somme

print(sommeRecBis([1, 2, 3]))
print(sommeRecBis([]))
print(sommeRecBis([6, 42.2, 34]))

%doctest sommeRecBis


I am working on the file "2015-12-03 - TD16 - Récursivité et tableaux.ipynb".
The first task is:
Exercice 2. Écrire une méthode récursive pour calculer la somme des éléments d'une liste. Vous écrirez également le contrat.
Can you write Python code for it?

def sommeRec(l):
    '''
    :entree l: une liste de nombres (entiers ou flottants)
    :sortie somme: la somme des éléments de la liste
    :pre-conditions: la liste peut être vide
    :post-condition: somme contient la somme des éléments de la liste, et est donc du même type que les éléments.
    >>> sommeRec([1, 2, 3])
    6
    >>> sommeRec([])
    0
    >>> sommeRec([6, 42.2, 34])
    82.2
    '''
    somme=0
    if len(l)>1:
        somme=l[0]+sommeRec(l[1:])
    elif len(l)==1:
        somme+=l[0]
    return somme
print(sommeRec([1, 2, 3]))
print(sommeRec([]))
print(sommeRec([6, 42.2, 34]))

%doctest sommeRec


I am working on the file "2015-12-03 - TD16 - Récursivité et tableaux.ipynb".
The first task is:
Exerice 3. Écrire un algorithme qui permet de rechercher un nombre dans un tableau trié. Proposez une solution récursive et une solution non récursive.
Can you write Python code for it?

def rechercheTab(tab,a):
    '''
    :entree tab: un tableau de nombres (entiers ou flottants) triés
    :entree a: le nombre recherché
    :sortie i: l'indice de la case du tableau dans laquelle se trouve le nombre.
    :pré-conditions: le tableau est trié par ordre croissant de valeur.
    :post-condition: l'indice de la première occurrence trouvée du nombre est renvoyé.
     Si le nombre n'est pas présent dans le tableau, on retourne -1.
    :Remarque : on appelle ce type de recherche "recherche par dichotomie".
    >>> rechercheTab([0,1,2,3,4],1)
    1
    '''
    i=-1
    b=len(tab)//2
    if tab[b]==a:
        i=b
    elif b == 0:
        i = -1
    elif tab[b]>a: # Si la valeur du milieu du tableau est plus grande que la valeur recherchée, on recherche dans la partie gauche du tableau
        i=rechercheTab(tab[:b],a)
    else: # Sinon, on recherche dans la partie droite et on gère le décalage des indices.
        i=rechercheTab(tab[b:],a)
        if i != -1:
            i = i+b
    return i

print(rechercheTab([0,1,2,3,4],1))
print(rechercheTab([0,1,2,3,4],5))
print(rechercheTab([0,1,2,3,4],4))
print(rechercheTab([0,1.3,2.7,3.4],0))


I am working on the file "2015-12-10 - TD17 - Tableaux et tris, trace et complexité.ipynb".
The first task is:
L'algorithme ci-dessus applique une recherche dichotomique :
on utilise le fait que
* la racine de n est forcément comprise entre 0 et n, et que
* les racines de deux nombres sont dans le même ordre que ces nombres.
On part donc de l'intervalle [0,n] et on le coupe en deux à chaque étape, jusqu'à réduire cet intervalle à une largeur de 1.
Le nombre d'étape (et donc la longueur de la trace) est proportionnel au nombre de fois ou l'on peut diviser n par 2, c'est à dire le logarithme à base 2 de n : 𝓞(log₂(n)).
On peut se convaincre, avec les courbe ci-dessous, que cet algorithme est bien plus efficace que le précédent pour de grandes valeurs de n.
Can you write Python code for it?

%pylab inline
xs = range(1,300)
plot(xs, sqrt(xs), "r-", label=" √n")
plot(xs, log2(xs), "g-", label="log2(n)")
legend(loc="lower right")


I am working on the file "2015-12-10 - TD17 - Tableaux et tris, trace et complexité.ipynb".
The first task is:
Rappels sur la trace d'exécution d'un algorithme et la complexité
(merci à Pierre-Antoine Champin pour cette section)
La trace d'exécution d'un algorithme est constituée
en prenant une "photo" de toutes les variables de cet algorithme
aux instants suivants :

au début
à chaque while
à la fin

La trace est un "compte-rendu" de l'exécution de l'algorithme.
Donnez la trace d'exécution de l'algorithme suivant pour les valeurs suivantes de n : 91, 100, 500.
Can you write Python code for it?

"""
:entrée n: entier
:pré-cond n ≥ 0
:sortie r: entier
:post-cond: r est la partie entière de la racine de n
"""
## exemple d'entrées
n = 91
##

r = 0
while r*r <= n:
    r = r+1
r = r-1

## pour voir la sortie
print(r)
##


I am working on the file "2015-12-23-titanic-initial-exploration.ipynb".
The first task is:
Cabin seems to have a lot of NaN spaces. How much information is there?
Can you write Python code for it?


I am working on the file "2015-12-23-titanic-initial-exploration.ipynb".
The first task is:
IMPORTANT: Age has NaN
Can you write Python code for it?

titanic_data[pd.notnull(titanic_data['Age'])].shape

titanic_data['Age'].describe()

# Plot of non-null age
sns.distplot(titanic_data[pd.notnull(titanic_data['Age'])]['Age'], kde=False)


I am working on the file "2015-12-23-titanic-initial-exploration.ipynb".
The first task is:
Observations

Tickets vary and most likely don't affect survivorship, or at least the ticket #'s don't. There are other variables attached to the ticket (price, location, class) that most likely have a clearer affect on survivorship.
What do the columns mean?
What do the embarked codes mean?

variable source
survival        Survival (0 = No; 1 = Yes)
pclass          Passenger Class (1 = 1st; 2 = 2nd; 3 = 3rd)
name            Name
sex             Sex
age             Age
sibsp           Number of Siblings/Spouses Aboard
parch           Number of Parents/Children Aboard
ticket          Ticket Number
fare            Passenger Fare
cabin           Cabin
embarked        Port of Embarkation (C = Cherbourg; Q = Queenstown; S = Southampton)
Can you write Python code for it?


I am working on the file "2015-12-23-titanic-initial-exploration.ipynb".
The first task is:
Possible Question
How did married couples fair? e.g. number both died, just one?
Can you write Python code for it?

married = titanic_data[((titanic_data['SibSp'] > 0) & (titanic_data['Age'] > 18))]
married.shape

sns.distplot(married[pd.notnull(married['Age'])]['Age'], kde=False, color='r')

not_married = titanic_data[((titanic_data['SibSp'] == 0) & (titanic_data['Age'] > 18))]
sns.distplot(not_married[pd.notnull(not_married['Age'])]['Age'], kde=False)


I am working on the file "2015-12-23-titanic-initial-exploration.ipynb".
The first task is:
What is the distribution of class, port, gender?
Can you write Python code for it?

first = second = third = 0
cher = queens = shamp = some = 0
male = female = 0

for index, row in titanic_data.iterrows():# Determine classes
    if row['Pclass'] == 1:
        first += 1
    elif row['Pclass'] == 2:
        second += 2
    else:
        third += 1

    if row['Embarked'] == 'C':
        cher += 1
    elif row['Embarked'] == 'Q':
        queens += 1
    elif row['Embarked'] == 'S':
        shamp +=1
    else:
        some += 1

    if row['Sex'] == 'male':
        male += 1
    else:
        female += 1

print '''\tFirst: %d
        Second: %d
        Third: %d \n
        Cherbourg: %d
        Queenstown: %d
        Southampton: %d
        nill: %d \n
        Male: %d
        Female: %d''' % (first, second, third, cher, queens, shamp, some, male, female)


I am working on the file "2015-12-24-titanic-gender-grouping.ipynb".
The first task is:
Not really what I was looking for. Was hoping to see survived and died side by side.
Can you write Python code for it?

# Not null ages
female_survived_nn = gender_survived.get_group('female')[pd.notnull(gender_survived.get_group('female')['Age'])]
female_died_nn = gender_died.get_group('female')[pd.notnull(gender_died.get_group('female')['Age'])]

male_survived_nn = gender_survived.get_group('male')[pd.notnull(gender_survived.get_group('male')['Age'])]
male_died_nn = gender_died.get_group('male')[pd.notnull(gender_died.get_group('male')['Age'])]

fig, (ax1, ax2) = plt.subplots(ncols=2, sharey=True)

sns.distplot(female_survived_nn['Age'], kde=False, ax=ax1, bins=20)
sns.distplot(male_survived_nn['Age'], kde=False, ax=ax2, bins=20)

sns.distplot(female_died_nn['Age'], kde=False, ax=ax1, color='r', bins=20)
sns.distplot(male_died_nn['Age'], kde=False, ax=ax2, color='r', bins=20)


I am working on the file "2015-2016.Meetings/05.deep-learning-with-keras/mnist_mlp_censored.ipynb".
The first task is:
Do the same for our test set, X_test. Don't forget to check how many data points it has with the shape method.
Can you write Python code for it?


I am working on the file "2015-2016.Meetings/05.deep-learning-with-keras/mnist_mlp_censored.ipynb".
The first task is:
Each of our 60,000 handwritten digits comes prepackaged as a 28x28 matrix, which you can think of as a primitive 28x28 pixel photo of a digit. The elements in the matrix take on values between 0 and 255, depending on how dark the writing is in that "pixel".

But we usually train our models on feature vectors, not matrices. Go ahead and use numpy's reshape method to reshape our matrix into a 28x28 = 784-dimensional vector.
Signature
np.reshape(a, newshape, order='C')
Docstring
Gives a new shape to an array without changing its data.
Parameters
a : array_like
Array to be reshaped.
newshape : int or tuple of ints
    The new shape should be compatible with the original shape. If
    an integer, then the result will be a 1-D array of that length.
    One shape dimension can be -1. In this case, the value is inferred
    from the length of the array and remaining dimensions.
Returns
reshaped_array : ndarray
    This will be a new view object if possible; otherwise, it will
    be a copy.
See Also
ndarray.reshape : Equivalent method.
Can you write Python code for it?

# Reshape the 60,000 28x28 matrices in X_train into a 60000x784 dimensional matrix


I am working on the file "2015-2016.Meetings/05.deep-learning-with-keras/mnist_mlp_censored.ipynb".
The first task is:
Here is the blueprint for our model. It's your job to write the code. Reference the Keras docs (or look over this quick introduction for all the code you need to know).

Add a dense hidden layer 512 neurons wide that accepts our 784 dimensional input. (What does it mean for a layer to be dense?).
Give these neurons a 'sigmoid' activation function.
Add another dense hidden layer 512 neurons wide, each neuron with a sigmoid activation function.
Finally, add a dense layer 10 neurons wide with a 'softmax' activation function. (What do you think this layer is doing?)
Can you write Python code for it?


I am working on the file "2015-2016.Meetings/05.deep-learning-with-keras/mnist_mlp_censored.ipynb".
The first task is:
Our model requires two other parameters: a loss (objective) function and an optimization method. For our loss function we will use what Keras calls 'categorical_crossentropy', but which also goes by 'multiclass logloss'. Keras has a fair amount of objective functions to choose from, but categorical_crossentropy is the only one that really makes sense here.
We will use stochastic gradient descent ('sgd') for our optimization method. Stochastic, because it randomly selects a sample from our training data to calculate the gradient during backpropogation (or iterates sequentially over a random shuffling of the data). Gradient descent because... err, it's doing gradient descent.
Can you write Python code for it?


I am working on the file "2015-2016.Meetings/05.deep-learning-with-keras/mnist_mlp_censored.ipynb".
The first task is:
This part just makes sure numpy casts our data values to the correct type.
Can you write Python code for it?

X_train = X_train.astype('float32')
X_test = X_test.astype('float32')


I am working on the file "2015-2016.Meetings/05.deep-learning-with-keras/mnist_mlp_censored.ipynb".
The first task is:
When training neural networks, it's important to normalize our data. There are multiple reasons for this. Normalizing lets us not worry about any scaling effects our input might have on our model (for example, one feature is measured in millimeters, and another in miles). Furthermore, when we randomly initialize the weights of our neurons, we won't have to worry (as much) that our neurons will become saturated. Saturation occurs when the output of our neuron is very near one or zero, no matter the input. Remember, the output of a (sigmoidal) neuron is a linear combination of its inputs passed into a logistic function. The gradient of the logistic function is very small when its output is near 1 or 0, hence the gradient in the backpropogation algorithm will be very small and our neurons weights will update too slowly.
Normalize (Well, standardize, techinically) X_train so that its mean is 0 and standard deviation 1. (Hint: subtract the mean from each datapoint then divide by the standard deviation). Also standardize X_test, but use the mean and standard deviation from the training data.
Can you write Python code for it?


I am working on the file "2015-2016.Meetings/05.deep-learning-with-keras/mnist_mlp_uncensored.ipynb".
The first task is:
At last, the heavy lifting portion of our code, the fitting of the model.
Call model.fit with our training data, a batch_size of 128, and 1 epoch.
A batch is how many datapoints our neural net uses to calculate the gradient before taking a step in the gradient descent process. Batches are taken sequentially from the entire training set during optimization.
An epoch is a single pass through the data during optimization. For example, if we are training on 100 data points, and our batch size is 10, then we will have completed an epoch after processing 10 batches.
Can you write Python code for it?

# Call model.fit with batch_size = 128 and nb_epoch = 1
# If you get an I/O error, make sure your model
# completely finishes compiling before running this line.
model.fit(X_train, Y_train, batch_size=128, nb_epoch = 1,
         validation_data=(X_test, Y_test), show_accuracy=True)

results = model.evaluate(X_test, Y_test, show_accuracy=True)
print("Loss on Test:", results[0])
print("Accuracy on Test:", results[1])


I am working on the file "2015-2016.Meetings/05.deep-learning-with-keras/mnist_mlp_uncensored.ipynb".
The first task is:
But since we're training on the CPU and don't want to wait all day for our models to train, we'll use only the first 6000 data points for our actual training set.
Can you write Python code for it?

X_train = X_train[:6000]
y_train = y_train[:6000]
X_test = X_test[:5000] # we'll use a half-sized test set, too
y_test = y_test[:5000]


I am working on the file "2015-2016.Meetings/05.deep-learning-with-keras/mnist_mlp_uncensored.ipynb".
The first task is:
Do the same for our test set, X_test. Don't forget to check how many data points it has with the shape method.
Can you write Python code for it?

# Do the same for X_test
X_test = X_test.reshape(10000, 784)


I am working on the file "2015-2016.Meetings/05.deep-learning-with-keras/mnist_mlp_uncensored.ipynb".
The first task is:
Each of our 60,000 handwritten digits comes prepackaged as a 28x28 matrix, which you can think of as a primitive 28x28 pixel photo of a digit. The elements in the matrix take on values between 0 and 255, depending on how dark the writing is in that "pixel".

But we usually train our models on feature vectors, not matrices. Go ahead and use numpy's reshape method to reshape our matrix into a 28x28 = 784-dimensional vector.
Signature
np.reshape(a, newshape, order='C')
Docstring
Gives a new shape to an array without changing its data.
Parameters
a : array_like
Array to be reshaped.
newshape : int or tuple of ints
    The new shape should be compatible with the original shape. If
    an integer, then the result will be a 1-D array of that length.
    One shape dimension can be -1. In this case, the value is inferred
    from the length of the array and remaining dimensions.
Returns
reshaped_array : ndarray
    This will be a new view object if possible; otherwise, it will
    be a copy.
See Also
ndarray.reshape : Equivalent method.
Can you write Python code for it?

# Reshape the 60,000 28x28 matrices in X_train into a 60000x784 dimensional matrix
X_train = X_train.reshape(60000, 784)


I am working on the file "2015-2016.Meetings/05.deep-learning-with-keras/mnist_mlp_uncensored.ipynb".
The first task is:
Here is the blueprint for our model. It's your job to write the code. Reference the Keras docs (or look over this quick introduction for all the code you need to know).

Add a dense hidden layer 512 neurons wide that accepts our 784 dimensional input. (What does it mean for a layer to be dense?).
Give these neurons a 'sigmoid' activation function.
Add another dense hidden layer 512 neurons wide, each neuron with a sigmoid activation function.
Finally, add a dense layer 10 neurons wide with a 'softmax' activation function. (What do you think this layer is doing?)
Can you write Python code for it?

# Your multi-layer perceptron here

# sigmoid, no dropout
model.add(Dense(output_dim=684, input_dim=784))
model.add(Activation('sigmoid'))
model.add(Dense(684))
model.add(Activation('sigmoid'))
model.add(Dense(10))
model.add(Activation('softmax'))

# ReLU, no dropout
'''
model.add(Dense(output_dim=684, input_dim=784))
model.add(Activation('relu'))
model.add(Dense(684))
model.add(Activation('relu'))
model.add(Dense(10))
model.add(Activation('softmax'))
'''
'''
# ReLU, dropout
model.add(Dense(output_dim=684, input_dim=784))
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Dense(684))
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Dense(10))
model.add(Activation('softmax'))
'''


I am working on the file "2015-2016.Meetings/05.deep-learning-with-keras/mnist_mlp_uncensored.ipynb".
The first task is:
Now we can finally start building our model! Keras has two model types. Sequential (below) is a typical neural network with layers stacked one on top of the other. The other model, Graph, models any network that can be represented as a directed acyclic graph (DAG). Can you imagine a neural network that violates one or both of the DAG properties? :-)
Can you write Python code for it?


I am working on the file "2015-2016.Meetings/05.deep-learning-with-keras/mnist_mlp_uncensored.ipynb".
The first task is:
Our model requires two other parameters: a loss (objective) function and an optimization method. For our loss function we will use what Keras calls 'categorical_crossentropy', but which also goes by 'multiclass logloss'. Keras has a fair amount of objective functions to choose from, but categorical_crossentropy is the only one that really makes sense here.
We will use stochastic gradient descent ('sgd') for our optimization method. Stochastic, because it randomly selects a sample from our training data to calculate the gradient during backpropogation (or iterates sequentially over a random shuffling of the data). Gradient descent because... err, it's doing gradient descent.
Can you write Python code for it?


I am working on the file "2015-2016.Meetings/05.deep-learning-with-keras/mnist_mlp_uncensored.ipynb".
The first task is:
This part just makes sure numpy casts our data values to the correct type.
Can you write Python code for it?


I am working on the file "2015-2016.Meetings/05.deep-learning-with-keras/mnist_mlp_uncensored.ipynb".
The first task is:
When training neural networks, it's important to normalize our data. There are multiple reasons for this. Normalizing lets us not worry about any scaling effects our input might have on our model (for example, one feature is measured in millimeters, and another in miles). Furthermore, when we randomly initialize the weights of our neurons, we won't have to worry (as much) that our neurons will become saturated. Saturation occurs when the output of our neuron is very near one or zero, no matter the input. Remember, the output of a (sigmoidal) neuron is a linear combination of its inputs passed into a logistic function. The gradient of the logistic function is very small when its output is near 1 or 0, hence the gradient in the backpropogation algorithm will be very small and our neurons weights will update too slowly.
Normalize (Well, standardize, techinically) X_train so that its mean is 0 and standard deviation 1. (Hint: subtract the mean from each datapoint then divide by the standard deviation). Also standardize X_test, but use the mean and standard deviation from the training data.
Can you write Python code for it?

# Standardize X_train and X_test
train_mean = np.mean(X_train)
train_std = np.std(X_train)
X_train -= train_mean
X_train /= train_std
X_test -= train_mean
X_test /= train_std


I am working on the file "2015.04.20-img-qiime-analyses/db-metagenome-correlation.ipynb".
The first task is:
And just for the record, these are Paul Dijkstra's central C metabolic network "EC codes of interest".
Can you write Python code for it?

ec_of_interest = StringIO("""1.1.1.286 (isocitrate dehydrogenase)
1.1.1.37 (malate dehydrogenase to oxaloacetate)
1.1.1.38 (malate dehydrogenase /malic enzyme to pyruvate)
1.1.1.39 (malate dehydrogenase /malic enzyme to pyruvate)
1.1.1.40 (malate dehydrogenase/ malic enzyme to pyruvate)
1.1.1.41 (isocitrate dehydrogenase)
1.1.1.42 (isocitrate dehydrogenase)
1.1.1.44 (6 phosphogluconate dehydrogenase)
1.1.1.49 (glucose-6P dehydrogenase to gluconolactone-P)
1.1.1.82 (malate dehydrogenase to oxaloacetate)
1.1.1.343 (phosphogluconate dehydrogenase)
1.1.1.351 (phosphogluconate dehydrogenase)
1.1.5.4 (malate oxidoreductase to oxaloacetate)
1.2.1.9 (glyceraldehyde dehydrogenase - glycerald to glycerate)
1.2.1.12 (glyceraldehyde3P dehydrogenase)
1.2.1.59 (glyceraldehyde3P dehydrogenase)
1.2.1.79 (succinate semialdehyde dehydrogenase to succinate)
1.2.1.9 (glyceraldehyde3P dehydrogenase)
1.2.4.1 (pyruvate dehydrogenase – split CO2)
1.2.4.2 (oxoglutarate dehydrogenase /decarboxylase/PDH complex E3 – split CO2 from oxoglutarate)
1.2.7.1 (pyruvate synthase /oxidoreductase to acetyl-CoA)
1.2.7.3 (oxoglutarate synthase to succinyl-coa)
1.2.7.6 (glyceraldehyde3P dehydrogenase /oxidoreductase to 3Pglycerate)
1.3.5.1 (succinate dehydrogenase to fumarate)
1.3.5.4 (fumarate reductase to succinate)
1.8.1.4 (dihydrolipoyl dehydrogenase – PDH and OGL complex – NADH generating subunit)
2.2.1.1 (transketolase ppp reaction)
2.2.1.2 (transaldolase ppp reaction)
2.3.1.12 (PDH complex number 3 produces acetyl CoA)
2.3.1.61 (dihydropoyllysine-residue succinyl transferase – complex 3 to succinyl-CoA)
2.3.3.1 (citrate synthase OAA to citrate)
2.3.3.16 (citrate synthase OAA to citrate)
2.3.3.8 (citrate synthase OAA to citrate)
2.7.1.1 (hexokinase – to G6P)
2.7.1.11 (6 phosphofructokinase F6 to F1,6)
2.7.1.146 (ADP-specific phosphofructokinase from F6P to F16P)
2.7.1.147 (ADP specific glucokinase to G6P)
2.7.1.2 (glucokinase)
2.7.1.40 (pyruvate kinase to PEP)
2.7.1.90 (phosphofructokinase 6 to 16 fructose)
2.7.1.146 (ADP specific...


I am working on the file "2015.04.20-img-qiime-analyses/db-metagenome-correlation.ipynb".
The first task is:
It also looks like, in general, the abundances are correlated across soils, which is also a good control (i.e., they are fairly repeatable across metagenomes, so we're probably not just looking at random noise).
Can you write Python code for it?


I am working on the file "2015.04.20-img-qiime-analyses/db-metagenome-correlation.ipynb".
The first task is:
It looks like the counts on a per-metagenome basis are not just reflections of the abundances of those EC codes in the database, which is good. This was performed as a control.
Can you write Python code for it?


I am working on the file "2015.04.20-img-qiime-analyses/db-metagenome-correlation.ipynb".
The first task is:
These are the Central C metabolic network EC code abundances in the Fierer et al 2012 soil metagenomes, as computed with QIIME 1.9.0 by Greg Caporaso. Here I load these, and then add in the corresponding EC counts from the QIIME/IMG reference database (25 Oct 2012).
Can you write Python code for it?

ec_biom = pd.read_csv('observation_table_L1.txt', sep='\t', header=1, index_col=0)
ec_biom['IMG Count'] = [img_ec_counts['Gene ID'][e] for e in ec_biom.index]

ec_biom


I am working on the file "2015.07.02-scipy2015-experiments/forensics.ipynb".
The first task is:
Do some fancy sequence exploration to show off the repr...
Can you write Python code for it?

for rec in skbio.io.read(fp, format='fasta'):
    print(rec, rec.id)
    break


I am working on the file "2015.07.02-scipy2015-experiments/forensics.ipynb".
The first task is:
Figure out how many sequences we have for each subject, and then how many unique sequences we have for each subject
Can you write Python code for it?

for k, v in data.items():
    print k, len(v), len(set(v))

for k, v in data.items():
    data[k] = set(v)
    print k, len(data[k])


I am working on the file "2015.07.02-scipy2015-experiments/forensics.ipynb".
The first task is:
Hash the sequences on a per subject basis
Can you write Python code for it?

from collections import defaultdict
data = defaultdict(list)
for rec in skbio.io.read(fp, format='fasta'):
    subject = rec.id.split('.')[0]
    data[subject].append(rec)


I am working on the file "2015.07.02-scipy2015-experiments/forensics.ipynb".
The first task is:
The largest union of sequences always provides the best match of subject to keyboard.
Can you write Python code for it?

# maybe load these data into a distance matrix to show that repr?
for i in ['M2', 'M3', 'M9']:
    for j in ['K1', 'K2', 'K3']:
        print i, j, len(data[i] & data[j])
    print ''


I am working on the file "2015.07.02-scipy2015-experiments/human-genome.ipynb".
The first task is:
Step 2 (no skbio here)
Can you write Python code for it?

import pandas as pd
import numpy as np

# this file isn't public b/c I'm not sure that I want it to be. Greg will share the link...
greg_data_fp = "genome_Gregory_Caporaso_Full_20150702133442.txt"
df = pd.read_table(greg_data_fp,
                   comment='#',
                   names=["rsid", "chromosome", "position", "genotype"])
chromosome_in_range = df['chromosome'] == chromosome
position_in_range = np.logical_and(start_pos <= df['position'], end_pos >= df['position'])

gregs_data = df[np.logical_and(chromosome_in_range, position_in_range)]
gregs_data[:10]


I am working on the file "2015.07.02-scipy2015-experiments/human-genome.ipynb".
The first task is:
Step 3
Can you write Python code for it?

ref

# Find out what base is in the reference for one of my SNPs. In this case,
# I'm heterozygous for A/G at this position, and the reference has a G.
ref[33179371 - start_pos]

# Find out how many times I'm the same as the reference.

count_same = 0
count = 0
for e in gregs_data.iterrows():
    count += 1
    if str(ref[e[1].position - start_pos]) in e[1].genotype:
        count_same += 1

print count_same / count, count


I am working on the file "2015.07.02-scipy2015-experiments/human-genome.ipynb".
The first task is:
Step 4
Building fasta from vcf files
 - trying this with GATK (requires java 1.7.0 or greater) based on the Biostars discussion here.
Can you write Python code for it?

# First we need to create the index and dict files, as described here:
# https://www.broadinstitute.org/gatk/guide/article?id=2798

reference_dict_fp = '%s.dict' % reference_fasta_fp
!java -jar /Applications/picard-tools-1.135/picard.jar CreateSequenceDictionary R= $reference_fasta_fp O= $reference_dict_fp

reference_idx_fp = '%s.fai' % reference_fasta_fp
!samtools faidx $reference_fasta_fp

# Then, get the vcf file corresponding to this region - I don't think we can change this URL,
# it points to a file that I initiated creation of on the web server.

!wget http://browser.1000genomes.org/tmp/slicer/filtered_21.33177777-33708621.ALL.chr21.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz
!gunzip filtered_21.33177777-33708621.ALL.chr21.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz
!head filtered_21.33177777-33708621.ALL.chr21.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf

!java -Xmx2g -jar /Applications/GenomeAnalysisTK.jar -R ensembl.txt -T FastaAlternateReferenceMaker -o MY_REFERENCE_WITH_SNPS_FROM_VCF.fa --variant filtered_21.33177777-33708621.ALL.chr21.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf

## For some reason, the result is the same as my reference sequence... Is this a problem with how I'm
## using GATK, or with my VCF file (i.e., there really is no variation from the reference). Also,
## this is only generating one sequence. I want one per subject...

## Looking at the vcf file, I see a lot of "0|0" in the first column of subject variant data,
## so it may really be that there is no variation in this region and I should pick another. Puzzling though
## that I have so much variation if that's the case - am I doing something wrong there??

vcp_snped_ref = skbio.io.read('MY_REFERENCE_WITH_SNPS_FROM_VCF.fa', into=skbio.DNA)
len(vcp_snped_ref), len(ref)
vcp_snped_ref.fraction_diff(ref)


I am working on the file "2015.07.02-scipy2015-experiments/human-genome.ipynb".
The first task is:
Step 5 (not really started yet)
Can you write Python code for it?

# This is the 1000 genomes sample (i.e., host) metadata (John, do you know where the lookup is for the "pop" field?)

!wget ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/integrated_call_samples_v3.20130502.ALL.panel
!head -n 50 integrated_call_samples_v3.20130502.ALL.panel


I am working on the file "2015.07.12-species-classifiers.ipynb".
The first task is:
Next, let's apply this to all of our unaligned sequence and find out how many reference sequences our pattern matches.
Can you write Python code for it?

seq_count = 0
match_count = 0
for seq in skbio.io.read(unaligned_ref_fp, format='fasta',
                         constructor=skbio.DNA):
    seq_count += 1
    for match in seq.find_with_regex(regex):
        match_count += 1

match_percentage = (match_count / seq_count) * 100
print('{0} of {1} ({2:.2f}%) sequences have exact matches to the regular expression.'.format(match_count, seq_count, match_percentage))


I am working on the file "2015.07.12-species-classifiers.ipynb".
The first task is:
Several recent studies of amplicon taxonomic assignment methods  (Mizrahi-Man et al. 2013, Werner et al. 2012) have suggested that training Naive Bayes taxonomic classifiers against only the region of a sequence that was amplified, rather than a full length sequence, will give better taxonomic assignment results. So, lets start by slicing our reference sequences by finding some commonly used 16S primers so we train only on the fragment of the 16S that we would amplify in an amplicon survey.
We'll define the forward and reverse primers as skbio.DNA objects. The primers that we're using here are pulled from Supplementary File 1 of Caporaso et al. 2012. Note that we're reverse complementing the reverse primer when we load it here so that it's in the same orientation as our reference sequences.
Can you write Python code for it?

fwd_primer = skbio.DNA("GTGCCAGCMGCCGCGGTAA", {'id':'fwd-primer'})
rev_primer = skbio.DNA("GGACTACHVGGGTWTCTAAT", {'id':'rev-primer'}).reverse_complement()


I am working on the file "2015.07.12-species-classifiers.ipynb".
The first task is:
So we're matching only about 80% of our reference sequences with this pattern. The implication for this application is that we'd only know how to slice 80% of our sequences, and as a result, we'd only have 80% of our sequence to train on. In addition to this being a problem because we want to train on as many sequences possible, it's very likely that there are certain taxonomic groups are left out all together. So, using regular expressions this way won't work.
However... this is exactly what multiple sequence alignments are good for. If we could match our primers against aligned reference sequences, then finding matches in 80% of our sequences would give us an idea of how to slice all of our sequences, since the purpose of a multiple sequence alignment is to normalize the position numbers across all of the sequences in a sequence collection. The problem is that the gaps in the alignment would make it harder to match our regular expression, as gaps would show up that disrupt our matches. We can get around this using the ignore parameter to DNA.find_with_regex, which takes a boolean vector (a fancy name for an array or list of boolean values) indicating positions that should be ignore in the regular expression match. Let's try applying our regular expression to the aligned reference sequences and keeping track of where each match starts and stops.
Can you write Python code for it?

starts = []
stops = []
for seq in skbio.io.read(aligned_ref_fp, format='fasta',
                         constructor=skbio.DNA):
    for match in seq.find_with_regex(regex, ignore=seq.gaps()):
        starts.append(match.start)
        stops.append(match.stop)


I am working on the file "2015.07.12-species-classifiers.ipynb".
The first task is:
The typical way to approach the problem of finding the boundaries of a short sequence in a longer sequence would be to use pairwise alignment. But, we're going to try a different approach here since pairwise alignment is inherently slow (it scales quadratically). Because these are sequencing primers, they're designed to be unique (so there shouldn't be multiple matches of a primer to a sequence), and they're designed to match as many sequences as possible. So let's try using regular expressions to match our sequencing primers in the reference database. Regular expression matching scales lineaerly, so is much faster to apply to many sequences.
First, we'll define a function to generate a regular expression from a Sequence object. This functionality will be in scikit-bio's next official release (it was recently added as part of issue #1005).
Can you write Python code for it?

def seq_to_regex(seq):
    """ Convert a sequence to a regular expression
    """
    result = []
    sequence_class = seq.__class__
    for base in str(seq):
        if base in sequence_class.degenerate_chars:
            result.append('[{0}]'.format(
                ''.join(sequence_class.degenerate_map[base])))
        else:
            result.append(base)

    return ''.join(result)


I am working on the file "2015.07.12-species-classifiers.ipynb".
The first task is:
We can then apply this to define a regular expression that will match our forward primer, the following sequence, and then the reverse primer. We can use the resulting matches then to find the region of our sequences that is bound by our forward and reverse primer.
Can you write Python code for it?

regex = '({0}.*{1})'.format(seq_to_regex(fwd_primer),
                            seq_to_regex(rev_primer))

regex


I am working on the file "2015.07.22-upgma-v-nj/experiments.ipynb".
The first task is:
Build a WPGMA tree
Can you write Python code for it?

from scipy.cluster.hierarchy import weighted
lm = weighted(dm.condensed_form())
wpgma_tree = TreeNode.from_linkage_matrix(lm, dm.ids)
# reroot
new_root = wpgma_tree.find('hemoglobin-human').ancestors()[0]
wpgma_tree = wpgma_tree.root_at(new_root)

wpgma_tree_fp = 'globin-aln.wpgma.ph'
wpgma_tree.write(wpgma_tree_fp)

ete3.Tree(wpgma_tree_fp, format=1).render("%%inline", units="mm", tree_style=ts)


I am working on the file "2015.07.22-upgma-v-nj/experiments.ipynb".
The first task is:
Building a neighbor joining tree
A common reason for building a multiple sequence alignment is for making an inference about the evolutionary relationships between the organisms. We'll do this here using the neighbor joining (nj) function in scikit-bio.
The input to nj is a distance matrix, representing the pairwise distances between the sequences in the alignment. This can be obtained using the distances method, which by default will compute the Hamming distances between all pairs of sequences in the alignment:
We can now build a neighbor joining tree from this distance matrix. At this point we don't have an easy and good visualization approach in scikit-bio for phylogenetic trees, though we're working on a solution. You can track progress on that here.
In the meantime, we're going to export a newick-formatted version of the tree which can be opened with a phylogenetic tree viewing program. We'll show some exported images that we obtained using Archeoptyrx, which works on OS X, Windows, and Linux, but there are many popular tree viewing programs, and most support the newick file format. A popular one for OS X is FigTree.
First, let's build the neighbor-joining tree:
Can you write Python code for it?

from skbio.tree import nj
nj_tree = nj(dm)


I am working on the file "2015.07.22-upgma-v-nj/experiments.ipynb".
The first task is:
To get a better understanding of the evolutionary history, we can use outgroup rooting to place a root in the tree. When doing outgroup rooting, you want to find the tip corresponding to the most distantly related sequence in the alignment, and look at its most recent ancestral node. Because we know a priori that the hemoglobin sequence is the most distantly related of the sequences in our alignment (because it's not myoglobin, so by definition had to have begun diverging before all of our myoglobin sequences began diverging), you know that its first ancestral node should be the common ancestor of all sequences in the tree.
First we find the node that we want to be the new root.
Can you write Python code for it?


I am working on the file "2015.07.22-upgma-v-nj/experiments.ipynb".
The first task is:
We can then write this to file in newick format:
Can you write Python code for it?

nj_tree_fp = 'globin-aln.nj.ph'
nj_tree.write(nj_tree_fp)

ete3.Tree(nj_tree_fp, format=1).render("%%inline", units="mm", tree_style=ts)


I am working on the file "2015/20150111_2DPlotsonPythonP3.ipynb".
The first task is:
Five plots in one window, different rows.
Can you write Python code for it?


I am working on the file "2015/20150111_2DPlotsonPythonP3.ipynb".
The first task is:
Plots of trigonometric functions divided in the same window.
Can you write Python code for it?

from pylab import *

t = arange(0.0,2.0,0.01)

fig, ax = subplots(3, 2, sharex=True)

ax[0, 0].plot(t, sin(pi/8*t), color='green', linestyle='-', linewidth=3)
ax[0, 1].plot(t, cos(pi/8*t), color='red', linestyle='--', linewidth=3)
ax[1, 0].plot(t, tan(pi/8*t), color='cyan', linestyle='-.', linewidth=3)
ax[1, 1].plot(t, 1/cos(pi/8*t), color='magenta', linestyle=':', linewidth=3)
ax[2, 0].plot(t, 1/sin(pi/8*t), color='yellow', linestyle='--', linewidth=3)
ax[2, 1].plot(t, 1/tan(pi/8*t), color='black', linestyle=':', linewidth=3)

show()


I am working on the file "2015/20150111_2DPlotsonPythonP3.ipynb".
The first task is:
Two plots in one window, different columns.
Can you write Python code for it?

from pylab import *

t = arange(0.0,2.0,0.01)
y1 = sin(2*pi*t)
y2 = cos(2*pi*t)

fig, ax = subplots(1, 2, sharex=True)

ax[0].plot(t, y1, color='green', linestyle='-.', linewidth=3)
ax[1].plot(t, y2, color='red', linestyle=':', linewidth=3)

show()


I am working on the file "2015/20150111_2DPlotsonPythonP3.ipynb".
The first task is:
subplot(): plots in 3 rows, and 2 columns.
Can you write Python code for it?


I am working on the file "2015/ferran/day12.ipynb".
The first task is:
Day 12.2
A function to transform terms of the form "[key]":"red" into a single character 'R'.
Can you write Python code for it?

def transform_reds(s):
    q = re.compile('\"[\w]+\"\:\"red\"')
    return q.sub('R', s)


I am working on the file "2015/ferran/day12.ipynb".
The first task is:
Test
Can you write Python code for it?


I am working on the file "2015/ferran/day12.ipynb".
The first task is:
Track the regions to ignore: when an 'R' is found at depth d we keep this information; we ignore the span between the last $[d-1,d]$ transition (sink down) and the next $[d,d-1]$ transition (float up). Those regions will be erased.
Can you write Python code for it?

def regions_to_erase(s):
    regions = []
    curr_depth = 0
    last_sink = {}
    red = None
    for i, c in enumerate(s):
        if c == '{':
            curr_depth += 1
            if red is None:
                last_sink[curr_depth] = i
        elif c == 'R':
            ignore = True
            if red is None:
                red = curr_depth
        elif c == '}':
            if red is not None:
                if curr_depth == red:
                    regions.append([last_sink[curr_depth], i])
                    red = None
            curr_depth -= 1
    return regions


I am working on the file "2015/ferran/day7.ipynb".
The first task is:
Although the approach passes the test, it does not end in reasonable time for the full input.
Approach 4: With Lazy Evaluation in R
The approach now is to exploit the lazy evaluation capabilities in R. So we leverage Python to create an R script that does the job.
Can you write Python code for it?

def rscript_command(var, l):
    vocab =  {'AND'    : 'bitwAnd',
              'OR'     : 'bitwOr',
              'LSHIFT' : 'bitwShiftL',
              'RSHIFT' : 'bitwShiftR'}
    if len(l) == 3:
        func = vocab[l[1]]
        arg1 = l[0] if l[0].isdigit() else 'var_' + l[0] + '()'
        arg2 = l[2] if l[2].isdigit() else 'var_' + l[2] + '()'
        return 'var_{0} <- function(a={1}, b={2})'.format(var, arg1, arg2) + ' {' + '{0}(a,b)'.format(func) + '}'
    elif len(l) == 2:
        func = 'bitwNot'
        arg1 = l[1] if l[1].isdigit() else 'var_' + l[1] + '()'
        return 'var_{0} <- function(a={1})'.format(var, arg1) + ' {' + '{0}(a)'.format(func) + '}'
    else:
        arg1 = l[0] if l[0].isdigit() else 'var_' + l[0] + '()'
        return 'var_{0} <- function(a={1})'.format(var, arg1) + ' {' + 'a' + '}'

def generate_rscript(commands, target):
    with open('day7_commands.R', 'wt') as f:
        for k, v in commands.items():
            f.write(rscript_command(k, v)+'\n')
        f.write('var_' + target + '()')


I am working on the file "2015/ferran/day7.ipynb".
The first task is:
Solution
Can you write Python code for it?

commands = display('inputs/input7.txt')
evaluate('a')


I am working on the file "2015/ferran/day7.ipynb".
The first task is:
Test
Can you write Python code for it?

commands = display('inputs/input7.test2.txt')

commands = display('inputs/input7.test2.txt')
for k, v in commands.items():
    command_str = '{0} = LazyVar(lambda: {1})'.format('var_' + k, ''.join(translate(v)))
    print(command_str)
    exec(command_str)

def test():
    assert(var_d.func() == 72)
    assert(var_e.func() == 507)
    assert(var_f.func() == 492)
    assert(var_g.func() == 114)
    assert(var_h.func() == 65412)
    assert(var_i.func() == 65079)
    assert(var_x.func() == 123)
    assert(var_y.func() == 456)

test()


commands = display('inputs/input7.test2.txt')
generate_rscript(commands, 'd')

! cat day7_commands.R

!Rscript day7_commands.R


I am working on the file "2015/ferran/day7.ipynb".
The first task is:
Test 0
Can you write Python code for it?

commands = display('inputs/input7.test1.txt')
commands

evaluate('a')


I am working on the file "2015/ferran/day7.ipynb".
The first task is:
Test 1
Can you write Python code for it?

commands = display('inputs/input7.test2.txt')
commands

test()


I am working on the file "20150223-cohpy-memoization.ipynb".
The first task is:
After the 2015-02-23 CohPy presentation of the above, it was suggested to use the .get() method of dictionaries to avoid the if statement inside helper(). The point is to make the code simpler and more elegant. Unfortunately, the .get() method would not change the dictionary, so the return value from the .get() method would have to be saved. That new memoize function would look like the following. It does succeed in getting rid the if statement, but the 'results[x] = results' looks ugly. Even worse, f(x) is always called, defeating the memoization.
Can you write Python code for it?

def memoize(f):
    results = {}
    def helper(x):
        results[x] = results.get(x, f(x))
        return results[x]
    return helper

@memoize
def fib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib(n-1) + fib(n-2)

%timeit fib(n)
fib(n)


I am working on the file "20150223-cohpy-memoization.ipynb".
The first task is:
Another example of how to avoid the parentheses, without having to explicitly index each element of the tuple, is to use a leading '*' to unpack the tuple into separate arguments to the print function.
Can you write Python code for it?

for item in sorted(c.items(), key=(lambda x: x[0]), reverse=True):
    print(*item)
print()
print(sum(c.values()))


I am working on the file "20150223-cohpy-memoization.ipynb".
The first task is:
No wonder it is so slow. By the way, do you see a pattern in the numbers?
By the way, here's another way of sorting and iterating through the counts.
Can you write Python code for it?

for item in sorted(c.items(), key=(lambda x: x[0]), reverse=True):
    print(item)
print()
print(sum(c.values()))


I am working on the file "20150223-cohpy-memoization.ipynb".
The first task is:
Now let's clean up the code. First, let's remove the counting stuff, since we know the answer for that now.
Can you write Python code for it?

def fib(n):
    global cache

    if n in cache:
        return cache[n]
    if n == 0:
        f = 0
    elif n == 1:
        f = 1
    else:
        f = fib(n-1) + fib(n-2)
    cache[n] = f
    return f

def foo(n):
    global cache

    cache = {}
    fib(n)
%timeit foo(n)


I am working on the file "20150223-cohpy-memoization.ipynb".
The first task is:
Taking out the diagnostic counting code made it faster yet.
The caching code makes the fib() function ugly and I don't
like needing to remember to initialize the cache,
so let's separate the caching into a separate function.
Note that memoize() doesn't know anything about the fib() function,
except that it takes a single argument.
memoize() can be applied to many single argument functions.
Can you write Python code for it?

def memoize(f):
    results = {}
    def helper(x):
        if x not in results:
            results[x] = f(x)
        return results[x]
    return helper

def fib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib(n-1) + fib(n-2)

fib = memoize(fib)

def foo(n):
    def fib(n):
        if n == 0:
            return 0
        elif n == 1:
            return 1
        else:
            return fib(n-1) + fib(n-2)
    fib = memoize(fib)
    %timeit -n 1 fib(n)

for i in range(20):
    foo(n)


I am working on the file "20150223-cohpy-memoization.ipynb".
The first task is:
That is over 40,000 times faster than the naïve version. Let's look at the details.
Can you write Python code for it?

c = Counter()
cache = {}
fib(n)
for i in sorted(c.keys(), reverse=True):
    print(i, cache[i], c[i])
print()
print(sum(c.values()))


I am working on the file "20150223-cohpy-memoization.ipynb".
The first task is:
That yielded about the same speed. The code is now both simple and fast.
How would one measure the speed of starting with an empty cache each time?
Review
What are the limitations of memoize() and @memoize?

Only works for functions that accept a single argument.
The function argument must be acceptable as a dictionary key.
Only works for functions, not generators.
Size of results dictionary might become too large.

How would you work around those limitations?

Only works for functions that accept a single argument.
Try args and *kwargs stuff.

The function argument(s) must be acceptable as a dictionary key.
Might try a partial solution. Save the result when the input can be used as a dictionary key, otherwise, just calculate the result.

Only works for functions, not generators.
Could use cache internal to generator (but would lack elegant general wrapper function or decorator like memoize).

Size of results dictionary might become too large.
Might limit the size of the dictionary.
Beyond that, just calculate the result.
Might also save only the results for the most common input, or the results which are the most "expensive" to calculate.

Might have multi-stage caching, where first cache is dictionary within Python, and secondary or tertiary caches are saved outside of Python, such as in a file, a database, or a server.
Can you write Python code for it?

# By the way, this computer is modest.
!lscpu


I am working on the file "20150223-cohpy-memoization.ipynb".
The first task is:
That yielded yet another big speed increase. It's now over a million times faster than the original function.
Can you write Python code for it?


I am working on the file "20150223-cohpy-memoization.ipynb".
The first task is:
That's faster yet, but perhaps deceptively so.
%timeit did fib(n) many times, but initialized the results cache only once.
Next, let's use the syntactic sugar of a function decorator to eliminate the fib = memoize(fib) statement.
The @memoize decorator can be used for many single argument functions.
Can you write Python code for it?

@memoize
def fib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib(n-1) + fib(n-2)

%timeit fib(n)


I am working on the file "20150223-cohpy-memoization.ipynb".
The first task is:
The problem is that the naïve function recalculates Fibonacci numbers.
Some code is added to keep track of how many times the function is called for each input value.
Can you write Python code for it?

from collections import Counter

def fib(n):
    global c

    c[n] += 1
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib(n-1) + fib(n-2)

c = Counter()
fib(n)

for i in sorted(c, reverse=True):
    print(i, c[i])
print()
print(sum(c.values()))


I am working on the file "20150223-cohpy-memoization.ipynb".
The first task is:
We start with a function that calculates fibonacci numbers.
Its code is very much like the definition of Fibonacci numbers.
Can you write Python code for it?

def fib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib(n-1) + fib(n-2)


I am working on the file "20150327-dojo-join.ipynb".
The first task is:
It accepts any kind of iterable that gives strings.
The following example uses a trivial generator function.
Can you write Python code for it?

def foo():
    yield 'hello'
    yield 'gnew'
    yield 'world'

' and '.join(foo())


I am working on the file "20150601-dojo-in-membership-tests-lists-tuples-sets-dicts.ipynb".
The first task is:
Determining if something is in the set was about 2.5 million times faster
than determining if something is in the list.
Can you write Python code for it?


I am working on the file "20150601-dojo-in-membership-tests-lists-tuples-sets-dicts.ipynb".
The first task is:
Determining if something is in the tuple
takes about the same time as determining if something is in the list.
Can you write Python code for it?


I am working on the file "20150723_inversion.ipynb".
The first task is:
3ème partie : travail sur la méthode d'accordage
Un des problèmes des parties précédentes provient du fait que l'accordage modifie au fur et à mesure les tensions des cordes et que lorsqu'on veut accorder la deuxième corde, on se pose la question de sa tension cible, qui a déjà changé par rapport aux conditions initiales.
Il y a deux possibilités :

calculer la tension d'arrivée à l'issue de l'accordage de la première corde prédite par le modèle pour toutes les cordes et l'utiliser pour calculer la tension cible de la deuxième corde
remesurer toutes les tensions des cordes et ajouter l'incrément de tension nécessaire à la deuxième corde selon ce qui a été calculé par le modèle

Idéalement, c'est à dire si le modèle est correct et que les paramètres ont été bien estimés, les deux méthodes se rejoignent.
Dans cette partie, nous allons confronter dans un premier temps les variations mesurées à l'issue d'un désaccordage avec les variations prédites. Dans un deuxième temps, nous conclurons sur la méthode à utiliser parmi les deux possibilités évoquées ci-dessus.
Validation précise du modèle
Plan de cette partie :

ré-inverser le modèle avec des nouvelles mesures indépendantes de la première série ci-dessus ; comparer les rho_i obtenus (on s'attend à trouver à peu près la même chose !)
faire les étapes d'accordage une par une et comparer à chaque étape la fréquence mesurée sur la guitare pour chaque corde ainsi que celle prédite par la méthode

Inversion du modèle
Propriétés de la guitare connues à l'avance :
Can you write Python code for it?

freqs = np.array([82.4, 110., 146.8, 196., 246.9, 329.6]) # frequencies of the guitar strings, from low E to high E, in Hz
calibration_tensions = np.array([7.94, 8.84, 8.34, 6.67, 4.99, 5.94]) * 9.81 # calibration tensions found on package (in kg) converted to N
mu = calibration_tensions / (4 * 0.648**2 * freqs**2)
psi = 4 * 0.648**2 * mu


I am working on the file "20150723_inversion.ipynb".
The first task is:
Afin de faire les prochaines mesures, on écrit la partie restante de l'application avec PyQt.
On désaccorde successivement le mi grave et le la pour obtenir les relevés suivants (à 0.1 Hz) :
Can you write Python code for it?


I am working on the file "20150723_inversion.ipynb".
The first task is:
Bizarre la vérification...
Can you write Python code for it?

tuning_mat = compute_tuning_matrix(psi, rho)

tuning_mat

target_freqs = freqs.copy()

current_freqs = history[2, :]

target_freqs - current_freqs

target_dF = (target_freqs - current_freqs) ** 2

Delta_F = np.linalg.solve(tuning_mat, target_dF)
Delta_F

np.sqrt(Delta_F)

for _ in np.sqrt(Delta_F):
    print("{:.2f}".format(_))


I am working on the file "20150723_inversion.ipynb".
The first task is:
Ceci est peut être dû à l'erreur à l'issue de la régression linéaire... on va quand même essayer d'implémenter la méthode d'inversion pour l'accordage.
ou au mauvais $\mu_i$, qui ont été choisis... au hasard ?
Construction de la matrice nécessaire à la méthode d'accordage
Il faut ici encore construire la matrice responsable de l'accordage.
Can you write Python code for it?

tuning_mat = np.zeros((6, 6))

for other_string in range(6):
    for tuning_string in range(6):
        if tuning_string == other_string:
            tuning_mat[other_string, tuning_string] = 1.
        else:
            tuning_mat[other_string, tuning_string] = \
                    psi[tuning_string] / psi[other_string] * \
                        (- rho[other_string] / (1 + np.sum([rho[i] for i in range(6) if i != tuning_string])))

tuning_mat

np.dot(tuning_mat, np.array([1, 0, 0, 0, 0, 0]))


I am working on the file "20150723_inversion.ipynb".
The first task is:
Cela devrait marcher. Voyons voir quels deltas de fréquence il faut imposer.
Can you write Python code for it?


I am working on the file "20150723_inversion.ipynb".
The first task is:
Cette fois-ci, la prédiction n'est pas si mauvaise...
Prédictions successives du modèle
On peut maintenant calculer la matrice d'accord (dans le cadre du modèle linéaire).
Can you write Python code for it?

tuning_mat = compute_tuning_matrix(psi, rho)

tuning_mat


I am working on the file "20150723_inversion.ipynb".
The first task is:
Conclusion : ça n'a pas marché et le pont n'a pas arrêté de monter. Je pense que cela vient d'autre chose : la tension du set de cordes n'était pas adaptée !!!
Je change donc de set de cordes et je recommence.
2ème partie : nouvelles cordes, avec propriétés connues
J'ai mis des nouvelles cordes dont les propriétés sont les suivantes :

mi aigu .009 5.94 kg
si .011 4.99 kg
sol .016 6.67 kg
ré .026 8.34 kg
la .036 8.84 kg
mi grave .046 7.94 kg

On en déduit les $\mu_i$ :
Can you write Python code for it?

freqs = np.array([82.4, 110., 146.8, 196., 246.9, 329.6]) # frequencies of the guitar strings, from low E to high E, in Hz
calibration_tensions = np.array([7.94, 8.84, 8.34, 6.67, 4.99, 5.94]) * 9.81 # calibration tensions found on package (in kg) converted to N
mu = calibration_tensions / (4 * 0.648**2 * freqs**2)

mu


I am working on the file "20150723_inversion.ipynb".
The first task is:
Essai pratique
Testons !
Can you write Python code for it?


I am working on the file "20150723_inversion.ipynb".
The first task is:
Et maintenant, on peut prédire les changements à partir d'un état vers un autre :
Can you write Python code for it?

def predict_changes(initial_T, final_T, tuning_mat):
    """predicts changes in tuning (frequency) from a vector of tensions"""
    print("initial tunings: {:.2f}, {:.2f}, {:.2f}, {:.2f}, {:.2f}, {:.2f}".format(*[x for x in np.sqrt(initial_T))]))
    print("final tunings: {:.2f}, {:.2f}, {:.2f}, {:.2f}, {:.2f}, {:.2f}".format(*[x for x in np.sqrt(final_T))]))
    print("predicted tunings")


I am working on the file "20150723_inversion.ipynb".
The first task is:
Etape 0 : le programme
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt

initial_freqs = current_freqs.copy()


I am working on the file "20150723_inversion.ipynb".
The first task is:
Il nous faut construire le vecteur des $\Delta T _ i$ à partir des mesures de fréquences. On construit d'abord le vecteur des $F_i$
Can you write Python code for it?


I am working on the file "20150723_inversion.ipynb".
The first task is:
Inversion du système ainsi construit
On peut maintenant inverser le système :
Can you write Python code for it?

total_mat.shape

total_rhs.shape

rho, err, rank, eigs = np.linalg.lstsq(total_mat, total_rhs)

err

rho

err


I am working on the file "20150723_inversion.ipynb".
The first task is:
J'ai envie de penser que cela suffit.
à faire demain :

refaire une inversion
refaire un accordage en quantifiant l'erreur en demi-tons par rapport à la note cible

20150811
Effet du serrage des boulons en tête de guitare sur l'accordage :

avant : 83.3,112.8,150.0,201.3,250.2,334.3
après : 83.1,112.9,149.7,201.3,250.4,334.5

Conclusion : petit effet, on va négliger ça.
J'utilise le protocole d'accordage suivant (attention différent de celui de l'étape précédente où j'avais descendu le mi et monté le la) :

je monte le mi grave
je descend le la

Au préalable, je met toutes les vis dans leurs positions neutres sur le vibrato pour le fine tuning.
Inversion
Can you write Python code for it?

freqs = np.array([82.4, 110., 146.8, 196., 246.9, 329.6]) # frequencies of the guitar strings, from low E to high E, in Hz
calibration_tensions = np.array([7.94, 8.84, 8.34, 6.67, 4.99, 5.94]) * 9.81 # calibration tensions found on package (in kg) converted to N
mu = calibration_tensions / (4 * 0.648**2 * freqs**2)
psi = 4 * 0.648**2 * mu

history = np.array([[83.7,112.7,151.3,204.2,251.2,333.4],
                    [84.9,112.3,150.9,203.3,250.5,332.8],
                    [85.0,109.4,151.1,203.9,250.9,333.0]])
history

T = (history ** 2) * psi

T.sum(axis=1) / 9.81

mat1, rhs1 = make_matrix(T[1, :], T[0, :])
mat2, rhs2 = make_matrix(T[2, :], T[1, :])
total_mat = np.vstack((mat1, mat2))
total_rhs = np.vstack((rhs1[:, np.newaxis],
                       rhs2[:, np.newaxis]))
rho, err, rank, eigs = np.linalg.lstsq(total_mat, total_rhs)

rho

err

np.dot(total_mat, rho)

total_rhs


I am working on the file "20150723_inversion.ipynb".
The first task is:
J'ai remesuré la longueur des cordes et j'ai trouvé 65 cm. Je décide de garder la longueur standard de 0.648 m.
On en déduit les Psi_i :
Can you write Python code for it?


I am working on the file "20150723_inversion.ipynb".
The first task is:
L'écart que l'on cherche à obtenir est donc :
Can you write Python code for it?


I am working on the file "20150723_inversion.ipynb".
The first task is:
La
Can you write Python code for it?

measured_freqs = [82.3,110.0,148.8,194.1,244.5,329.2]
expected_freqs = tuning_step(tuning_mat, initial_freqs, Delta_F, 2)
print_strings(measured_freqs)
print_strings(expected_freqs)
plt.plot(measured_freqs, 'o')
plt.plot(expected_freqs, 'o')


I am working on the file "20150723_inversion.ipynb".
The first task is:
Le $\Delta f$ à appliquer peut se calculer en faisant la différence à obtenir :
Can you write Python code for it?

target_freqs = freqs.copy()

current_freqs = df.values[2, :]


I am working on the file "20150723_inversion.ipynb".
The first task is:
Les fréquences cibles des cordes sont :
Can you write Python code for it?

for target, string in zip([tuning_step(tuning_mat, initial_freqs, Delta_F, i)[i-1] for i in range(1, 7)],
                          ["low E", "A", "D", "G", "B", "high E"]):
    print("string: {}, target frequency: {:.1f}".format(string, target))


I am working on the file "20150723_inversion.ipynb".
The first task is:
Mi :
Can you write Python code for it?

measured_freqs = [83.2,110.75,147.17,196.55,246.67,329.84]
expected_freqs = tuning_step(tuning_mat, initial_freqs, Delta_F, 6)
print_strings(measured_freqs)
print_strings(expected_freqs)


I am working on the file "20150723_inversion.ipynb".
The first task is:
On doit le mettre au carré :
Can you write Python code for it?

target_dF = (target_freqs - current_freqs) ** 2

Delta_F = np.linalg.solve(tuning_mat, target_dF)
Delta_F


I am working on the file "20150723_inversion.ipynb".
The first task is:
On doit prendre la racine de ce grand F pour trouver les fréquences à appliquer :
Can you write Python code for it?

np.sqrt(Delta_F)

np.sqrt(np.dot(tuning_mat, Delta_F))

current_freqs

current_freqs + np.sqrt(np.dot(tuning_mat, Delta_F))


I am working on the file "20150723_inversion.ipynb".
The first task is:
On peut maintenant construire les deux matrices :
Can you write Python code for it?

mat1, rhs1 = make_matrix(T[1, :], T[0, :])
mat2, rhs2 = make_matrix(T[2, :], T[1, :])

mat2

rhs2


I am working on the file "20150723_inversion.ipynb".
The first task is:
On peut maintenant inverser la matrice en calculant l'accordage cible.
Can you write Python code for it?


I am working on the file "20150723_inversion.ipynb".
The first task is:
On peut maintenant remesurer les différentes fréquences sur la guitare :
Can you write Python code for it?

lines = """83,55	94,70	193,7	138,8	203	190
89,2	93,3	192,55	135,2	200,55	186,9
87,8	99,2	191,25	130,9	197,85	183,7""".replace(',', '.').split('\n')

history = np.array([line.split('\t') for line in lines], dtype=np.float)
history

T = (history ** 2) * psi

mat1, rhs1 = make_matrix(T[1, :], T[0, :])
mat2, rhs2 = make_matrix(T[2, :], T[1, :])
total_mat = np.vstack((mat1, mat2))
total_rhs = np.vstack((rhs1[:, np.newaxis],
                       rhs2[:, np.newaxis]))

rho, err, rank, eigs = np.linalg.lstsq(total_mat, total_rhs)

rho

err


I am working on the file "20150723_inversion.ipynb".
The first task is:
On peut maintenant écrire la séquence attendue :
Can you write Python code for it?

print("initial: {:.1f}, {:.1f}, {:.1f}, {:.1f}, {:.1f}, {:.1f}".format(*[x for x in current_freqs]))
for step in range(6):
    new_F = np.sqrt(current_freqs**2 + np.dot(tuning_mat, Delta_F * (np.arange(6) <= step)))
    print(" step {}: {:.1f}, {:.1f}, {:.1f}, {:.1f}, {:.1f}, {:.1f}".format(step, *[x for x in new_F]))


I am working on the file "20150723_inversion.ipynb".
The first task is:
On peut regarder la somme des tensions (et la remettre en kg pour avoir une petite idée) :
Can you write Python code for it?


I am working on the file "20150723_inversion.ipynb".
The first task is:
On peut remarquer que l'inversion ne donne pas des valeurs satisfaisantes car les signes du produit matrice vecteur ne respectent pas celles observées (que des moins ou que des plus). On peut imaginer faire plus de mesures pour voir ce que ça donne : je vais rajouter un désaccordage.
Can you write Python code for it?

history = np.vstack((history,
                     np.array([82.3,109.5,151.3,204.0,251.1,333.3])))

history

T = (history ** 2) * psi

mat3, rhs3 = make_matrix(T[3, :], T[2, :])

total_mat = np.vstack((mat1, mat2, mat3))
total_rhs = np.vstack((rhs1[:, np.newaxis],
                       rhs2[:, np.newaxis],
                       rhs3[:, np.newaxis]))
rho, err, rank, eigs = np.linalg.lstsq(total_mat, total_rhs)

rho

err

np.dot(total_mat, rho)

total_rhs


I am working on the file "20150723_inversion.ipynb".
The first task is:
On s'attend à ce qu'il n'y ait qu'une valeur qui a augmenté, ici c'est la première.
Can you write Python code for it?

np.nonzero(dT > 0)

tuned_string = np.nonzero(dT > 0)[0]
assert tuned_string.size == 1
tuned_string = tuned_string[0]

tuned_string

cnt = 0
for string in range(6):
        if string == tuned_string:
            continue
        else:
            for other_string in range(6):
                if other_string == tuned_string:
                    mat[cnt, other_string] = 0
                elif other_string == string:
                    mat[cnt, other_string] = dT[tuned_string] + dT[string]
                else:
                    mat[cnt, other_string] = dT[string]
            cnt += 1

mat[0]

mat[1]

mat


I am working on the file "20150723_inversion.ipynb".
The first task is:
On vérifie que ces termes sont les bons.
Can you write Python code for it?

psi[0] / psi[1] * (- rho[1] / (1 + np.sum([rho[k] for k in range(6) if k != 0])))


I am working on the file "20150723_inversion.ipynb".
The first task is:
On écrit la fonction pour calculer la matrice :
Can you write Python code for it?

def compute_tuning_matrix(psi, rho):
    tuning_mat = np.zeros((6, 6))

    for other_string in range(6):
        for tuning_string in range(6):
            if tuning_string == other_string:
                tuning_mat[other_string, tuning_string] = 1.
            else:
                tuning_mat[other_string, tuning_string] = \
                        psi[tuning_string] / psi[other_string] * \
                            (- rho[other_string] / (1 + np.sum([rho[i] for i in range(6) if i != tuning_string])))
    return tuning_mat

compute_tuning_matrix(psi, rho)


I am working on the file "20150723_inversion.ipynb".
The first task is:
Re :
Can you write Python code for it?

measured_freqs = [82.66,110.12,146.65,204.46,251.57,333.51]
expected_freqs = tuning_step(tuning_mat, initial_freqs, Delta_F, 3)
print_strings(measured_freqs)
print_strings(expected_freqs)


I am working on the file "20150723_inversion.ipynb".
The first task is:
Si :
Can you write Python code for it?

measured_freqs = [82.97,110.52,147.1,196.29,246.53,334.24]
expected_freqs = tuning_step(tuning_mat, initial_freqs, Delta_F, 5)
print_strings(measured_freqs)
print_strings(expected_freqs)


I am working on the file "20150723_inversion.ipynb".
The first task is:
Sol :
Can you write Python code for it?

measured_freqs = [82.86,110.39,146.96,195.81,251.94,333.99]
expected_freqs = tuning_step(tuning_mat, initial_freqs, Delta_F, 4)
print_strings(measured_freqs)
print_strings(expected_freqs)


I am working on the file "20150723_inversion.ipynb".
The first task is:
Tout est de même signe, c'est pas si mal. Et l'erreur est faible.
Accordage
Can you write Python code for it?

tuning_mat = compute_tuning_matrix(psi, rho)

target_freqs = freqs.copy()
current_freqs = history[2, :]
target_dF = target_freqs**2 - current_freqs**2
Delta_F = np.linalg.solve(tuning_mat, target_dF)
Delta_F

initial_freqs = current_freqs.copy()
initial_freqs

for target, string in zip([tuning_step(tuning_mat, initial_freqs, Delta_F, i)[i-1] for i in range(1, 7)],
                          ["low E", "A", "D", "G", "B", "high E"]):
    print("string: {}, target frequency: {:.1f}".format(string, target))


I am working on the file "20150723_inversion.ipynb".
The first task is:
attention au calcul des target_dF, c'est :
$$
\delta F = f_{target}^2 - f_{current}^2
$$
Can you write Python code for it?

target_freqs = freqs.copy()
current_freqs = history[2, :]
target_dF = target_freqs**2 - current_freqs**2
Delta_F = np.linalg.solve(tuning_mat, target_dF)
Delta_F

np.sqrt(current_freqs**2 + np.dot(tuning_mat, Delta_F))

target_freqs


I am working on the file "20150723_inversion.ipynb".
The first task is:
attention les tensions sont bien en Newton, pas en kg comme dans la feuille Numbers.
Construction de la matrice à inverser pour obtenir les raideurs du modèle
On peut maintenant construire une première matrice de 5 lignes fois 6 colonnes.
Can you write Python code for it?

T_end = T[1, :]
T_start = T[0, :]

mat = np.zeros((5, 6))
dT = T_end - T_start

dT

mat


I am working on the file "20150723_inversion.ipynb".
The first task is:
note importante : j'utilise ici des $mu_i$ qui sont lus sur une pochette de cordes. Je n'ai aucune garantie que cela soient bien les vrais $mu_i$ nécessaires aux cordes qui sont sur ma guitare !
Can you write Python code for it?

freqs = np.array([82.4, 110., 146.8, 196., 246.9, 329.6]) # frequencies of the guitar strings, from low E to high E, in Hz
calibration_tensions = np.array([9.59, 11.61, 11.22, 8.43, 8.09, 8.9]) * 9.81 # calibration tensions found on package (in kg) converted to N
mu = calibration_tensions / (4 * 0.648**2 * freqs**2)
mu

psi = 4 * 0.648**2 * mu

psi

T = (df ** 2).values * psi
T


I am working on the file "20150730-notebook-intro/Intro notebook.ipynb".
The first task is:
...and Julia...
Can you write Python code for it?

%load_ext julia.magic

%%julia
using DataFrames
df = DataFrame(x1=[1, 2], x2=["foo", "bar"])


I am working on the file "20150730-notebook-intro/Intro notebook.ipynb".
The first task is:
And much, much more:
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo("aIXED26Wppg")


I am working on the file "20150730-notebook-intro/Intro notebook.ipynb".
The first task is:
Kernels exist for many languages.  This is extremely useful.
Why is it important?

Mixing code and narrative is an effective and proven communication technique for many contexts (see also literate programming)
It is a good working environment for exploring and explain complex ideas requiring a mix of code, data, math, media, and text, easier and faster than raw HTML
Its openness creates many possibilities
It is a terrific teaching tool
It plays okay with version control, renders inline in Github
We can use it in our daily work (I have started to)

Getting started
The notebook structure is similar to that offered by other statistical and mathematical packages like Stata or Mathematica.  It evolved from the IPython project, and is now known as Jupyter.  It is a typical part of scientific work to record actions in some kind of lab notebook, and this modality flows naturally from that mindset - record what you do, share with others.
With that in mind, what do you need to get started?
Can you write Python code for it?


I am working on the file "20150730-notebook-intro/Intro notebook.ipynb".
The first task is:
There is a whole range of widget types:
Can you write Python code for it?

from IPython.html import widgets
[n for n in dir(widgets) if not n.endswith('Widget') and n[0] == n[0].upper() and not n[0] == '_']


I am working on the file "20150730-notebook-intro/Intro notebook.ipynb".
The first task is:
Working with other languages
If you look closely at the spelling of Jupyter, you might notice that it implies more than just Python.  Like, for instance, R:
Can you write Python code for it?

%load_ext rmagic

%%R
r <- rnorm(100)
plot(r)


I am working on the file "20150730-notebook-intro/Intro notebook.ipynb".
The first task is:
You can do most anything here that you could do in the Python REPL, indeed this is basically a web front-end to the Python REPL, or more precisely, to the IPython REPL.  IPython is an enhanced wrapper around the standard Python REPL that's been around (and I've been using) for over ten years.
Note that the code cell above was numbered when it executed.  Adding another cell and executing it updates the number, leaving a train of execution order.
Can you write Python code for it?


I am working on the file "20150916_OGC_Reuse_under_licence/.ipynb_checkpoints/20150916_OGC_Reuse_under_licence-checkpoint_conflict-20150909-085502.ipynb".
The first task is:
Background
You need to install the RISE Ipython Library from Damián Avila for dynamic presentations
To convert and run this as a static presentation run the following command:
Can you write Python code for it?

!ipython nbconvert Template.ipynb --to slides --post serve


I am working on the file "20150916_OGC_Reuse_under_licence/.ipynb_checkpoints/20150916_OGC_Reuse_under_licence-checkpoint_conflict-20150909-085502.ipynb".
The first task is:
Key activities

Inline things
Exporting
as html5
static - and locally served: - !ipython nbconvert Template.ipynb --to slides --post serve
SPEAKER NOTES work in the above

static - !ipython nbconvert Template.ipynb --to slides

as pdf

hosting:
on slideshare
I believe I should use the pdf
what about slideviewer[http://slideviewer.herokuapp.com/]

on github:
just create a git project with these presentations in them :-)

PDF output - hack from Damian

cd in the directory where your slideshow lives
add this custom.css file: https://gist.github.com/damianavila/6211198
run this little snippet: https://gist.github.com/damianavila/6211211
run python -m SimpleHTTPServer 8001
open Mozilla Firefox browser and point to localhost:8001
add ?print.pdf to the end of the url (ie, http://127.0.0.1:8001/your-ipynb.slides.html?print-pdf)
print to pdf (use Landscape orientation)

The environment
In order to replicate my environment you need to know what I have installed!
Set up watermark
This describes the versions of software used during the creation.
Please note that critical libraries can also be watermarked as follows:
python
%watermark -v -m -p numpy,scipy
Can you write Python code for it?

%install_ext https://raw.githubusercontent.com/rasbt/python_reference/master/ipython_magic/watermark.py
%load_ext watermark

%watermark -a "Anthony Beck"  -d -v -m -g

#List of installed conda packages
!conda list

#List of installed pip packages
!pip list


I am working on the file "20150916_OGC_Reuse_under_licence/.ipynb_checkpoints/20150916_OGC_Reuse_under_licence-checkpoint_conflict-20150909-085502.ipynb".
The first task is:
To close this instances press control 'c' in the ipython notebook terminal console
Static presentations allow the presenter to see speakers notes (use the 's' key)
If running dynamically run the scripts below
Pre load some useful libraries
Can you write Python code for it?

#Future proof python 2
from __future__ import print_function #For python3 print syntax
from __future__ import division

# def
import IPython.core.display

# A function to collect user input - ipynb_input(varname='username', prompt='What is your username')

def ipynb_input(varname, prompt=''):
        """Prompt user for input and assign string val to given variable name."""
        js_code = ("""
            var value = prompt("{prompt}","");
            var py_code = "{varname} = '" + value + "'";
            IPython.notebook.kernel.execute(py_code);
        """).format(prompt=prompt, varname=varname)
        return IPython.core.display.Javascript(js_code)

# inline

%pylab inline


I am working on the file "20150916_OGC_Reuse_under_licence/.ipynb_checkpoints/20150916_OGC_Reuse_under_licence-checkpoint_conflict-20150910-195436.ipynb".
The first task is:
Running dynamic presentations
You need to install the RISE Ipython Library from Damián Avila for dynamic presentations
To convert and run this as a static presentation run the following command:
Can you write Python code for it?

!ipython nbconvert 20150916_OGC_Reuse_under_licence.ipynb --to slides --post serve


I am working on the file "20150916_OGC_Reuse_under_licence/.ipynb_checkpoints/20150916_OGC_Reuse_under_licence-checkpoint_conflict-20150910-195436.ipynb".
The first task is:
To close this instances press control 'c' in the ipython notebook terminal console
Static presentations allow the presenter to see speakers notes (use the 's' key)
If running dynamically run the scripts below
Pre load some useful libraries
Can you write Python code for it?


I am working on the file "20150916_OGC_Reuse_under_licence/.ipynb_checkpoints/20150916_OGC_Reuse_under_licence-checkpoint_conflict-20150910-195436.ipynb".
The first task is:
\newpage
Here's something I prepared earlier
A live presentation.....
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo('tkRB5Rp1_W4')


I am working on the file "20150916_OGC_Reuse_under_licence/.ipynb_checkpoints/20150916_OGC_Reuse_under_licence-checkpoint_conflict-20150915-091746.ipynb".
The first task is:
Running dynamic presentations
You need to install the RISE Ipython Library from Damián Avila for dynamic presentations
To convert and run this as a static presentation run the following command:
Can you write Python code for it?


I am working on the file "20150916_OGC_Reuse_under_licence/.ipynb_checkpoints/20150916_OGC_Reuse_under_licence-checkpoint_conflict-20150915-091746.ipynb".
The first task is:
The environment
In order to replicate my environment you need to know what I have installed!
Set up watermark
This describes the versions of software used during the creation.
Please note that critical libraries can also be watermarked as follows:
python
%watermark -v -m -p numpy,scipy
Can you write Python code for it?


I am working on the file "20150916_OGC_Reuse_under_licence/.ipynb_checkpoints/20150916_OGC_Reuse_under_licence-checkpoint_conflict-20150915-091746.ipynb".
The first task is:
\newpage
Here's something I prepared earlier
A live presentation (for those who weren't at the event).....
Can you write Python code for it?


I am working on the file "20150916_OGC_Reuse_under_licence/.ipynb_checkpoints/20150916_OGC_Reuse_under_licence-checkpoint_conflict-20150915-181258.ipynb".
The first task is:
The environment
In order to replicate my environment you need to know what I have installed!
Set up watermark
This describes the versions of software used during the creation.
Please note that critical libraries can also be watermarked as follows:
python
%watermark -v -m -p numpy,scipy
Can you write Python code for it?


I am working on the file "20150916_OGC_Reuse_under_licence/.ipynb_checkpoints/20150916_OGC_Reuse_under_licence-checkpoint_conflict-20150915-181258.ipynb".
The first task is:
To close this instances press control 'c' in the ipython notebook terminal console
Static presentations allow the presenter to see speakers notes (use the 's' key)
If running dynamically run the scripts below
Pre load some useful libraries
Can you write Python code for it?


I am working on the file "20150916_OGC_Reuse_under_licence/.ipynb_checkpoints/20150916_OGC_Reuse_under_licence-checkpoint_conflict-20150915-181258.ipynb".
The first task is:
\newpage
About me

Research Fellow, University of Nottingham: orcid
Director, Geolytics Limited - A spatial data analytics consultancy

About this presentation

Available on GitHub - https://github.com/AntArch/Presentations_Github/
Fully referenced PDF

\newpage
A potted history of mapping
In the beginning was the geoword
and the word was cartography

\newpage

Cartography was king. Static representations of spatial knowledge with the cartographer deciding what to represent.
\newpage
And then there was data .........

\newpage

Restrictive data
\newpage

Making data interoperable and open
\newpage
Technical interoperability - levelling the field

\newpage
Facilitating data driven visualization

From Map to Model The changing paradigm of map creation from cartography to data driven visualization
\newpage

\newpage

\newpage

\newpage
What about non-technical interoperability issues?

Issues surrounding non-technical interoperability include:

Policy interoperabilty
Licence interoperability
Legal interoperability
Social interoperability

We will focus on licence interoperability
\newpage

There is a multitude of formal and informal data.
\newpage

Each of these data objects can be licenced in a different way. This shows some of the licences described by the RDFLicence ontology
\newpage
What is a licence?
Wikipedia state:

A license may be granted by a party ("licensor") to another party ("licensee") as an element of an agreement between those parties.
A shorthand definition of a license is "an authorization (by the licensor) to use the licensed material (by the licensee)."

Concepts (derived from Formal Concept Analysis) surrounding licences
\newpage
Two lead organisations have developed legal frameworks for content licensing:

Creative Commons (CC) and
Open Data Commons (ODC).

Until the release of CC version 4, published in November 2013, the CC licence did not cover data. Between them, CC and ODC licences can cover all forms of digital work.

There are many other licence types
Many are bespoke
Bespoke...
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo('jUzGF401vLc')


I am working on the file "20150916_OGC_Reuse_under_licence/.ipynb_checkpoints/20150916_OGC_Reuse_under_licence-checkpoint_conflict-20150915-181258.ipynb".
The first task is:
\newpage
Here's something I prepared earlier
A live presentation (for those who weren't at the event).....
Can you write Python code for it?


I am working on the file "20150916_OGC_Reuse_under_licence/20150916_OGC_Reuse_under_licence.ipynb".
The first task is:
To close this instances press control 'c' in the ipython notebook terminal console
Static presentations allow the presenter to see speakers notes (use the 's' key)
If running dynamically run the scripts below
Pre load some useful libraries
Can you write Python code for it?


I am working on the file "20150916_OGC_Reuse_under_licence/20150916_OGC_Reuse_under_licence.ipynb".
The first task is:
\newpage
About me

Research Fellow, University of Nottingham: orcid
Director, Geolytics Limited - A spatial data analytics consultancy

About this presentation

Available on GitHub - https://github.com/AntArch/Presentations_Github/
Fully referenced PDF

\newpage
A potted history of mapping
In the beginning was the geoword
and the word was cartography

\newpage

Cartography was king. Static representations of spatial knowledge with the cartographer deciding what to represent.
\newpage
And then there was data .........

\newpage

Restrictive data
\newpage

Making data interoperable and open
\newpage
Technical interoperability - levelling the field

\newpage
Facilitating data driven visualization

From Map to Model The changing paradigm of map creation from cartography to data driven visualization
\newpage

\newpage

\newpage

\newpage
What about non-technical interoperability issues?

Issues surrounding non-technical interoperability include:

Policy interoperabilty
Licence interoperability
Legal interoperability
Social interoperability

We will focus on licence interoperability
\newpage

There is a multitude of formal and informal data.
\newpage

Each of these data objects can be licenced in a different way. This shows some of the licences described by the RDFLicence ontology
\newpage
What is a licence?
Wikipedia state:

A license may be granted by a party ("licensor") to another party ("licensee") as an element of an agreement between those parties.
A shorthand definition of a license is "an authorization (by the licensor) to use the licensed material (by the licensee)."

Concepts (derived from Formal Concept Analysis) surrounding licences
\newpage
Two lead organisations have developed legal frameworks for content licensing:

Creative Commons (CC) and
Open Data Commons (ODC).

Until the release of CC version 4, published in November 2013, the CC licence did not cover data. Between them, CC and ODC licences can cover all forms of digital work.

There are many other licence types
Many are bespoke
Bespoke...
Can you write Python code for it?


I am working on the file "201509ETL-master/201509ETL-master/20150911 ETL.ipynb".
The first task is:
http://release.seleniumhq.org/selenium-ide/2.9.0/selenium-ide-2.9.0.xpi
Can you write Python code for it?

# -*- coding: utf-8 -*-
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import NoAlertPresentException
import  time, re

driver = webdriver.Firefox()
driver.implicitly_wait(3)
base_url = "http://www.agoda.com"

driver.get(base_url + "/zh-tw/city/taipei-tw.html")
driver.find_element_by_id("CheckInMonthYear").click()

driver.implicitly_wait(1)
Select(driver.find_element_by_id("CheckInMonthYear")).select_by_visible_text(u"2015年11月")
driver.implicitly_wait(1)
driver.find_element_by_id("search-submit").click()
driver.implicitly_wait(1)
driver.implicitly_wait(3)
driver.find_element_by_link_text(u"下一頁").click()