I am working on the file "Jupyter/MimicII/0a Fill Database WaveForm Headers.ipynb".
The first task is:
Le agregamos el subject_id y la fecha del record a los campos
Can you write Python code for it?

fields['subject_id'] = subject_id
fields['recordDate'] = recordDate
fields['database'] = "mimic2"


I am working on the file "Jupyter/MimicII/0a Fill Database WaveForm Headers.ipynb".
The first task is:
cerramos conexion
Can you write Python code for it?


I am working on the file "Jupyter/MimicII/0a Fill Database WaveForm Headers.ipynb".
The first task is:
convertimos los campos en un diccionario
Can you write Python code for it?

columns = fields.keys()
values = [fields[column] for column in columns]
print(columns)


I am working on the file "Jupyter/MimicIII/01 Fill Database WaveForm Headers.ipynb".
The first task is:
2) Leemos el archivo con las WaveForm que vamos a utilizar
Can you write Python code for it?

target_url = "https://physionet.org/physiobank/database/mimic3wdb/matched/RECORDS-waveforms"
data = urllib.request.urlopen(target_url) # it's a file like object and works just like a file
lines = data.readlines();
line = str(lines[2])
line


I am working on the file "Jupyter/MimicIII/01 Fill Database WaveForm Headers.ipynb".
The first task is:
3) Limpiamos los caracteres extraños y Dividimos la cadena donde pXXNNNN-YYYY-MM-DD-hh-mm donde XXNNNN es el identificador unico del paciente SUBJECT_ID y YYYY-MM-DD-hh-mm es la fecha de la estadia del paciente'
Can you write Python code for it?

line = line.replace('b\'','').replace('\'','').replace('\\n','')
splited = line.split("/")
carpeta,subCarpeta,onda = line.split("/")
carpeta = carpeta+"/"+subCarpeta
subject_id = subCarpeta.replace('p','')
recordDate = onda.replace(subCarpeta+"-","")
print("subject_id: ",subject_id)
print("recordDate: ",recordDate)
print("onda: ",onda)
print("carpeta: ",carpeta)


I am working on the file "Jupyter/MimicIII/01 Fill Database WaveForm Headers.ipynb".
The first task is:
4) Leemos el encabezado del waveform, para obtener la información del paciente que almacenaremos
Can you write Python code for it?

try:
    sig, fields = wfdb.srdsamp(onda,pbdir='mimic3wdb/matched/'+carpeta, sampto=1)
    print(fields)
except Exception as inst:
    print("onda vacia")


I am working on the file "Jupyter/MimicIII/01 Fill Database WaveForm Headers.ipynb".
The first task is:
Creamos la tabla donde quedaran almacenados los datos
Can you write Python code for it?

table = "waveformFields"
#cur.execute("DROP TABLE "+table)
cur.execute("CREATE TABLE IF NOT EXISTS "+table+
            " (id serial PRIMARY KEY,"+
            "comments character varying(255)[],"+
            "fs integer, signame character varying(255)[],"+
            "units character varying(255)[],"+
 "subject_id integer,"+
            "recordDate character varying(255));")


I am working on the file "Jupyter/MimicIII/01 Fill Database WaveForm Headers.ipynb".
The first task is:
Hacemos commit
Can you write Python code for it?


I am working on the file "Jupyter/MimicIII/01 Fill Database WaveForm Headers.ipynb".
The first task is:
Insertamos los datos
Can you write Python code for it?

insert_statement = 'insert into '+table+' (%s) values %s'

print(cur.mogrify(insert_statement, (AsIs(','.join(columns)), tuple(values))))
if notExist:
    cur.execute(insert_statement, (AsIs(','.join(columns)), tuple(values)))


I am working on the file "Jupyter/MimicIII/01 Fill Database WaveForm Headers.ipynb".
The first task is:
Le agregamos el subject_id y la fecha del record a los campos
Can you write Python code for it?

fields['subject_id'] = subject_id
fields['recordDate'] = recordDate


I am working on the file "Jupyter/MimicIII/01 Fill Database WaveForm Headers.ipynb".
The first task is:
Nos conectamos a la base de datos postgres donde almacenaremos los datos
Can you write Python code for it?

conn = psycopg2.connect("dbname=mimic")
cur = conn.cursor()


I am working on the file "Jupyter/MimicIII/01 Fill Database WaveForm Headers.ipynb".
The first task is:
convertimos los campos en un diccionario
Can you write Python code for it?


I am working on the file "Jupyter/Non-Negative Matrix Factorization (NMF).ipynb".
The first task is:
A test Array
Can you write Python code for it?

Original = [
    [5,3,0,1],
    [4,0,0,1],
    [3,2,0,0],
    [7,0,4,1],
    [0,2,5,0]
]
Original = np.array(Original)

W = model.fit_transform(Original)
H = model.components_

print("W\n",np.round(W,2))
print("H\n",np.round(H,2))


I am working on the file "Jupyter/Non-Negative Matrix Factorization (NMF).ipynb".
The first task is:
Now lets see how mush close to X are WxH and lets call it crossValue
Can you write Python code for it?

crossValue = np.dot(W,H)
print("crossValue \n",crossValue)
print("rounded Values\n",np.round(crossValue,2))
print("Original\n",Original)

import matplotlib.pyplot as plt
def plotCompare(Original,prediction):
    N = Original.shape[0]
    last = Original.shape[1]-1
    ind = np.arange(N)  # the x locations for the groups
    width = 0.17       # the width of the bars
    fig, ax = plt.subplots()
    rects1 = ax.bar(ind, Original[:,last], width, color='r')
    rects2 = ax.bar(ind + width, prediction[:,last], width, color='b')
#    rects3 = ax.bar(ind + width+width, np.round(prediction[:,last],2), width, color='g')

    # add some text for labels, title and axes ticks
    ax.set_ylabel('Last Value')
    ax.set_title('Row Values')
    ax.set_xticks(ind+ width / last)
    ax.set_xticklabels(('G1', 'G2', 'G3', 'G4','G5','G6'))

    ax.legend((rects1[0], rects2[0]), ('Original', 'Cross Value'))

    plt.show()

plotCompare(Original,crossValue)


I am working on the file "Jupyter/Non-Negative Matrix Factorization (NMF).ipynb".
The first task is:
Using an independent Implementation from
http://www.quuxlabs.com/blog/2010/09/matrix-factorization-a-simple-tutorial-and-implementation-in-python/
Modified by DavidGutierrez

R : A matrix to be factorized, dimension N x M

K : The number of latent features

Steps : The maximum number of steps to perform the optimisation

Alpha : The learning rate

Beta : The regularization parameter

The final matrices W and H
Can you write Python code for it?

def matrix_factorization(R, K = 2, steps=5000, alpha=0.0002, beta=0.02,error = 0.001):
    W = np.random.rand(len(R),K)
    H = np.random.rand(K,len(R[0]))
    for step in range(steps):
        for i in range(len(R)):
            for j in range(len(R[i])):
                if R[i][j] > 0:
                    eij = R[i][j] - np.dot(W[i,:],H[:,j])
                    for k in range(K):
                        W[i][k] = W[i][k] + alpha * (2 * eij * H[k][j] - beta * W[i][k])
                        H[k][j] = H[k][j] + alpha * (2 * eij * W[i][k] - beta * H[k][j])
#        eR = np.dot(W,H)
        e = 0
        for i in range(len(R)):
            for j in range(len(R[i])):
                if R[i][j] > 0:
                    e = e + pow(R[i][j] - np.dot(W[i,:],H[:,j]), 2)
                    for k in range(K):
                        e = e + (beta/2) * ( pow(W[i][k],2) + pow(H[k][j],2) )
        if e < error:
            break
    return W,H

W, H = matrix_factorization(Original,K)

W

H

prediction = np.dot(W,H)
print(prediction)

np.around(prediction,2)

Original

plotCompare(Original,prediction)


I am working on the file "Jupyter/Workflow.ipynb".
The first task is:
N.B. You can ignore any warning about unverified HTTP requests. Response code 204 means that the REST call succeed.
Step 2: BatchfeatureRemoval - Removal of batch specific features
In MS studies you may, if you have many samples, prepare the samples in batches. By doing so you introduce a risk of having features that are unique within a batch, which is not desirable.
In this step we remove features that have a coverage of 80% within one batch, but not in any other. In other words, here we remove batch specific features.
The input data of this step comes form the blank filter, hence in the JSON parameters we will set the previous step as parent. In this way Chronos will make sure to run the microservices in the correct order. Please go through the following code snippet, run it and check the Chronos interface.
Can you write Python code for it?

url="https://admin:"+password+"@"+control+"/chronos/scheduler/dependency"
json="""
{
    "parents" : ["blank-filter"],
    "cpus": "0.25",
    "mem": "128",
    "epsilon" : "PT10M",
    "name" : "batchfeature-removal",
    "container": {
        "type": "DOCKER",
        "image": "farmbio/batchfeatureremoval",
        "volumes": [{
            "hostPath": "/mnt/container-volumes/jupyter",
            "containerPath": "/data",
            "mode": "RW"
         }]
    },
    "command" : "Rscript BatchfeatureRemoval.r /data/output_BlankFilter.xls /data/output_BatchfeatureRemoval.xls",
    "owner" : "user@example.com"
}
"""
response=requests.post(url, headers = {'content-type' : 'application/json'}, data=json, verify=False)
print("HTTP response code: " + str(response.status_code))


I am working on the file "Jupyter/Workflow.ipynb".
The first task is:
Please run the following code snippet and insert your MANTL admin password
Can you write Python code for it?


I am working on the file "Jupyter/Workflow.ipynb".
The first task is:
Please run the following code snippet to get some input data for the workflow
Can you write Python code for it?

import urllib.request
urllib.request.urlretrieve(
    "https://raw.githubusercontent.com/phnmnl/workflow-demo/master/data/inputdata_workshop.xls", # mdownload URL
    "inputdata_workshop.xls" # local path
)


I am working on the file "Jupyter/Workflow.ipynb".
The first task is:
Step 1: Blank filter - Contaminants Removal
When perfoming any mass spectrometry (MS) study, it happens that you get plastic or other contaminent within your samples. When you furher analyze your samples with MS, these contaminant will be recorded together with the metabolites. To be able to detect and filter these out you often add blank samples (samples with only DMSO in them)  in between the normal samples in the runorder.
In this step we aim to remove the contaminants detected in the blanks, from the rest of our samples. The theory behind it is to remove everything that has an intensity of X in the blanks compared to the samples. For example, everything that, in the blanks, has an intensity of 1% or higher of the intensities in the other samples.
The input data in the prerequisites will be used here as input. Hence, we need to mount the Jupyther working directory (/mnt/container-volumes/jupyter) as a volume in the docker container. Please go through the following code snippet, and use the Chronos REST API documentation, to figure out the meaning of the JSON data that is sent with the HTTP request. Once you are done with that please run the snippet and check the Chronos interface (which you can access throug the MANTL UI).
Can you write Python code for it?

url="https://admin:"+password+"@"+control+"/chronos/scheduler/iso8601"
json="""
{
    "schedule" : "R0/2030-01-01T12:00:00Z/PT1H",
    "cpus": "0.25",
    "mem": "128",
    "epsilon" : "PT10M",
    "name" : "blank-filter",
    "container": {
        "type": "DOCKER",
        "image": "farmbio/blankfilter",
        "volumes": [{
            "hostPath": "/mnt/container-volumes/jupyter",
            "containerPath": "/data",
            "mode": "RW"
         }]
    },
    "command" : "Rscript BlankFilter.r /data/inputdata_workshop.xls /data/output_BlankFilter.xls",
    "owner" : "user@example.com"
}
"""
response=requests.post(url, headers = {'content-type' : 'application/json'}, data=json, verify=False)
print("HTTP response code: " + str(response.status_code))


I am working on the file "Jupyter/Workflow.ipynb".
The first task is:
Step 4: Splitter - Splits data according to sample names
In order to parallelize microservices, we need to split the data into the subsets that will be processed simultaneously. In this step we divide the data basing on the first five letters of the sample names.
Please go through the following code snippet, run it and check the Chronos interface.
Can you write Python code for it?

url="https://admin:"+password+"@"+control+"/chronos/scheduler/dependency"
json="""
{
    "parents" : ["log2-transformation"],
    "cpus": "0.25",
    "mem": "128",
    "epsilon" : "PT10M",
    "name" : "splitter",
    "container": {
        "type": "DOCKER",
        "image": "farmbio/splitter",
        "volumes": [{
            "hostPath": "/mnt/container-volumes/jupyter",
            "containerPath": "/data",
            "mode": "RW"
         }]
    },
    "command" : "Rscript Splitter.r /data/output_log2transformation.xls /data/output_splitter",
    "owner" : "user@example.com"
}
"""
response=requests.post(url, headers = {'content-type' : 'application/json'}, data=json, verify=False)
print("HTTP response code: " + str(response.status_code))


I am working on the file "Jupyter/Workflow.ipynb".
The first task is:
Step 5: CV - Calculate the coefficient of variation
In this step we calculate the coefficient of variation for each feature present within the samples.
In the previous step we divided the working set by sample. It is very convenient now to run multiple instances of the CV microservice in parallel, in order to save running time. In the code snippet we use the header of the inputdata_workshop.xls file to figure out the file names that will come out of the previous step. Then for each file we submit a new job to Chronos.
Please go through the following code snippet, run it and check the Chronos interface.
Can you write Python code for it?

import os
os.mkdir("output_cv") # Create a folder for CV output
# Figure out samples from the header of the input file
with open('inputdata_workshop.xls', 'r') as f: header = f.readline()
samples = list(set(map(lambda s: s[1:-1][:5],header.split("\t"))))
samples.remove('BLANK') # BLANK doesn't need to be processed
# Create a microservice for each sample
url="https://admin:"+password+"@"+control+"/chronos/scheduler/dependency"
for s in samples:
    json="""
    {
        "parents" : ["splitter"],
        "cpus": "0.25",
        "mem": "128",
        "epsilon" : "PT10M",
        "name" : "cv-%(jobname)s",
        "container": {
            "type": "DOCKER",
            "image": "farmbio/cv",
            "volumes": [{
                "hostPath": "/mnt/container-volumes/jupyter",
                "containerPath": "/data",
                "mode": "RW"
             }]
        },
        "command" : "Rscript CV.r /data/output_splitter/%(sample)s.xls /data/output_cv/%(sample)s_cv.xls",
        "owner" : "user@example.com"
    }
    """ % {"sample" : s, "jobname": s.replace(".", "_")}
    response=requests.post(url, headers = {'content-type' : 'application/json'}, data=json, verify=False)
    print(s + " HTTP response code: " + str(response.status_code))


I am working on the file "Jupyter/Workflow.ipynb".
The first task is:
Step 6: Merger - Merges several files into one, columnwise
In step 5 we processed many samples in parallel, and before proceding to the feature selection we need to merge them again in a single file. This step can be run only after all of the jobs that have been generated by the previous step are finished. Hence, in the JSON we will specify all the jobs from step 5 in the parent field.
Please go through the following code snippet, run it and check the Chronos interface.
Can you write Python code for it?

url="https://admin:"+password+"@"+control+"/chronos/scheduler/dependency"
# Format job names from step 5 in JSON array format
jobNames = "[" + ",".join(map(lambda s: '"cv-'+s.replace(".", "_")+'"',samples)) + "]"
json="""
{
    "parents" : %(parents)s,
    "cpus": "0.25",
    "mem": "128",
    "epsilon" : "PT10M",
    "name" : "merger",
    "container": {
        "type": "DOCKER",
        "image": "farmbio/merger",
        "volumes": [{
            "hostPath": "/mnt/container-volumes/jupyter",
            "containerPath": "/data",
            "mode": "RW"
         }]
    },
    "command" : "Rscript Merger.r /data/output_cv /data/output_Merger.xls",
    "owner" : "user@example.com"
}
""" % {"parents" : jobNames}
#print("HTTP response code: " + json)
response=requests.post(url, headers = {'content-type' : 'application/json'}, data=json, verify=False)
print("HTTP response code: " + str(response.status_code))


I am working on the file "Jupyter/Workflow.ipynb".
The first task is:
Step 7: FeatureSelection - Extract features with low CV
In this final step we extract stable features, basing on the median coefficient of variation (step 5).
Please go through the following code snippet, run it and check the Chronos interface.
Can you write Python code for it?

url="https://admin:"+password+"@"+control+"/chronos/scheduler/dependency"
json="""
{
    "parents" : ["merger"],
    "cpus": "0.25",
    "mem": "128",
    "epsilon" : "PT10M",
    "name" : "feature-selection",
    "container": {
        "type": "DOCKER",
        "image": "farmbio/featureselection",
        "volumes": [{
            "hostPath": "/mnt/container-volumes/jupyter",
            "containerPath": "/data",
            "mode": "RW"
         }]
    },
    "command" : "Rscript FeatureSelection.r /data/output_log2transformation.xls /data/output_Merger.xls /data/output_FeatureSelection.xls",
    "owner" : "user@example.com"
}
"""
response=requests.post(url, headers = {'content-type' : 'application/json'}, data=json, verify=False)
print("HTTP response code: " + str(response.status_code))


I am working on the file "Jupyter/languages_and_corpus.ipynb".
The first task is:
Esperanto
Can you write Python code for it?

import nltk

print( nltk.corpus.udhr.raw('Esperanto-UTF8'))


I am working on the file "Jupyter/languages_and_corpus.ipynb".
The first task is:
For moving towards a better application of Category Theory onto the Esperanto language, we used a pre-tagged corpus for this purpose.
Here is a sample of the pre-tagged data.
We relied upon the regular structure of the language to form the requisite Regular-Expressions for this purpose.
Can you write Python code for it?

text = '''
Longe/RB vivadis/VBD en/IN paco/NNS tiu/DT gento/NNS trankvila,/JJ de/IN Kristanismo/NNS netusita/JJ gis/IN
dek-tria/dek-tria jarcento./NNS De/IN la/DT cetera/JJ mondo/NNS forkasita/JJ per/IN marcoj/NNP kaj/CC densaj/JJ
arbaregoj,/NNP kie/RB kuras/VBP gis/IN nun/VB sovagaj/JJ urbovoj,/NNP la/DT popolo/NNS dauris/VBD adori/ii la/DT
fortojn/NNP de/IN la/DT naturo/NNS sub/IN gigantaj/JJ kverkoj,/NNP vivanta/JJ templo/NNS de/IN la/DT dioj./NNP

Tie/RB tamen/RB ekbatalis/VBD okcidenta/JJ volo/NNS kun/IN orienta/JJ pacienco./NNS En/IN la/DT mezepoko/NNS
teutonaj/JJ kavaliroj/NNP tiun/DT landon/NNS almilitis,/VBD polaj/JJ nobeloj/NNP gin/PRP ligis/VBD al/IN sia/PRP$
stato,/NNS moskova/JJ caro/NNS gin/PRP atakis./VBD Dume/RB alkuradis/VBD el/IN tuta/JJ mondo/NNS
persekutataj/JJ Hebreoj/NNP por/IN starigi/ii manlaboron/NNS kaj/CC komercon/NNS lau/IN invito/NNS
rega./JJ Tiel/RB alia/JJ gento/NNS tre/RB maljuna/JJ trovis/VBD tie/RB novan/JJ Palestinon/NNS kaj/CC fondis/VBD
urbojn/NNP au/CC plenigis/VBD ilin./PRP'''

[nltk.tag.str2tuple(t) for t in text.split()]


I am working on the file "Jupyter/languages_and_corpus.ipynb".
The first task is:
Here we plot the list of POS per language.
Can you write Python code for it?

import matplotlib.pyplot as plt
#plt.rcdefaults()
import numpy as np

plt.style.use('bmh')

# Example data
#people = set(words_types)

x_pos = np.arange(len(list_of_languages))

plt.bar(x_pos, num_pos,  color='r', align='center', alpha=0.4)

plt.xticks(x_pos,lang_names  ,  rotation=45 )

#plt.xlabel('Performance')
#plt.title('How fast do you want to go today?')

plt.show()


I am working on the file "Jupyter/languages_and_corpus.ipynb".
The first task is:
Here, Give a brief overview of Esperanto
Similarly we chose the UDHR in all the languages ( 22 in number ) shown in the following world map.
Can you write Python code for it?

import os
os.getcwd()

from IPython.display import Image
from IPython.core.display import HTML

PATH = "C:\\Users\\user\\Desktop\\Language Space and Mind"
Image(filename = PATH + "\\canvas.png", width=1000, height=500)

list_of_languages = [

['English', ['English-Latin1'] ]
,['Esperanto', ['Esperanto-UTF8']],
['German', ['German_Deutsch-Latin1']]
,['French', ['French_Francais-Latin1']]
,['Russian', ['Russian-UTF8','Russian_Russky-Cyrillic','Russian_Russky-UTF8']]
,['Farsi', ['Farsi_Persian-UTF8', 'Farsi_Persian-v2-UTF8']]
,['Finnish',['Finnish_Suomi-Latin1']]
,['Hungarian',['Hungarian_Magyar-Latin1','Hungarian_Magyar-Latin2','Hungarian_Magyar-UTF8']]
,['Turkish', ['Turkish_Turkce-Turkish','Turkish_Turkce-UTF8']]
,['Mongolian', ['Mongolian_Khalkha-Cyrillic','Mongolian_Khalkha-UTF8']]
,['Chinese',['Chinese_Mandarin-GB2312']]
,['Japanese',['Japanese_Nihongo-EUC','Japanese_Nihongo-SJIS','Japanese_Nihongo-UTF8']]
,['Korean',['Korean_Hankuko-UTF8']]
,['Hebrew',['Hebrew_Ivrit-Hebrew','Hebrew_Ivrit-UTF8']]
,['Hindi',['Hindi-UTF8','Hindi_web-UTF8']]
,['Kazakh',['Kazakh-Cyrillic','Kazakh-UTF8']]
,['Swedish',['Swedish_Svenska-Latin1']]
,['Icelandic' ,['Icelandic_Yslenska-Latin1']]
,['Sanskrit ' ,['Sanskrit-UTF8']]
,['Latin',['Latin_Latina-Latin1', 'Latin_Latina-v2-Latin1']]
,['Greek', ['Greek_Ellinika-Greek', 'Greek_Ellinika-UTF8']]
,['Swahili', ['Swaheli-Latin1','Swahili_Kiswahili-Latin1']]

]


I am working on the file "Jupyter/languages_and_corpus.ipynb".
The first task is:
Plot styling
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt

plt.style.use('bmh')
%matplotlib inline

fig_size = plt.rcParams["figure.figsize"]
fig_size[0] = 14
fig_size[1] = 9
plt.rcParams["figure.figsize"] = fig_size

lang_names=[]
for i in range(len(list_of_languages)):
    lang_names.append((list_of_languages[i][0]))


I am working on the file "Jupyter/languages_and_corpus.ipynb".
The first task is:
The Parts of Speech Tagging in these languages
Can you write Python code for it?

import nltk

for i in range(len(list_of_languages)):
    print(list_of_languages[i][0])
    words = nltk.pos_tag(nltk.corpus.udhr.words(list_of_languages[i][1][0]))

    words_types = []
    for i in range(len(words)):
        words_types.append(words[i][1])

    print(set(words_types))

    print("\n\n\n\n\n\n\n")


I am working on the file "Jupyter/plot_compare_reduction.ipynb".
The first task is:
Selecting dimensionality reduction with Pipeline and GridSearchCV
This example constructs a pipeline that does dimensionality
reduction followed by prediction with a support vector
classifier. It demonstrates the use of GridSearchCV and
Pipeline to optimize over different classes of estimators in a
single CV run -- unsupervised PCA and NMF dimensionality
reductions are compared to univariate feature selection during
the grid search.
Additionally, Pipeline can be instantiated with the memory
argument to memoize the transformers within the pipeline, avoiding to fit
again the same transformers over and over.
Note that the use of memory to enable caching becomes interesting when the
fitting of a transformer is costly.
Illustration of Pipeline and GridSearchCV

This section illustrates the use of a Pipeline with
 GridSearchCV
Can you write Python code for it?

# Authors: Robert McGibbon, Joel Nothman, Guillaume Lemaitre

from __future__ import print_function, division

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_digits
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.svm import LinearSVC
from sklearn.decomposition import PCA, NMF
from sklearn.feature_selection import SelectKBest, chi2

print(__doc__)

pipe = Pipeline([
    ('reduce_dim', PCA()),
    ('classify', LinearSVC())
])

N_FEATURES_OPTIONS = [2, 4, 8]
C_OPTIONS = [1, 10, 100, 1000]
param_grid = [
    {
        'reduce_dim': [PCA(iterated_power=7), NMF()],
        'reduce_dim__n_components': N_FEATURES_OPTIONS,
        'classify__C': C_OPTIONS
    },
    {
        'reduce_dim': [SelectKBest(chi2)],
        'reduce_dim__k': N_FEATURES_OPTIONS,
        'classify__C': C_OPTIONS
    },
]
reducer_labels = ['PCA', 'NMF', 'KBest(chi2)']

grid = GridSearchCV(pipe, cv=3, n_jobs=1, param_grid=param_grid)
digits = load_digits()
grid.fit(digits.data, digits.target)

mean_scores = np.array(grid.cv_results_['mean_test_score'])
# scores are in the order of param_grid iteration, which is alphabetical
mean_scores = mean_scores.reshape(len(C_OPTIONS), -1, len(N_FEATURES_OPTIONS))
# select score for best C
mean_scores = mean_scores.max(axis=0)
bar_offsets = (np.arange(len(N_FEATURES_OPTIONS)) *
               (len(reducer_labels) + 1) + .5)

plt.figure()
COLORS = 'bgrcmyk'
for i, (label, reducer_scores) in enumerate(zip(reducer_labels, mean_scores)):
    plt.bar(bar_offsets + i, reducer_scores, label=label, color=COLORS[i])

plt.title("Comparing feature reduction techniques")
plt.xlabel('Reduced number of features')
plt.xticks(bar_offsets + len(reducer_labels) / 2, N_FEATURES_OPTIONS)
plt.ylabel('Digit classification accuracy')
plt.ylim((0, 1))
plt.legend(loc='upper left')


I am working on the file "JupyterDataSeeker.ipynb".
The first task is:
Gists
Can you write Python code for it?

gists = [g for g in gh.gists()]

gists[0].as_dict()


I am working on the file "JupyterDataSeeker.ipynb".
The first task is:
GitHub Authorization
Note: Be careful. Don't check in to version control your username and password.
Can you write Python code for it?

gh = github3.login('willingc', '')

type(gh)


I am working on the file "JupyterDataSeeker.ipynb".
The first task is:
Issues in a Jupyter Repo
Can you write Python code for it?

def list_issues(token, owner='jupyter', repo=repo):
    for issue in token.iter_repo_issues(owner=owner, repository=repo):
        if issue.state == 'open':
            print(issue.number, issue.title)


I am working on the file "JupyterDataSeeker.ipynb".
The first task is:
Issues in a subproject repo
Can you write Python code for it?

for issue in willingc.iter_repo_issues('jupyter','notebook'):
    if issue.state == 'open':
        print(issue.number, issue.title)
myissues = willingc.iter_repo_issues('jupyter','notebook')

for issue in gh.iter_repo_issues(owner='jupyter', repository='jupyter'):
    print(issue.number, issue.title)
    print(issue.labels)
    print(issue.created_at)
    print(issue.updated_at)


I am working on the file "JupyterDataSeeker.ipynb".
The first task is:
Jupyter Subprojects
User Interface, Kernels, Formatting and conversion, Education, Deployment, Architecture
Can you write Python code for it?

sub_jup_ui = ['jupyter/notebook', 'jupyter/jupyter_console', 'jupyter/qtconsole']
sub_kernels = ['ipython/ipython', 'ipython/ipythonwidgets', 'ipython/ipyparallel']
sub_formatting = ['jupyter/nbconvert', 'jupyter/nbformat']
sub_education = ['jupyter/nbgrader']
sub_deployment= ['jupyter/jupyterhub', 'jupyter/jupyter-drive', 'jupyter/nbviewer','jupyter/tmpnb', 'jupyter/tmpnb-deploy','jupyter/dockerspawner','jupyter/docker-stacks']
sub_architecture = ['jupyter/jupyter_client', 'jupyter/jupyter_core']

for subproject in sub_jup_ui:
    print(subproject)


I am working on the file "JupyterDataSeeker.ipynb".
The first task is:
User
Can you write Python code for it?

willingc = gh.user('willingc')

type(willingc)

print(willingc.name)

# print followers
for f in willingc.followers():
    print(f)

# print a dictionary of user's content
print(willingc.as_dict())


I am working on the file "JupyterFiles/CDOPricingNEW.ipynb".
The first task is:
Adjusted Binomial Model
The adjusted binomial model is given as
$$\begin{aligned}
E[min(L(T),K)] &= \sum_{k=0}^{N} h(k) \min (Lk, K) \
h(k) &= \int_{- \infty}^{\infty} \phi (z) g(k) dz \
g(k) &= \alpha f(k) \text{ for } k \neq l,u \
g(l) &= f(l) + (1 - \alpha)(u - m) \
g(u) &= f(u) - (1 - \alpha)(l - m) \
l &= \text{floor} (m) \
m &= p N \
u &= l + 1 \
p &= \frac{1}{N} \sum_{i=1}^N \frac{(1 - R_i)F_i}{L} p_i \
\alpha &= \frac{v_E N + (u - m)^2 -((l -m)^2 - (u -m)^2)(u -m)}
{v_A N + (u - m)^2 -((l -m)^2 - (u -m)^2)(u -m)} \
v_E &= \frac{1}{N^2} \sum_{i=1}^{N} \left( \frac{(1-R_i)F_i}{L}\right)^2 p_i(1 - p_i) \
v_A &= \frac{p(1-p)}{N} \
f(k) &= \frac{N!}{(N - k)! k!} p^k (1 - p)^{N - k} \
L &= \frac{1}{N} \sum_{i=1}^{N} (1 - R_i) F_i
\end{aligned}$$
Can you write Python code for it?

def Q_adjbinom(t, K1, K2, Fs, Rs, betas, Qs):
    if Qs[0](t) == 1:
        return 1.0 # initial value -- avoids weird nan return

    N = len(Fs)
    Cs = [norm.ppf(1 - Q(t)) for Q in Qs]
    L = sum([(1 - R) * F for R, F in zip(Rs, Fs)]) / N
    def choose(n, k): # Calculates binomial coeffecient: n choose k.
        if k == 0 or k == n:
            return 1
        return choose(n - 1, k - 1) + choose(n - 1, k)
    def g(k,z):
        ps = [norm.cdf((C - beta * z) / sqrt(1 - beta * beta)) for C, beta in zip(Cs, betas)]
        p_avg = sum([(1 - R) * F / L * p for R, F, p in zip(Rs, Fs, ps)]) / N
        f = lambda k: choose(N, k) * p_avg ** k * (1 - p_avg) ** (N - k)
        vA = p_avg * (1 - p_avg) / N
        vE = 1 / N / N * sum([((1 - R) * F / L) ** 2 * p * (1 - p) for R, F, p in zip(Rs, Fs, ps)])
        m = p_avg * N
        l = int(m)
        u = l + 1
        o = (u - m) ** 2 + ((l - m) ** 2 - (u - m) ** 2) * (u - m)
        alpha = (vE * N + o) / (vA * N + o)
        if k == l:
            return f(l) + (1 - alpha) * (u - m)
        if k == u:
            return f(u) - (1 - alpha) * (l - m)
        return alpha * f(k)
    I = lambda k: quad(lambda z: norm.pdf(z) * g(k, z), -10, 10)[0]
    emin = lambda K: sum([I(k) * min(L * k, K) for k in range(0, N + 1)])
    return 1 - (emin(K2) - emin(K1)) / (K2 - K1)


I am working on the file "JupyterFiles/CDOPricingNEW.ipynb".
The first task is:
Example
Can you write Python code for it?

from numpy import exp
from datetime import date, timedelta
K1 = 0.03
K2 = 0.07
Fs = [0.3, 0.8]
Rs = [0.40, 0.60]
def expdecay(today, rate):
    return lambda t: exp(-1 * (t - today).days / 365 * rate)
today = date(2012,1, 1)
Qs = [expdecay(today, 0.0120), expdecay(today, 0.0160)]
betas = [0.30, 0.40]
tvalues = [today + timedelta(days = 30) * n for n in range(37)] #3 years
# LHP perameters average the other perameters
R = 0.50
Q = expdecay(today, 0.0140)
beta = 0.35

# Takes FOREVER
lhpcurve = [Q_lhp(t, K1, K2, R, beta, Q) for t in tvalues]
adjointbinomialcurve = [Q_adjbinom(t, K1, K2, Fs, Rs, betas, Qs) for t in tvalues]
gaussiancurve = [Q_gauss(t, K1, K2, Fs, Rs, betas, Qs) for t in tvalues]

%matplotlib inline
import matplotlib.pyplot as plt
plt.plot(tvalues, lhpcurve, 'ro', #red
         tvalues, adjointbinomialcurve, 'bo', #blue
         tvalues, gaussiancurve, 'go' #green
         )
plt.show

## Test Spreads
maturity = tvalues
remaining_payments = tvalues
Z = expdecay(today, 0.0140)

#print(lhpcurve[1])
N = len(lhpcurve)
print(N)
#print(today)
#print(maturity)
CDS1 = CDS(today=today, maturity = maturity, remaining_payments = remaining_payments, R = R, Z = Z, Q = lhpcurve)

CDS1.parSpread(N=N)


I am working on the file "JupyterFiles/CDOPricingNEW.ipynb".
The first task is:
Large Homogenous Portfolio (LHP) Model
The LHP tranche pricing model is given by the following equations:
$$\begin{aligned}
E[\min (L(T), K)] &= (1-R) \Phi_{2}(C(T), -A(K), - \beta) + K \Phi(A(K)) \
A(K) &= \frac{1}{\beta} \left( C(T) - \sqrt{1 - \beta^{2}} \Phi^{-1} \left( \frac{K}{1-R} \right) \right) \
C(T) &= \Phi^{-1} (1 - Q(T))
\end{aligned}$$
Here:
* $\Phi_{2}$ is the cdf of the bivariate normal distribution. The third perameter being correlation.
* $\beta^{2}$ the correlation
* $R$ is the recovery rate.
* $Q(T)$ is the survival curve
Can you write Python code for it?

from scipy.stats import norm, mvn #normal and bivariate normal
from numpy       import sqrt

def Q_lhp(t, K1, K2, R, beta, Q):
    """Calculates the Tranche survival curve for the LHP model.

    Args:
        T (datetime.date): Should be given in "days" (no hours, minutes, etc.)
        K1 (float): The starting value of the tranche. Its value should be
            between 0 & 1.
        K2 (float): The final value of the tranche.
        beta (float): Correlation perameter. Its value should be between 0 and 1.
        R (float): Recovery rate. Usually between 0 and 1.
        Q (callable function): A function Q that takes in a dateime.date and
            outputs a float from 0 to 1. It is assumed that Q(0) = 1 and Q is
            decreasing. Represents survival curve of each credit.
    """
    if Q(t) == 1:
        return 1 # prevents infinity
    def emin(K):
    # Calculates E[min(L(T), K)] in LHP model
        print(Q(t))
        C = norm.ppf(1 - Q(t))
        A = (1 / beta) * (C - sqrt(1 - beta * beta) * norm.ppf(K / (1 - R)))
        return (1 - R) * mvn.mvndst(upper = [C, -1 * A],
                                    lower = [0,0],
                                    infin = [0,0], # set lower bounds = -infty
                                    correl =  -1 * beta)[1] + K * norm.cdf(A)
    return 1 - (emin(K2) - emin(K1)) / (K2 - K1)


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT2.ipynb".
The first task is:
Save seismograms
Can you write Python code for it?

## Save seismograms
np.save("Seismograms/FD_1D_DX4_DT2",Seismogramm)

## Plot seismograms
fig, (ax1, ax2, ax3) = plt.subplots(3, 1)
fig.subplots_adjust(hspace=0.4,right=1.6, top = 2 )

ax1.plot(t,Seismogramm[0,:])
ax1.set_title('Seismogram 1')
ax1.set_ylabel('Amplitude')
ax1.set_xlabel('Time in s')
ax1.set_xlim(0, T)

ax2.plot(t,Seismogramm[1,:])
ax2.set_title('Seismogram 2')
ax2.set_ylabel('Amplitude')
ax2.set_xlabel('Time in s')
ax2.set_xlim(0, T)

ax3.plot(t,Seismogramm[2,:])
ax3.set_title('Seismogram 3')
ax3.set_ylabel('Amplitude')
ax3.set_xlabel('Time in s')
ax3.set_xlim(0, T);


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT2.ipynb".
The first task is:
Time stepping
Can you write Python code for it?

# Init Seismograms
Seismogramm=np.zeros((3,nt)); # Three seismograms

# Calculation of some coefficients
i_dx=1.0/(dx)

## Time stepping
print("Starting time stepping...")
for n in range(2,nt):

        # Inject source wavelet
        p[xscr]=p[xscr]+q[n]

        # Update velocity
        for kx in range(5,nx-4):
            # Calculating spatial derivative
            p_x=i_dx*9.0/8.0*(p[kx+1]-p[kx])-i_dx*1.0/24.0*(p[kx+2]-p[kx-1])

            # Update velocity
            vx[kx]=vx[kx]-dt/rho[kx]*p_x

        # Update pressure
        for kx in range(5,nx-4):

            # Calculating spatial derivative
            vx_x= i_dx*9.0/8.0*(vx[kx]-vx[kx-1])-i_dx*1.0/24.0*(vx[kx+1]-vx[kx-2])

            # Update pressure
            p[kx]=p[kx]-l[kx]*dt*(vx_x)

        # Save seismograms
        Seismogramm[0,n]=p[xrec1]
        Seismogramm[1,n]=p[xrec2]
        Seismogramm[2,n]=p[xrec3]
print("Finished time stepping!")


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT2_fast.ipynb".
The first task is:
Input Parameter
Can you write Python code for it?

# Discretization
c1=20   # Number of grid points per dominant wavelength
c2=0.5  # CFL-Number
nx=2000 # Number of grid points
T=10     # Total propagation time

# Source Signal
f0= 10      # Center frequency Ricker-wavelet
q0= 1       # Maximum amplitude Ricker-Wavelet
xscr = 100  # Source position (in grid points)

# Receiver
xrec1=400  # Position Reciever 1 (in grid points)
xrec2=800  # Position Reciever 2 (in grid points)
xrec3=1800 # Position Reciever 3 (in grid points)

# Velocity and density
modell_v = np.hstack((1000*np.ones((int(nx/2))),1500*np.ones((int(nx/2)))))
rho=np.hstack((1*np.ones((int(nx/2))),1.5*np.ones((int(nx/2)))))


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT2_fast.ipynb".
The first task is:
Save seismograms
Can you write Python code for it?

## Save seismograms
np.save("Seismograms/FD_1D_DX4_DT2_fast",Seismogramm)

## Plot seismograms
fig, (ax1, ax2, ax3) = plt.subplots(3, 1)
fig.subplots_adjust(hspace=0.4,right=1.6, top = 2 )

ax1.plot(t,Seismogramm[0,:])
ax1.set_title('Seismogram 1')
ax1.set_ylabel('Amplitude')
ax1.set_xlabel('Time in s')
ax1.set_xlim(0, T)

ax2.plot(t,Seismogramm[1,:])
ax2.set_title('Seismogram 2')
ax2.set_ylabel('Amplitude')
ax2.set_xlabel('Time in s')
ax2.set_xlim(0, T)

ax3.plot(t,Seismogramm[2,:])
ax3.set_title('Seismogram 3')
ax3.set_ylabel('Amplitude')
ax3.set_xlabel('Time in s')
ax3.set_xlim(0, T);


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT2_fast.ipynb".
The first task is:
Source signal - Ricker-wavelet
Can you write Python code for it?

tau=np.pi*f0*(t-1.5/f0)
q=q0*(1.0-2.0*tau**2.0)*np.exp(-tau**2)

# Plotting source signal
plt.figure(3)
plt.plot(t,q)
plt.title('Source signal Ricker-Wavelet')
plt.ylabel('Amplitude')
plt.xlabel('Time in s')
plt.draw()


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT2_fast.ipynb".
The first task is:
Time stepping
Can you write Python code for it?

# Init Seismograms
Seismogramm=np.zeros((3,nt)); # Three seismograms

# Calculation of some coefficients
i_dx=1.0/(dx)
kx=np.arange(5,nx-4)

print("Starting time stepping...")
## Time stepping
for n in range(2,nt):

        # Inject source wavelet
        p[xscr]=p[xscr]+q[n]

        # Calculating spatial derivative
        p_x=i_dx*9.0/8.0*(p[kx+1]-p[kx])-i_dx*1.0/24.0*(p[kx+2]-p[kx-1])

        # Update velocity
        vx[kx]=vx[kx]-dt/rho[kx]*p_x

        # Calculating spatial derivative
        vx_x= i_dx*9.0/8.0*(vx[kx]-vx[kx-1])-i_dx*1.0/24.0*(vx[kx+1]-vx[kx-2])

        # Update pressure
        p[kx]=p[kx]-l[kx]*dt*(vx_x);

        # Save seismograms
        Seismogramm[0,n]=p[xrec1]
        Seismogramm[1,n]=p[xrec2]
        Seismogramm[2,n]=p[xrec3]

print("Finished time stepping!")


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT3_ABS.ipynb".
The first task is:
Create space and time vector
Can you write Python code for it?

x=np.arange(0,dx*nx,dx) # Space vector
t=np.arange(0,T,dt)     # Time vector
nt=np.size(t)           # Number of time steps

# Plotting model
fig, (ax1, ax2) = plt.subplots(1, 2)
fig.subplots_adjust(wspace=0.4,right=1.6)
ax1.plot(x,modell_v)
ax1.set_ylabel('VP in m/s')
ax1.set_xlabel('Depth in m')
ax1.set_title('P-wave velocity')

ax2.plot(x,rho)
ax2.set_ylabel('Density in g/cm^3')
ax2.set_xlabel('Depth in m')
ax2.set_title('Density');


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT3_ABS.ipynb".
The first task is:
Input Parameter
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT3_ABS.ipynb".
The first task is:
Preparation
Can you write Python code for it?

# Init wavefields
vx=np.zeros(nx)
p=np.zeros(nx)
vx_x=np.zeros(nx)
p_x=np.zeros(nx)
vx_x2=np.zeros(nx)
p_x2=np.zeros(nx)
vx_x3=np.zeros(nx)
p_x3=np.zeros(nx)

# Calculate first Lame-Paramter
l=rho * modell_v * modell_v

cmin=min(modell_v.flatten())  # Lowest P-wave velocity
cmax=max(modell_v.flatten())  # Highest P-wave velocity
fmax=2*f0                     # Maximum frequency
dx=cmin/(fmax*c1)             # Spatial discretization (in m)
dt=dx/(cmax)*c2               # Temporal discretization (in s)
lampda_min=cmin/fmax          # Smallest wavelength

# Output model parameter:
print("Model size: x:",dx*nx,"in m")
print("Temporal discretization: ",dt," s")
print("Spatial discretization: ",dx," m")
print("Number of gridpoints per minimum wavelength: ",lampda_min/dx)


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT3_ABS.ipynb".
The first task is:
Source signal - Ricker-wavelet
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT3_ABS_fast.ipynb".
The first task is:
Create space and time vector
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT3_ABS_fast.ipynb".
The first task is:
Input Parameter
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT3_ABS_fast.ipynb".
The first task is:
Preparation
Can you write Python code for it?

# Init wavefields
vx=np.zeros(nx)
p=np.zeros(nx)
vx_x=np.zeros((nx),float)
p_x=np.zeros((nx),float)
vx_x2=np.zeros((nx),float)
p_x2=np.zeros((nx),float)
vx_x3=np.zeros((nx),float)
p_x3=np.zeros((nx),float)

# Calculate first Lame-Paramter
l=rho * modell_v * modell_v

cmin=min(modell_v.flatten())  # Lowest P-wave velocity
cmax=max(modell_v.flatten())  # Highest P-wave velocity
fmax=2*f0                     # Maximum frequency
dx=cmin/(fmax*c1)             # Spatial discretization (in m)
dt=dx/(cmax)*c2               # Temporal discretization (in s)
lampda_min=cmin/fmax          # Smallest wavelength

# Output model parameter:
print("Model size: x:",dx*nx,"in m")
print("Temporal discretization: ",dt," s")
print("Spatial discretization: ",dx," m")
print("Number of gridpoints per minimum wavelength: ",lampda_min/dx)


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT4_ABS.ipynb".
The first task is:
Preparation
Can you write Python code for it?

# Init wavefields
vx=np.zeros(nx)
p=np.zeros(nx)
vx=np.zeros(nx)
p=np.zeros(nx)
vx_x=np.zeros(nx)
p_x=np.zeros(nx)
vx_x2=np.zeros(nx)
p_x2=np.zeros(nx)
vx_x3=np.zeros(nx)
p_x3=np.zeros(nx)
vx_x4=np.zeros(nx)
p_x4=np.zeros(nx)

# Calculate first Lame-Paramter
l=rho * modell_v * modell_v

cmin=min(modell_v.flatten())  # Lowest P-wave velocity
cmax=max(modell_v.flatten())  # Highest P-wave velocity
fmax=2*f0                     # Maximum frequency
dx=cmin/(fmax*c1)             # Spatial discretization (in m)
dt=dx/(cmax)*c2               # Temporal discretization (in s)
lampda_min=cmin/fmax          # Smallest wavelength

# Output model parameter:
print("Model size: x:",dx*nx,"in m")
print("Temporal discretization: ",dt," s")
print("Spatial discretization: ",dx," m")
print("Number of gridpoints per minimum wavelength: ",lampda_min/dx)


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT4_ABS.ipynb".
The first task is:
Save seismograms
Can you write Python code for it?

## Save seismograms
np.save("Seismograms/FD_1D_DX4_DT4_ABS",Seismogramm)

## Plot seismograms
fig, (ax1, ax2, ax3) = plt.subplots(3, 1)
fig.subplots_adjust(hspace=0.4,right=1.6, top = 2 )

ax1.plot(t,Seismogramm[0,:])
ax1.set_title('Seismogram 1')
ax1.set_ylabel('Amplitude')
ax1.set_xlabel('Time in s')
ax1.set_xlim(0, T)

ax2.plot(t,Seismogramm[1,:])
ax2.set_title('Seismogram 2')
ax2.set_ylabel('Amplitude')
ax2.set_xlabel('Time in s')
ax2.set_xlim(0, T)

ax3.plot(t,Seismogramm[2,:])
ax3.set_title('Seismogram 3')
ax3.set_ylabel('Amplitude')
ax3.set_xlabel('Time in s')
ax3.set_xlim(0, T);


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT4_ABS.ipynb".
The first task is:
Source signal - Ricker-wavelet
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT4_ABS.ipynb".
The first task is:
Time stepping
Can you write Python code for it?

# Init Seismograms
Seismogramm=np.zeros((3,nt)); # Three seismograms

# Calculation of some coefficients
i_dx=1.0/(dx)

print("Starting time stepping...")
## Time stepping
for n in range(2,nt):

        # Inject source wavelet
        p[xscr]=p[xscr]+q[n]

        # Update velocity
        for kx in range(5,nx-4):
            # Calculating spatial derivative
            p_x[kx]=i_dx*9.0/8.0*(p[kx+1]-p[kx])-i_dx*1.0/24.0*(p[kx+2]-p[kx-1])

            # Update velocity
            vx[kx]=vx[kx]-dt/rho[kx]*(13.0/12.0*p_x[kx]-5.0/24.0*p_x2[kx]+1.0/6.0*p_x3[kx]-1.0/24.0*p_x4[kx])

        # np.np.zeros old spatial derivations for Adam-Bashforth method
        np.copyto(p_x4,p_x3)
        np.copyto(p_x3,p_x2)
        np.copyto(p_x2,p_x)

        # Update pressure
        for kx in range(5,nx-4):

            # Calculating spatial derivative
            vx_x[kx]= i_dx*9.0/8.0*(vx[kx]-vx[kx-1])-i_dx*1.0/24.0*(vx[kx+1]-vx[kx-2])

            # Update pressure
            p[kx]=p[kx]-l[kx]*dt*(13.0/12.0*vx_x[kx]-5.0/24.0*vx_x2[kx]+1.0/6.0*vx_x3[kx]-1.0/24.0*vx_x4[kx])

        # np.np.zeros old spatial derivations for Adam-Bashforth method
        np.copyto(vx_x4,vx_x3)
        np.copyto(vx_x3,vx_x2)
        np.copyto(vx_x2,vx_x)

        # Save seismograms
        Seismogramm[0,n]=p[xrec1]
        Seismogramm[1,n]=p[xrec2]
        Seismogramm[2,n]=p[xrec3]

print("Finished time stepping!")


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT4_ABS_fast.ipynb".
The first task is:
Preparation
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT4_ABS_fast.ipynb".
The first task is:
Save seismograms
Can you write Python code for it?

## Save seismograms
np.save("Seismograms/FD_1D_DX4_DT4_ABS_fast",Seismogramm)

## Plot seismograms
fig, (ax1, ax2, ax3) = plt.subplots(3, 1)
fig.subplots_adjust(hspace=0.4,right=1.6, top = 2 )

ax1.plot(t,Seismogramm[0,:])
ax1.set_title('Seismogram 1')
ax1.set_ylabel('Amplitude')
ax1.set_xlabel('Time in s')
ax1.set_xlim(0, T)

ax2.plot(t,Seismogramm[1,:])
ax2.set_title('Seismogram 2')
ax2.set_ylabel('Amplitude')
ax2.set_xlabel('Time in s')
ax2.set_xlim(0, T)

ax3.plot(t,Seismogramm[2,:])
ax3.set_title('Seismogram 3')
ax3.set_ylabel('Amplitude')
ax3.set_xlabel('Time in s')
ax3.set_xlim(0, T);


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT4_ABS_fast.ipynb".
The first task is:
Source signal - Ricker-wavelet
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT4_LW.ipynb".
The first task is:
Create space and time vector
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT4_LW.ipynb".
The first task is:
Input Parameter
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT4_LW.ipynb".
The first task is:
Preparation
Can you write Python code for it?

# Init wavefields
vx=np.zeros(nx)
p=np.zeros(nx)

# Calculate first Lame-Paramter
l=rho * modell_v * modell_v

cmin=min(modell_v.flatten())  # Lowest P-wave velocity
cmax=max(modell_v.flatten())  # Highest P-wave velocity
fmax=2*f0                     # Maximum frequency
dx=cmin/(fmax*c1)             # Spatial discretization (in m)
dt=dx/(cmax)*c2               # Temporal discretization (in s)
lampda_min=cmin/fmax          # Smallest wavelength

# Output model parameter:
print("Model size: x:",dx*nx,"in m")
print("Temporal discretization: ",dt," s")
print("Spatial discretization: ",dx," m")
print("Number of gridpoints per minimum wavelength: ",lampda_min/dx)


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT4_LW.ipynb".
The first task is:
Save seismograms
Can you write Python code for it?

## Save seismograms
np.save("Seismograms/FD_1D_DX4_DT4_LW",Seismogramm)

## Plot seismograms
fig, (ax1, ax2, ax3) = plt.subplots(3, 1)
fig.subplots_adjust(hspace=0.4,right=1.6, top = 2 )

ax1.plot(t,Seismogramm[0,:])
ax1.set_title('Seismogram 1')
ax1.set_ylabel('Amplitude')
ax1.set_xlabel('Time in s')
ax1.set_xlim(0, T)

ax2.plot(t,Seismogramm[1,:])
ax2.set_title('Seismogram 2')
ax2.set_ylabel('Amplitude')
ax2.set_xlabel('Time in s')
ax2.set_xlim(0, T)

ax3.plot(t,Seismogramm[2,:])
ax3.set_title('Seismogram 3')
ax3.set_ylabel('Amplitude')
ax3.set_xlabel('Time in s')
ax3.set_xlim(0, T);


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT4_LW.ipynb".
The first task is:
Source signal - Ricker-wavelet
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT4_LW.ipynb".
The first task is:
Time stepping
Can you write Python code for it?

# Init Seismograms
Seismogramm=np.zeros((3,nt)); # Three seismograms

# Calculation of some coefficients
i_dx=1.0/(dx)
i_dx3=1.0/(dx**3)
c9=dt**3/24.0

print("Starting time stepping...")
## Time stepping
for n in range(2,nt):

        # Inject source wavelet
        p[xscr]=p[xscr]+q[n]

        # Update velocity
        for kx in range(5,nx-4):
            # Calculating spatial derivative
            p_x=i_dx*9.0/8.0*(p[kx+1]-p[kx])-i_dx*1.0/24.0*(p[kx+2]-p[kx-1])
            p_xxx=i_dx3*(-3.0)*(p[kx+1]-p[kx])+i_dx3*(1)*(p[kx+2]-p[kx-1])

            # Update velocity
            vx[kx]=vx[kx]-dt/rho[kx]*p_x-l[kx]*c9*1/(rho[kx]**2.0)*(p_xxx)

        # Update pressure
        for kx in range(5,nx-4):

            # Calculating spatial derivative
            vx_x= i_dx*9.0/8.0*(vx[kx]-vx[kx-1])-i_dx*1.0/24.0*(vx[kx+1]-vx[kx-2])
            vx_xxx=i_dx3*(-3.0)*(vx[kx]-vx[kx-1])+i_dx3*(1)*(vx[kx+1]-vx[kx-2])

            # Update pressure
            p[kx]=p[kx]-l[kx]*dt*(vx_x)-l[kx]**2*c9*1/(rho[kx])*(vx_xxx)

        # Save seismograms
        Seismogramm[0,n]=p[xrec1]
        Seismogramm[1,n]=p[xrec2]
        Seismogramm[2,n]=p[xrec3]

print("Finished time stepping!")


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT4_LW_fast.ipynb".
The first task is:
Create space and time vector
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT4_LW_fast.ipynb".
The first task is:
Preparation
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT4_LW_fast.ipynb".
The first task is:
Save seismograms
Can you write Python code for it?

## Save seismograms
np.save("Seismograms/FD_1D_DX4_DT4_LW_fast",Seismogramm)

## Plot seismograms
fig, (ax1, ax2, ax3) = plt.subplots(3, 1)
fig.subplots_adjust(hspace=0.4,right=1.6, top = 2 )

ax1.plot(t,Seismogramm[0,:])
ax1.set_title('Seismogram 1')
ax1.set_ylabel('Amplitude')
ax1.set_xlabel('Time in s')
ax1.set_xlim(0, T)

ax2.plot(t,Seismogramm[1,:])
ax2.set_title('Seismogram 2')
ax2.set_ylabel('Amplitude')
ax2.set_xlabel('Time in s')
ax2.set_xlim(0, T)

ax3.plot(t,Seismogramm[2,:])
ax3.set_title('Seismogram 3')
ax3.set_ylabel('Amplitude')
ax3.set_xlabel('Time in s')
ax3.set_xlim(0, T);


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT4_LW_fast.ipynb".
The first task is:
Source signal - Ricker-wavelet
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_DX4_DT4_LW_fast.ipynb".
The first task is:
Time stepping
Can you write Python code for it?

# Init Seismograms
Seismogramm=np.zeros((3,nt)); # Three seismograms

# Calculation of some coefficients
i_dx=1.0/(dx)
i_dx3=1.0/(dx**3)
c9=dt**3/24.0
kx=np.arange(5,nx-4)

print("Starting time stepping...")
## Time stepping
for n in range(2,nt):

        # Inject source wavelet
        p[xscr]=p[xscr]+q[n]

        # Calculating spatial derivative
        p_x=i_dx*9.0/8.0*(p[kx+1]-p[kx])-i_dx*1.0/24.0*(p[kx+2]-p[kx-1])
        p_xxx=i_dx3*(-3.0)*(p[kx+1]-p[kx])+i_dx3*(1.0)*(p[kx+2]-p[kx-1])

        # Update velocity
        vx[kx]=vx[kx]-dt/rho[kx]*p_x-l[kx]*c9*1/(rho[kx]**2)*(p_xxx)

        # Calculating spatial derivative
        vx_x= i_dx*9.0/8.0*(vx[kx]-vx[kx-1])-i_dx*1.0/24.0*(vx[kx+1]-vx[kx-2])
        vx_xxx=i_dx3*(-3.0)*(vx[kx]-vx[kx-1])+i_dx3*(1.0)*(vx[kx+1]-vx[kx-2])

        # Update pressure
        p[kx]=p[kx]-l[kx]*dt*(vx_x)-l[kx]**2.0*c9*1.0/(rho[kx])*(vx_xxx)

        # Save seismograms
        Seismogramm[0,n]=p[xrec1]
        Seismogramm[1,n]=p[xrec2]
        Seismogramm[2,n]=p[xrec3]

print("Finished time stepping!")


I am working on the file "JupyterNotebook/1D/FD_1D_DX8_DT2.ipynb".
The first task is:
Create space and time vector
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_DX8_DT2.ipynb".
The first task is:
Input Parameter
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_DX8_DT2.ipynb".
The first task is:
Source signal - Ricker-wavelet
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_DX8_DT2.ipynb".
The first task is:
Time stepping
Can you write Python code for it?

# Init Seismograms
Seismogramm=np.zeros((3,nt)); # Three seismograms

# Calculation of some coefficients
i_dx=1.0/(dx)
kx=np.arange(5,nx-4)

print("Starting time stepping...")
## Time stepping
for n in range(2,nt):

        # Inject source wavelet
        p[xscr]=p[xscr]+q[n]

        # Update velocity
        for kx in range(6,nx-5):

            # Calculating spatial derivative
            p_x=i_dx*(1225.0/1024.0)*(p[kx+1]-p[kx])+i_dx*(-245.0/3072.0)*(p[kx+2]-p[kx-1])+i_dx*(49.0/5120.0)*(p[kx+3]-p[kx-2])+i_dx*(-5.0/7168.0)*(p[kx+4]-p[kx-3])

            # Update velocity
            vx[kx]=vx[kx]-dt/rho[kx]*p_x

        # Update pressure
        for kx in range(6,nx-5):

            # Calculating spatial derivative
            vx_x=i_dx*(1225.0/1024.0)*(vx[kx]-vx[kx-1])+i_dx*(-245.0/3072.0)*(vx[kx+1]-vx[kx-2])+i_dx*(49.0/5120.0)*(vx[kx+2]-vx[kx-3])+i_dx*(-5.0/7168.0)*(vx[kx+3]-vx[kx-4])

            # Update pressure
            p[kx]=p[kx]-l[kx]*dt*(vx_x);

        # Save seismograms
        Seismogramm[0,n]=p[xrec1]
        Seismogramm[1,n]=p[xrec2]
        Seismogramm[2,n]=p[xrec3]

print("Finished time stepping!")


I am working on the file "JupyterNotebook/1D/FD_1D_DX8_DT2_fast.ipynb".
The first task is:
Input Parameter
Can you write Python code for it?

# Discretization
c1=20   # Number of grid points per dominant wavelength
c2=0.5  # CFL-Number
nx=2000 # Number of grid points
T=10     # Total propagation time

# Source Signal
f0= 10      # Center frequency Ricker-wavelet
q0= 1       # Maximum amplitude Ricker-Wavelet
xscr = 100  # Source position (in grid points)

# Receiver
xrec1=400  # Position Reciever 1 (in grid points)
xrec2=800  # Position Reciever 2 (in grid points)
xrec3=1800 # Position Reciever 3 (in grid points)

# Velocity and density
modell_v = np.hstack((1000*np.ones((int(nx/2))),1500*np.ones((int(nx/2)))))
rho=np.hstack((1*np.ones((int(nx/2))),1.5*np.ones((int(nx/2)))))


I am working on the file "JupyterNotebook/1D/FD_1D_DX8_DT2_fast.ipynb".
The first task is:
Source signal - Ricker-wavelet
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_DX8_DT2_fast.ipynb".
The first task is:
Time stepping
Can you write Python code for it?

# Init Seismograms
Seismogramm=np.zeros((3,nt)); # Three seismograms

# Calculation of some coefficients
i_dx=1.0/(dx)
kx=np.arange(5,nx-4)

print("Starting time stepping...")
## Time stepping
for n in range(2,nt):

        # Inject source wavelet
        p[xscr]=p[xscr]+q[n]

        # Calculating spatial derivative
        p_x=i_dx*(1225.0/1024.0)*(p[kx+1]-p[kx])+i_dx*(-245.0/3072.0)*(p[kx+2]-p[kx-1])+i_dx*(49.0/5120.0)*(p[kx+3]-p[kx-2])+i_dx*(-5.0/7168.0)*(p[kx+4]-p[kx-3])

        # Update velocity
        vx[kx]=vx[kx]-dt/rho[kx]*p_x

        # Calculating spatial derivative
        vx_x=i_dx*(1225.0/1024.0)*(vx[kx]-vx[kx-1])+i_dx*(-245.0/3072.0)*(vx[kx+1]-vx[kx-2])+i_dx*(49.0/5120.0)*(vx[kx+2]-vx[kx-3])+i_dx*(-5.0/7168.0)*(vx[kx+3]-vx[kx-4])

        # Update pressure
        p[kx]=p[kx]-l[kx]*dt*(vx_x);

        # Save seismograms
        Seismogramm[0,n]=p[xrec1]
        Seismogramm[1,n]=p[xrec2]
        Seismogramm[2,n]=p[xrec3]

print("Finished time stepping!")


I am working on the file "JupyterNotebook/1D/FD_1D_compare.ipynb".
The first task is:
Plotting
Can you write Python code for it?

plt.figure(1,figsize=(10, 6))
Line1,=plt.plot(FD_1D_DX4_DT2[0,:])
Line2,=plt.plot(FD_1D_DX4_DT3_ABS[0,:])
Line3,=plt.plot(FD_1D_DX4_DT4_ABS[0,:])
Line4,=plt.plot(FD_1D_DX4_DT4_LW[0,:])
Line5,=plt.plot(FD_1D_DX8_DT2[0,:])
plt.legend((Line1, Line2, Line3, Line4, Line5), ('FD_1D_DX4_DT2', 'FD_1D_DX4_DT3_ABS', 'FD_1D_DX4_DT4_ABS', 'FD_1D_DX4_DT4_LW', 'FD_1D_DX8_DT2'))
plt.title('Seismogram 1')
plt.ylabel('Amplitude')
plt.xlabel('Time in s')

plt.figure(2,figsize=(10, 6))
Line1,=plt.plot(FD_1D_DX4_DT2[1,:])
Line2,=plt.plot(FD_1D_DX4_DT3_ABS[1,:])
Line3,=plt.plot(FD_1D_DX4_DT4_ABS[1,:])
Line4,=plt.plot(FD_1D_DX4_DT4_LW[1,:])
Line5,=plt.plot(FD_1D_DX8_DT2[1,:])
plt.legend((Line1, Line2, Line3, Line4, Line5), ('FD_1D_DX4_DT2', 'FD_1D_DX4_DT3_ABS', 'FD_1D_DX4_DT4_ABS', 'FD_1D_DX4_DT4_LW', 'FD_1D_DX8_DT2'))
plt.title('Seismogram 2')
plt.ylabel('Amplitude')
plt.xlabel('Time in s')

plt.figure(3,figsize=(10, 6))
Line1,=plt.plot(FD_1D_DX4_DT2[2,:])
Line2,=plt.plot(FD_1D_DX4_DT3_ABS[2,:])
Line3,=plt.plot(FD_1D_DX4_DT4_ABS[2,:])
Line4,=plt.plot(FD_1D_DX4_DT4_LW[2,:])
Line5,=plt.plot(FD_1D_DX8_DT2[2,:])
plt.legend((Line1, Line2, Line3, Line4, Line5), ('FD_1D_DX4_DT2', 'FD_1D_DX4_DT3_ABS', 'FD_1D_DX4_DT4_ABS', 'FD_1D_DX4_DT4_LW', 'FD_1D_DX8_DT2'))
plt.title('Seismogram 3')
plt.ylabel('Amplitude')
plt.xlabel('Time in s');


I am working on the file "JupyterNotebook/1D/FD_1D_compare.ipynb".
The first task is:
Read Seismograms
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_stability.ipynb".
The first task is:
Calculate Taylor coefficient
Calculate the Taylor coefficient in an arbitrary order for the
second-order derivative.
Usage:
Lets say you want to calculate the 4th order accurate second-order
FD-stencil. Then you have to set order=4 and the result would be used
as follow:
p_x = 1/DH * ( coeff(1) * (p(x+1)-p(x)) + coeff(2) * (p(x+2)-p(x-1)) )
where p_x is the derivative.
Can you write Python code for it?

def coeff(order):
    ## Check some conditions
    if int(order)%2!=0:
        print("Error: coeff")
        print("Order has to be an integer multiple of 2!")
        return
    if order==2:
        print("Error: coeff")
        print("Order has to be at least 4!")
        return
    ## Calculation
    c=np.transpose(np.hstack((1, np.zeros(int(order/2)-1))))
    M=np.zeros((int(order/2),int(order/2)))
    # Condition 1: \sum^{N/2}_{k=1} b_k(2k-1)=1
    for n in range(1,int(order/2+1)):
        M[0,n-1]=(2*n-1)
    # Condition 2:  \sum^{N/2}_{k=1} b_k(2k-1)^(2j-1)=0; j=2,3...N/2
    for j in range(2,int(order/2+1)):
        for n in range(1,int(order/2+1)):
            M[j-1,n-1]=(2*n-1)**(2*j-1)
    coeff=np.transpose(np.dot(np.linalg.inv(M),c))
    return(coeff)


I am working on the file "JupyterNotebook/1D/FD_1D_stability.ipynb".
The first task is:
Calculate spatial derivation impact
Assumption: Spatial sampling is at the Nyquist condition
Can you write Python code for it?

sum_fd_stencil=np.sum(np.abs(coeff(spatial_order)))
theta=np.arange(0,2*np.pi,0.01)
print("You choose a spatial order of ",spatial_order)


I am working on the file "JupyterNotebook/1D/FD_1D_stability.ipynb".
The first task is:
Input Parameter
Can you write Python code for it?


I am working on the file "JupyterNotebook/1D/FD_1D_stability.ipynb".
The first task is:
Temporal 2 order (Leapfrog)
Can you write Python code for it?

f=-1j*(((np.exp(-1j*theta))**(2.5)-(np.exp(-1j*theta))**1.5))
f2=(2*sum_fd_stencil)
f_ges=f/f2
print("2. Temporal Order has the stability limit CFL=",np.max(f_ges.real))


I am working on the file "JupyterNotebook/2D/FD_2D_DX4_DT2.ipynb".
The first task is:
Preparation
Can you write Python code for it?

# Init wavefields
vx=np.zeros((ny,nx))
vy=np.zeros((ny,nx))
p=np.zeros((ny,nx))
vx_x=np.zeros((ny,nx))
vy_y=np.zeros((ny,nx))
p_x=np.zeros((ny,nx))
p_y=np.zeros((ny,nx))

# Calculate first Lame-Paramter
l=rho * modell_v * modell_v

cmin=min(modell_v.flatten())  # Lowest P-wave velocity
cmax=max(modell_v.flatten())  # Highest P-wave velocity
fmax=2*f0                     # Maximum frequency
dx=cmin/(fmax*c1)             # Spatial discretization (in m)
dy=dx                         # Spatial discretization (in m)
dt=dx/(cmax)*c2               # Temporal discretization (in s)
lampda_min=cmin/fmax          # Smallest wavelength

# Output model parameter:
print("Model size: x:",dx*nx,"in m, y:",dy*ny,"in m")
print("Temporal discretization: ",dt," s")
print("Spatial discretization: ",dx," m")
print("Number of gridpoints per minimum wavelength: ",lampda_min/dx)


I am working on the file "JupyterNotebook/2D/FD_2D_DX4_DT2.ipynb".
The first task is:
Save seismograms
Can you write Python code for it?

## Save seismograms
np.save("Seismograms/FD_2D_DX4_DT2",Seismogramm)


I am working on the file "JupyterNotebook/2D/FD_2D_DX4_DT2.ipynb".
The first task is:
Source signal - Ricker-wavelet
Can you write Python code for it?

tau=np.pi*f0*(t-1.5/f0)
q=q0*(1.0-2.0*tau**2.0)*np.exp(-tau**2)

# Plotting source signal
plt.figure(3)
plt.plot(t,q)
plt.title('Source signal Ricker-Wavelet')
plt.ylabel('Amplitude')
plt.xlabel('Time in s')
plt.draw()


I am working on the file "JupyterNotebook/2D/FD_2D_DX4_DT2.ipynb".
The first task is:
Time stepping
Can you write Python code for it?

# Init Seismograms
Seismogramm=np.zeros((3,nt)); # Three seismograms

# Calculation of some coefficients
i_dx=1.0/(dx)
i_dy=1.0/(dy)
c1=9.0/(8.0*dx)
c2=1.0/(24.0*dx)
c3=9.0/(8.0*dy)
c4=1.0/(24.0*dy)
c5=1.0/np.power(dx,3)
c6=1.0/np.power(dy,3)
c7=1.0/np.power(dx,2)
c8=1.0/np.power(dy,2)
c9=np.power(dt,3)/24.0

## Time stepping
print("Starting time stepping...")
for n in range(2,nt):

        # Update velocity
        for kx in range(5,nx-4):
            for ky in range(5,ny-4):

                p_x=c1*(p[ky,kx+1]-p[ky,kx])-c2*(p[ky,kx+2]-p[ky,kx-1])
                p_y=c3*(p[ky+1,kx]-p[ky,kx])-c4*(p[ky+2,kx]-p[ky-1,kx])

                vx[ky,kx]=vx[ky,kx]-dt/rho[ky,kx]*p_x
                vy[ky,kx]=vy[ky,kx]-dt/rho[ky,kx]*p_y

        # Inject source wavelet
        p[yscr,xscr]=p[yscr,xscr]+q[n]

        # Update pressure
        for kx in range(5,nx-4):
            for ky in range(5,ny-4):

                vx_x=c1*(vx[ky,kx]-vx[ky,kx-1])-c2*(vx[ky,kx+1]-vx[ky,kx-2])
                vy_y=c3*(vy[ky,kx]-vy[ky-1,kx])-c4*(vy[ky+1,kx]-vy[ky-2,kx])

                p[ky,kx]=p[ky,kx]-l[ky,kx]*dt*(vx_x+vy_y)

        # Save seismograms
        Seismogramm[0,n]=p[yrec1,xrec1]
        Seismogramm[1,n]=p[yrec2,xrec2]
        Seismogramm[2,n]=p[yrec3,xrec3]

print("Finished time stepping!")


I am working on the file "JupyterNotebook/2D/FD_2D_DX4_DT2_fast.ipynb".
The first task is:
Create space and time vector
Can you write Python code for it?

x=np.arange(0,dx*nx,dx) # Space vector in X
y=np.arange(0,dy*ny,dy) # Space vector in Y
t=np.arange(0,T,dt)     # Time vector
nt=np.size(t)           # Number of time steps

# Plotting model
fig, (ax1, ax2) = plt.subplots(1, 2)
fig.subplots_adjust(wspace=0.4,right=1.6)
ax1.plot(x,modell_v)
ax1.set_ylabel('VP in m/s')
ax1.set_xlabel('Depth in m')
ax1.set_title('P-wave velocity')

ax2.plot(x,rho)
ax2.set_ylabel('Density in g/cm^3')
ax2.set_xlabel('Depth in m')
ax2.set_title('Density');


I am working on the file "JupyterNotebook/2D/FD_2D_DX4_DT2_fast.ipynb".
The first task is:
Input Parameter
Can you write Python code for it?

# Discretization
c1=30   # Number of grid points per dominant wavelength
c2=0.2  # CFL-Number
nx=300 # Number of grid points in X
ny=300 # Number of grid points in Y
T=1     # Total propagation time

# Source Signal
f0= 5      # Center frequency Ricker-wavelet
q0= 100       # Maximum amplitude Ricker-Wavelet
xscr = 150  # Source position (in grid points) in X
yscr = 150  # Source position (in grid points) in Y

# Receiver
xrec1=150; yrec1=120;  # Position Reciever 1 (in grid points)
xrec2=150; yrec2=150;  # Position Reciever 2 (in grid points)
xrec3=150; yrec3=180;# Position Reciever 3 (in grid points)

# Velocity and density
modell_v = 3000*np.ones((ny,nx))
rho=2.2*np.ones((ny,nx))


I am working on the file "JupyterNotebook/2D/FD_2D_DX4_DT2_fast.ipynb".
The first task is:
Plotting
Can you write Python code for it?

## Image plot
fig, ax = plt.subplots(1,1)
img = ax.imshow(p);
ax.set_title('P-Wavefield')
ax.set_xticks(range(0,nx+1,int(nx/5)))
ax.set_yticks(range(0,ny+1,int(ny/5)))
ax.set_xlabel('Grid-points in X')
ax.set_ylabel('Grid-points in Y')
fig.colorbar(img)

## Plot seismograms
fig, (ax1, ax2, ax3) = plt.subplots(3, 1)
fig.subplots_adjust(hspace=0.4,right=1.6, top = 2 )

ax1.plot(t,Seismogramm[0,:])
ax1.set_title('Seismogram 1')
ax1.set_ylabel('Amplitude')
ax1.set_xlabel('Time in s')
ax1.set_xlim(0, T)

ax2.plot(t,Seismogramm[1,:])
ax2.set_title('Seismogram 2')
ax2.set_ylabel('Amplitude')
ax2.set_xlabel('Time in s')
ax2.set_xlim(0, T)

ax3.plot(t,Seismogramm[2,:])
ax3.set_title('Seismogram 3')
ax3.set_ylabel('Amplitude')
ax3.set_xlabel('Time in s')
ax3.set_xlim(0, T);


I am working on the file "JupyterNotebook/2D/FD_2D_DX4_DT2_fast.ipynb".
The first task is:
Preparation
Can you write Python code for it?

# Init wavefields
vx=np.zeros(shape = (ny,nx))
vy=np.zeros(shape = (ny,nx))
p=np.zeros(shape = (ny,nx))
vx_x=np.zeros(shape = (ny,nx))
vy_y=np.zeros(shape = (ny,nx))
p_x=np.zeros(shape = (ny,nx))
p_y=np.zeros(shape = (ny,nx))

# Calculate first Lame-Paramter
l=rho * modell_v * modell_v

cmin=min(modell_v.flatten())  # Lowest P-wave velocity
cmax=max(modell_v.flatten())  # Highest P-wave velocity
fmax=2*f0                     # Maximum frequency
dx=cmin/(fmax*c1)             # Spatial discretization (in m)
dy=dx                         # Spatial discretization (in m)
dt=dx/(cmax)*c2               # Temporal discretization (in s)
lampda_min=cmin/fmax          # Smallest wavelength

# Output model parameter:
print("Model size: x:",dx*nx,"in m, y:",dy*ny,"in m")
print("Temporal discretization: ",dt," s")
print("Spatial discretization: ",dx," m")
print("Number of gridpoints per minimum wavelength: ",lampda_min/dx)


I am working on the file "JupyterNotebook/2D/FD_2D_DX4_DT2_fast.ipynb".
The first task is:
Save seismograms
Can you write Python code for it?

## Save seismograms
np.save("Seismograms/FD_2D_DX4_DT2_fast",Seismogramm)


I am working on the file "JupyterNotebook/2D/FD_2D_DX4_DT2_fast.ipynb".
The first task is:
Source signal - Ricker-wavelet
Can you write Python code for it?


I am working on the file "JupyterNotebook/2D/FD_2D_DX4_DT2_fast.ipynb".
The first task is:
Time stepping
Can you write Python code for it?

# Init Seismograms
Seismogramm=np.zeros((3,nt)); # Three seismograms

# Calculation of some coefficients
i_dx=1.0/(dx)
i_dy=1.0/(dy)
c1=9.0/(8.0*dx)
c2=1.0/(24.0*dx)
c3=9.0/(8.0*dy)
c4=1.0/(24.0*dy)
c5=1.0/np.power(dx,3)
c6=1.0/np.power(dy,3)
c7=1.0/np.power(dx,2)
c8=1.0/np.power(dy,2)
c9=np.power(dt,3)/24.0

# Prepare slicing parameter:
kxM2=slice(5-2,nx-4-2)
kxM1=slice(5-1,nx-4-1)
kx=slice(5,nx-4)
kxP1=slice(5+1,nx-4+1)
kxP2=slice(5+2,nx-4+2)

kyM2=slice(5-2,ny-4-2)
kyM1=slice(5-1,ny-4-1)
ky=slice(5,ny-4)
kyP1=slice(5+1,ny-4+1)
kyP2=slice(5+2,ny-4+2)

## Time stepping
print("Starting time stepping...")
for n in range(2,nt):

        # Inject source wavelet
        p[yscr,xscr]=p[yscr,xscr]+q[n]

        # Update velocity
        p_x[ky,kx]=c1*(p[ky,kxP1]-p[ky,kx])-c2*(p[ky,kxP2]-p[ky,kxM1])
        p_y[ky,kx]=c3*(p[kyP1,kx]-p[ky,kx])-c4*(p[kyP2,kx]-p[kyM1,kx])

        vx=vx-dt/rho*p_x
        vy=vy-dt/rho*p_y

        # Update pressure
        vx_x[ky,kx]=c1*(vx[ky,kx]-vx[ky,kxM1])-c2*(vx[ky,kxP1]-vx[ky,kxM2])
        vy_y[ky,kx]=c3*(vy[ky,kx]-vy[kyM1,kx])-c4*(vy[kyP1,kx]-vy[kyM2,kx])

        p=p-l*dt*(vx_x+vy_y)

        # Save seismograms
        Seismogramm[0,n]=p[yrec1,xrec1]
        Seismogramm[1,n]=p[yrec2,xrec2]
        Seismogramm[2,n]=p[yrec3,xrec3]

print("Finished time stepping!")


I am working on the file "JupyterWorkflow.ipynb".
The first task is:
Important to note is that if we make a function which does a thing, it doesnt do that thing untill you actually call the function, calling a function is easy: just type its name and assign the function output to a variable name (which we creatively call data here.

If we would call the function without assigning it to a variable name, it will still spit out the table you see below, we just would not be able to do anything with the table, as it has no name we can call.
Can you write Python code for it?


I am working on the file "JupyterWorkflow.ipynb".
The first task is:
Let's reduce transparency to see better.
Can you write Python code for it?


I am working on the file "JupyterWorkflow.ipynb".
The first task is:
Let's see the whole data set in this way not just this average. We will do a pivot table.
Can you write Python code for it?

pivoted = data.pivot_table("Total", index=data.index.time, columns=data.index.date)
pivoted.iloc[:5, :5]


I am working on the file "JupyterWorkflow.ipynb".
The first task is:
So great! We have not imported our dataset and we know it exists because we can show it. To find out what is actually interesting about the cyclists, we can plot the weekly sum of cyclists in both directions, as well as the total number of cyclists.

Interesting to see is the wave pattern (sinusoid) to the amount of cyclists on a yearly basis. Apparently, there are a lot fewer people who take the bycicle in the winter than there are in the summer. This isnt that surprising.
Can you write Python code for it?


I am working on the file "JupyterWorkflow.ipynb".
The first task is:
The graph below is boring, just skip it
Can you write Python code for it?

ax = data.resample('D').sum().rolling(365).sum().plot();
ax.set_ylim(0,None)


I am working on the file "JupyterWorkflow.ipynb".
The first task is:
The graph below shows an interesting pattern which looks like the daily commute! The number of cyclists going west is much higher in the morning than in the late afternoon. The opposite is true for the number of cyclists going east.

Apparently, people live to the East of the bridge, and work to the West of the bridge!
Can you write Python code for it?


I am working on the file "JupyterWorkflow.ipynb".
The first task is:
There seems to be a offset between the left and right sidewalk. Let's plot them. See their trends.
Can you write Python code for it?

data["Total"] = data["West"] + data["East"]

ax = data.resample("D").sum().rolling(365).sum().plot()
ax.set_ylim(0, None)


I am working on the file "JupyterWorkflow.ipynb".
The first task is:
This last plot is shows a bit more information. The plot is a graph for every day over the past couple of years - number of cyclists per hour of every day. It again shows the 2 peaks - one in the morning and one in the late afternoon. Obviously, this represents the daily commute.

However, in between the peaks, we can see a wave occuring in the middle of the day. This is probably the people that are going out for a bike ride in the middle of the day during the weekends!
Can you write Python code for it?

pivoted.plot(legend=False, alpha = 0.02);


I am working on the file "JupyterWorkflow3.ipynb".
The first task is:
SECOND PART
To make a python package so we and other people can use it for analysis.
Go to the directory
mkdir jupyterworkflow       create a directory
touch jupyterworkflow/init.py      initialize a python package
create a data.py in this directory.
import os
from urllib.request import urlretrieve
import pandas as pd
FREMONT_URL = "https://data.seattle.gov/api/views/65db-xm6k/rows.csv?accessType=DOWNLOAD"
/# create a function to only dowload this data if we need to download it, first run..
def get_fremont_data(filename="Fremont.csv", url=FREMONT_URL, force_download=False):
    """Download and cache the fremont data
Parameters
----------
filename :string (optional)
    loation to save the data
url: string (optional)
    web location of the data
force_download: bool (optional)
    if True, force redownload of data

Returns
-------
data: pandas.DataFrame
    The fremont bridge data
"""
if force_download or not os.path.exists(filename):
    urlretrieve(url, filename)
data = pd.read_csv("Fremont.csv", index_col="Date", parse_dates=True)
data.columns = ["West", "East"]
data["Total"] = data["West"] + data["East"]
return data
Can you write Python code for it?


I am working on the file "JupyterWorkflow4.ipynb".
The first task is:
SECOND PART
To make a python package so we and other people can use it for analysis.
Go to the directory
mkdir jupyterworkflow       create a directory
touch jupyterworkflow/init.py      initialize a python package
create a data.py in this directory.
import os
from urllib.request import urlretrieve
import pandas as pd
FREMONT_URL = "https://data.seattle.gov/api/views/65db-xm6k/rows.csv?accessType=DOWNLOAD"
/# create a function to only dowload this data if we need to download it, first run..
def get_fremont_data(filename="Fremont.csv", url=FREMONT_URL, force_download=False):
    """Download and cache the fremont data
Parameters
----------
filename :string (optional)
    loation to save the data
url: string (optional)
    web location of the data
force_download: bool (optional)
    if True, force redownload of data

Returns
-------
data: pandas.DataFrame
    The fremont bridge data
"""
if force_download or not os.path.exists(filename):
    urlretrieve(url, filename)
data = pd.read_csv("Fremont.csv", index_col="Date", parse_dates=True)
data.columns = ["West", "East"]
data["Total"] = data["West"] + data["East"]
return data
Can you write Python code for it?


I am working on the file "Jupyter_notebook_introduction.ipynb".
The first task is:
Display of graphics
When displaying graphics, you should have them inline, meaning that they are displayed directly in the Jupyter notebook and not in a separate window.  You can specify that, as in the cell below, using the %matplotlib inline magic function.
Generally, I prefer presenting graphics as scalable vector graphics (SVG).  Vector graphics are infinitely zoom-able; i.e., the graphics are represented as points, lines, curves, etc., in space, not as a set of pixel values as is the case with raster graphics (such as PNG).  By default, graphics are displayed as PNGs, but you can specify SVG as I have at the top of this document in the first code cell.
%config InlineBackend.figure_formats = {'svg',}

If SVG graphics aren't working in your browser PNG graphics at a high resolution can be used instead
%config InlineBackend.figure_formats = {'png', 'retina'}
Can you write Python code for it?

%matplotlib inline

%config InlineBackend.figure_formats = {'svg',}
#%config InlineBackend.figure_formats = {'png', 'retina'}


I am working on the file "Jupyter_notebook_introduction.ipynb".
The first task is:
However, now if we type in the variable, its value will be displayed.
Can you write Python code for it?


I am working on the file "Jupyter_notebook_introduction.ipynb".
The first task is:
If you evaluate a Python expression that returns a value, that value is displayed as output of the code cell.  This only happens, however, for the last line of the code cell.
Can you write Python code for it?

# Would show 9 if this were the last line, but it is not, so shows nothing
4 + 5

# I hope we see 11.
5 + 6


I am working on the file "Jupyter_notebook_introduction.ipynb".
The first task is:
Import packages for scientific computing
One of the things that makes Python so powerful for science are the plethora of packages for scientific computing. However, the need to import these packages and understand what they are is also confusing to new users.
Usually at the top of a notebook, you should put in a code cell that imports all the modules we'll need. For this notebook, we will import numpy so that we can use its numerical structures, the scipy.integrate module and matplotlib.pyplot for plotting. I have used the import commands that are standard used in scientific python that imports abbreviations for these modules (e.g. numpy is imported as np).
Can you write Python code for it?

import numpy as np
import scipy.integrate
import matplotlib.pyplot as plt


I am working on the file "Jupyter_notebook_introduction.ipynb".
The first task is:
These data can then be plotted as below with axes then being labeled.
Can you write Python code for it?

plt.plot(x, y)
plt.xlim((0, 2 * np.pi))
plt.xlabel('x')
plt.ylabel('sin(x)')
plt.title('Example plot')
plt.show()


I am working on the file "Jupyter_notebook_introduction.ipynb".
The first task is:
With this function in hand, we can now pick our initial conditions and time points, run the numerical integration, and then plot the result.
Can you write Python code for it?

# Parameters to use
p = np.array([10.0, 28.0, 8.0 / 3.0])

# Initial condition
r0 = np.array([0.1, 0.0, 0.0])

# Time points to sample
t = np.linspace(0.0, 80.0, 10000)

# Use scipy.integrate.odeint to integrate Lorentz attractor
r = scipy.integrate.odeint(lorenz_attractor, r0, t, args=(p,))

# Unpack results into x, y, z.
x, y, z = r.transpose()

# Plot the result
plt.plot(x, z, '-', linewidth=0.5)
plt.xlabel(r'$x(t)$', fontsize=18)
plt.ylabel(r'$z(t)$', fontsize=18)
plt.title(r'$x$-$z$ proj. of Lorenz attractor traj.')
plt.show()


I am working on the file "Jupyter_notebooks/Ising model simulations.ipynb".
The first task is:
2)
Can you write Python code for it?

#!/usr/bin/env python
"""
Monte Carlo simulation of the 2D Ising model
"""

from scipy import *
from scipy import weave
from pylab import *

Nitt = 1000000  # total number of Monte Carlo steps
N = 10          # linear dimension of the lattice, lattice-size= N x N
warm = 1000     # Number of warmup steps
measure=100     # How often to take a measurement

def CEnergy(latt):
    "Energy of a 2D Ising lattice at particular configuration"
    Ene = 0
    for i in range(len(latt)):
        for j in range(len(latt)):
            S = latt[i,j]
            WF = latt[(i+1)%N, j] + latt[i,(j+1)%N] + latt[(i-1)%N,j] + latt[i,(j-1)%N]
            Ene += -WF*S # Each neighbor gives energy 1.0
    return Ene/2. # Each par counted twice

def RandomL(N):
    "Radom lattice, corresponding to infinite temerature"
    latt = zeros((N,N), dtype=int)
    for i in range(N):
        for j in range(N):
            latt[i,j] = sign(2*rand()-1)
    return latt

def SamplePython(Nitt, latt, PW):
    "Monte Carlo sampling for the Ising model in Pythons"
    Ene = CEnergy(latt)  # Starting energy
    Mn=sum(latt)         # Starting magnetization

    Naver=0       # Measurements
    Eaver=0.0
    Maver=0.0

    N2 = N*N
    for itt in range(Nitt):
        t = int(rand()*N2)
        (i,j) = (t % N, t/N)
        S = latt[i,j]
        WF = latt[(i+1)%N, j] + latt[i,(j+1)%N] + latt[(i-1)%N,j] + latt[i,(j-1)%N]
        P = PW[4+S*WF]
        if P>rand(): # flip the spin
            latt[i,j] = -S
            Ene += 2*S*WF
            Mn -= 2*S

        if itt>warm and itt%measure==0:
            Naver += 1
            Eaver += Ene
            Maver += Mn

    return (Maver/Naver, Eaver/Naver)

def SampleCPP(Nitt, latt, PW, T):
    "The same Monte Carlo sampling in C++"
    Ene = float(CEnergy(latt))  # Starting energy
    Mn = float(sum(latt))       # Starting magnetization

    # Measurements
    aver = zeros(5,dtype=float) # contains: [Naver, Eaver, Maver]

    code="""
    using namespace std;
    int N2 = N*N;
   ...


I am working on the file "Jupyter_notebooks/Ising model simulations.ipynb".
The first task is:
Ising model simulations
From:
* 1) Monte Carlo Simulation of the Ising Model using Python: Explanations, code and video
* 2) Full code with T infinite.
* 3) VPython code with explanations
* 3) Slides based on Newman book
1)  Monte Carlo Simulation of the Ising Model using Python: Explanations, code and video
Can you write Python code for it?


I am working on the file "Jupyter_notebooks/Notebook_1_CNN_Model_Training_with_Hyperparameter_Search.ipynb".
The first task is:
Create a directory to save the model parameters:
Can you write Python code for it?

model_name = 'Random_UTR_CNN'
model_params_dir = '../Results/{0}.Hyperparam.Opt/'.format(model_name)

if not os.path.exists(model_params_dir):
    os.mkdir(model_params_dir)


I am working on the file "Jupyter_notebooks/Notebook_1_CNN_Model_Training_with_Hyperparameter_Search.ipynb".
The first task is:
Hyperparameter search
Before training the model, we perform a hyperparameter search to narrow down which model architecture to use. Of course, we do a fair amount of narrowing ourselves by selecting which architectures are available for the search.
The dictionary 'hyperparams' has the same values as the 'space' object, 'space' is just the form that's compatible with hyperopt.
Can you write Python code for it?

hyperparams = {'conv_width' : [9, 13, 17, 25],
               'conv_filters' : [32, 64, 128, 256],
               'conv_layers' : [2, 3, 4],
               'dense_layers' : [1, 2],
               'conv_dropout' : [None, 0.15],
               'dense_dropout' : [None, 0.1, 0.25, 0.5],
               'dense_units' : [32, 64, 128, 256]}

space = {   'conv_width': hp.choice('conv_width', [9, 13, 17, 25]),
            'conv_filters': hp.choice('conv_filters', [32, 64, 128, 256]),
            'conv_layers': hp.choice('conv_layers', [2, 3, 4]),
            'dense_layers': hp.choice('dense_layers', [1, 2]),
            'conv_dropout': hp.choice('conv_dropout',  [None, 0.15]),
            'dense_dropout': hp.choice('dense_dropout', [None, 0.1, 0.25, 0.5]),
            'dense_units': hp.choice('dense_units', [32, 64, 128, 256]),
        }


I am working on the file "Jupyter_notebooks/Notebook_1_CNN_Model_Training_with_Hyperparameter_Search.ipynb".
The first task is:
Load our cleaned up data.
The csv should be tab-separated. The read counts are log2.
Can you write Python code for it?

data = pd.read_csv(data_dir + 'Random_UTRs.csv')


I am working on the file "Jupyter_notebooks/Notebook_1_CNN_Model_Training_with_Hyperparameter_Search.ipynb".
The first task is:
Make sure the model architecture and parameters are saved correctly
Can you write Python code for it?

!ls {model_params_dir}

model = keras.models.model_from_json(open(model_params_dir + 'model_arch.json').read())
model.load_weights(model_params_dir + 'model_weights.hdf5')
model.compile(loss='mean_squared_error', optimizer='adam')

Y_pred = model.predict(X, verbose=1)

# R^2 value for our predictions on the training set
print scipy.stats.pearsonr(Y[train_inds].flatten(),
                           Y_pred[train_inds].flatten())[0]**2

# and on the test set
print scipy.stats.pearsonr(Y[test_inds].flatten(),
                           Y_pred[test_inds].flatten())[0]**2


I am working on the file "Jupyter_notebooks/Notebook_1_CNN_Model_Training_with_Hyperparameter_Search.ipynb".
The first task is:
One-hot encoding of the sequences.
i.e. we're converting the sequences from being represented as a 50 character string of bases to a 4x50 matrix of 1's and 0's, with each row corresponding to a base and every column a position in the UTR.
Can you write Python code for it?

# one hot encoding of UTRs
# X = one hot encoding matrix
# Y = growth rates

def one_hot_encoding(df, seq_column, expression):

    bases = ['A','C','G','T']
    base_dict = dict(zip(bases,range(4))) # {'A' : 0, 'C' : 1, 'G' : 2, 'T' : 3}

    n = len(df)

    # length of the UTR sequence
    # we also add 10 empty spaces to either side
    total_width = df[seq_column].str.len().max() + 20

    # initialize an empty numpy ndarray of the appropriate size
    X = np.zeros((n, 1, 4, total_width))

    # an array with the sequences that we will one-hot encode
    seqs = df[seq_column].values

    # loop through the array of sequences to create an array that keras will actually read
    for i in range(n):
        seq = seqs[i]

        # loop through each individual sequence, from the 5' to 3' end
        for b in range(len(seq)):
            # this will assign a 1 to the appropriate base and position for this UTR sequence
            X[i, 0, base_dict[seq[b]], int(b + round((total_width - len(seq))/2.))] = 1.

        # keep track of where we are
        if (i%10000)==0:
            print i,

    X = X.astype(theano.config.floatX)
    Y = np.asarray(df[expression].values,
                   dtype = theano.config.floatX)[:, np.newaxis]

    return X, Y, total_width

X, Y, total_width = one_hot_encoding(data, 'UTR', 'growth_rate')


I am working on the file "Jupyter_notebooks/Notebook_1_CNN_Model_Training_with_Hyperparameter_Search.ipynb".
The first task is:
Save the model
Can you write Python code for it?

json_string = model.to_json()
open(model_params_dir + 'model_arch.json', 'w').write(json_string)
model.save_weights(model_params_dir + 'model_weights.h5')


I am working on the file "Jupyter_notebooks/Notebook_1_CNN_Model_Training_with_Hyperparameter_Search.ipynb".
The first task is:
Set plotting style
Can you write Python code for it?

plt.rcParams["patch.force_edgecolor"] = True
sns.set_style('whitegrid',
              {'axes.grid': True,
               'grid.linestyle': u'--',
               'axes.edgecolor': '0.1',
               'axes.labelcolor': '0',
               'axes.labelsize': 15,
               'axes.titlesize': 15,
               'legend.fontsize': 15,
               'xtick.labelsize': 15,
               'ytick.labelsize': 15,
               })


I am working on the file "Jupyter_notebooks/Notebook_1_CNN_Model_Training_with_Hyperparameter_Search.ipynb".
The first task is:
Split the data into test and training sets
If we have more reads for a given UTR at the outset, we can be more confident that we have made an accurate measurement. For this reason, we use those UTRs with the most reads to test our model on, because these should have the least experimental noise. We hold out the UTRs that fall in top 5% of reads at the first time point as our test set.
Can you write Python code for it?

# a sorted numpy array of UTR indexes, from least reads to most reads
sorted_inds = data.sort_values('t0').index.values

train_inds = sorted_inds[:int(0.95*len(sorted_inds))] # 95% of the data as the training set

test_inds = sorted_inds[int(0.95*len(sorted_inds)):] # UTRs with most reads at time point 0 as the test set

# set the seed before randomly shuffling the data
seed = 0.5
random.shuffle(train_inds, lambda :seed)


I am working on the file "Jupyter_notebooks/Notebook_1_CNN_Model_Training_with_Hyperparameter_Search.ipynb".
The first task is:
We also need a function that builds and fits the model, which we can pass to the hyperparameter search.

It also returns some information regarding overfitting, etc.
note that the data is included inside the function, I'm not passing it to the function.
Can you write Python code for it?

def f_nn(params):

    model = create_model(params)

    # track model overfitting
    earlyStopping = keras.callbacks.EarlyStopping(monitor = 'val_loss',
                                                  patience = 1,
                                                  verbose = 0,
                                                  mode = 'auto')
    history = keras.callbacks.History()

    # keep track of where we are while the code in this cell is running
    global n
    print "\n", n
    n+=1
    print params

    # fit the model
    # note that I'm not passing the data to this function, I've just included it here (i.e. I've
    # included X and Y)
    model.fit(X[train_inds],
              Y[train_inds],
              validation_split = 0.2,
              callbacks = [earlyStopping, history],
              verbose = 0,
              nb_epoch = 100)

    print 'MSE:',earlyStopping.best
    return {'loss': earlyStopping.best, 'status': STATUS_OK}


I am working on the file "Jupyter_notebooks/Notebook_1_CNN_Model_Training_with_Hyperparameter_Search.ipynb".
The first task is:
and take a look a the winning architecture
Can you write Python code for it?

opt_params = {}

for p in best:
    opt_params[p] = hyperparams[p][best[p]]

opt_params


I am working on the file "Jupyter_notebooks/Notebook_1_CNN_Model_Training_with_Hyperparameter_Search.ipynb".
The first task is:
pickle the model
Can you write Python code for it?

with open(model_params_dir + 'hyperparam_test.pkl', 'w') as f:
    pickle.dump(trials.trials, f)


I am working on the file "Jupyter_notebooks/Notebook_2_CNN_Predictions_of_Random_UTR_HIS3_data.ipynb".
The first task is:
Load model trained on random 5' UTRs
Can you write Python code for it?

!ls {model_params_dir}

model = keras.models.model_from_json(open(model_params_dir + 'model_arch.json').read())
model.load_weights(model_params_dir + 'model_weights.hdf5')
model.compile(loss='mean_squared_error', optimizer='adam')

Y_pred = model.predict(X, verbose=1)


I am working on the file "Jupyter_notebooks/Notebook_2_CNN_Predictions_of_Random_UTR_HIS3_data.ipynb".
The first task is:
Load the cleaned up data.
The csv should be tab-separated. The read counts are log2.
Can you write Python code for it?

data_dir = '../Data/'

data = pd.read_csv(data_dir + 'Random_UTRs.csv', index_col = 0)


I am working on the file "Jupyter_notebooks/Notebook_2_CNN_Predictions_of_Random_UTR_HIS3_data.ipynb".
The first task is:
One-hot encoding of the sequences.
i.e. we're converting the sequences from being represented as a 50 character string of bases to a 4x50 matrix of 1's and 0's, with each row corresponding to a base and every column a position in the UTR.
Note that we're doing the indexing a little differently than in Notebook 1 -- see comments in function
Can you write Python code for it?

# one hot encoding of UTRs
# X = one hot encoding matrix
# Y = growth rates

def one_hot_encoding(df, seq_column, expression):

    bases = ['A','C','G','T']
    base_dict = dict(zip(bases,range(4))) # {'A' : 0, 'C' : 1, 'G' : 2, 'T' : 3}

    n = len(df)

    # length of the UTR sequence
    # we also add 10 empty spaces to either side
    total_width = df[seq_column].str.len().max() + 20

    # initialize an empty numpy ndarray of the appropriate size
    X = np.zeros((n, 1, 4, total_width))

    # an array with the sequences that we will one-hot encode
    seqs = df[seq_column].values

    # loop through the array of sequences to create an array that keras will actually read
    for i in range(n):
        seq = seqs[i]

        # loop through each individual sequence, from the 5' to 3' end
        for b in range(len(seq)):
            # this will assign a 1 to the appropriate base and position for this UTR sequence
            # Note that this looks different than the same function in Notebook #1, but it does
            # the same thing.
            X[i, 0, base_dict[seq[b]], b + 10 + 50 - len(seq)] = 1.

        # keep track of where we are
        if (i%10000)==0:
            print i,

    X = X.astype(theano.config.floatX)
    Y = np.asarray(df[expression].values,
                   dtype = theano.config.floatX)[:, np.newaxis]

    return X, Y, total_width

X, Y, total_width = one_hot_encoding(data, 'UTR', 'growth_rate')


I am working on the file "Jupyter_notebooks/Notebook_2_CNN_Predictions_of_Random_UTR_HIS3_data.ipynb".
The first task is:
Plot results
Can you write Python code for it?

# data
x = Y_pred[test_inds].flatten()
y = Y[test_inds].flatten()

# calculate R^2
r2 = scipy.stats.pearsonr(x, y)[0]**2

g = sns.jointplot(x,
                  y,
                  stat_func = None,
                  kind = 'scatter',
                  s = 5,
                  alpha = 0.25,
                  size = 5)

g.ax_joint.set_xlabel('Predicted log$_2$ Growth Rate')
g.ax_joint.set_ylabel('Measured log$_2$ Growth Rate')

text = "R$^2$ = {:0.2}".format(r2)
plt.annotate(text, xy=(-5.5, 0.95), xycoords='axes fraction')

plt.title("CNN predictions of Random 5' UTR HIS3 data", x = -3, y = 1.25)


I am working on the file "Jupyter_notebooks/Notebook_2_CNN_Predictions_of_Random_UTR_HIS3_data.ipynb".
The first task is:
Save data and predictions to csv
Can you write Python code for it?

data['pred_growth_rate'] = Y_pred

data.to_csv(results_dir + 'Random_UTRs_with_predictions.csv')


I am working on the file "Jupyter_notebooks/Notebook_2_CNN_Predictions_of_Random_UTR_HIS3_data.ipynb".
The first task is:
The directory that contains information about the model parameters
Can you write Python code for it?

model_name = 'Random_UTR_CNN'
model_params_dir = '../Results/{0}.Hyperparam.Opt/'.format(model_name)


I am working on the file "Jupyter_notebooks/Notebook_3_CNN_Predictions_of_Native_UTR_HIS3_data.ipynb".
The first task is:
Create a directory to save results:
(if it doesn't already exist)
Can you write Python code for it?

results_dir = '../Results/{0}.ModelPredictions/'.format(model_name)

if not os.path.exists(model_params_dir):
    os.mkdir(model_params_dir)


I am working on the file "Jupyter_notebooks/Notebook_3_CNN_Predictions_of_Native_UTR_HIS3_data.ipynb".
The first task is:
Load model trained on random 5' UTRs
Can you write Python code for it?


I am working on the file "Jupyter_notebooks/Notebook_3_CNN_Predictions_of_Native_UTR_HIS3_data.ipynb".
The first task is:
Load the cleaned up data.
The csv should be tab-separated. The read counts are log2.
Can you write Python code for it?

data_dir = '../Data/'

native_data = pd.read_csv(data_dir + 'Native_UTRs.csv', index_col = 0)


I am working on the file "Jupyter_notebooks/Notebook_3_CNN_Predictions_of_Native_UTR_HIS3_data.ipynb".
The first task is:
One-hot encoding of the sequences.
i.e. we're converting the sequences from being represented as a 50 character string of bases to a 4x50 matrix of 1's and 0's, with each row corresponding to a base and every column a position in the UTR.
Note that we're doing the indexing a little differently than in Notebook 1 -- see comments in function
Can you write Python code for it?

# one hot encoding of UTRs
# X = one hot encoding matrix
# Y = growth rates

def one_hot_encoding(df, seq_column, expression):

    bases = ['A','C','G','T']
    base_dict = dict(zip(bases,range(4))) # {'A' : 0, 'C' : 1, 'G' : 2, 'T' : 3}

    n = len(df)

    # length of the UTR sequence
    # we also add 10 empty spaces to either side
    total_width = df[seq_column].str.len().max() + 20

    # initialize an empty numpy ndarray of the appropriate size
    X = np.zeros((n, 1, 4, total_width))

    # an array with the sequences that we will one-hot encode
    seqs = df[seq_column].values

    # loop through the array of sequences to create an array that keras will actually read
    for i in range(n):
        seq = seqs[i]

        # loop through each individual sequence, from the 5' to 3' end
        for b in range(len(seq)):
            # this will assign a 1 to the appropriate base and position for this UTR sequence
            # Note that this is different than the same function in Notebook #1 (since we're dealing
            # with sequences with nonuniform lengths)
            X[i, 0, base_dict[seq[b]], b + 10 + 50 - len(seq)] = 1.

        # keep track of where we are
        if (i%10000)==0:
            print i,

    X = X.astype(theano.config.floatX)
    Y = np.asarray(df[expression].values,
                   dtype = theano.config.floatX)[:, np.newaxis]

    return X, Y, total_width

X, Y, total_width = one_hot_encoding(native_data, 'UTR', 'growth_rate')


I am working on the file "Jupyter_notebooks/Notebook_3_CNN_Predictions_of_Native_UTR_HIS3_data.ipynb".
The first task is:
Plot results
Can you write Python code for it?

# data
x = Y_pred[test_inds].flatten()
y = Y[test_inds].flatten()

# calculate R^2
r2 = scipy.stats.pearsonr(x, y)[0]**2

g = sns.jointplot(x,
                  y,
                  stat_func = None,
                  kind = 'scatter',
                  s = 5,
                  alpha = 0.25,
                  size = 5)

g.ax_joint.set_xlabel('Predicted log$_2$ Growth Rate')
g.ax_joint.set_ylabel('Measured log$_2$ Growth Rate')

text = "R$^2$ = {:0.2}".format(r2)
plt.annotate(text, xy=(-5.5, 0.95), xycoords='axes fraction')

plt.title("CNN predictions of native 5' UTR HIS3 data", x = -3, y = 1.25)


I am working on the file "Jupyter_notebooks/Notebook_3_CNN_Predictions_of_Native_UTR_HIS3_data.ipynb".
The first task is:
Record indexes for UTRs with >100 reads in the input
If we have more reads for a given UTR at the outset, we can be more confident that we have made an accurate measurement. For this reason, we use those UTRs with the most reads to test our model on, because these should have the least experimental noise.
Can you write Python code for it?

# a numpy array of the indexes of UTRs with > 100 reads
test_inds = native_data.loc[native_data.t0 > 100].index.values


I am working on the file "Jupyter_notebooks/Notebook_4_Generating_Model_Directed_Evolution_of_UTRs_From_100_Native_and_Random_UTRs.ipynb".
The first task is:
As with the native library, we just want the UTRs that we're most confident about
Can you write Python code for it?

random_data = random_data[random_data.t0 > 100]

# define the range of growth rates
desired_growth_rates = np.linspace(random_data.growth_rate.min(),
                                   random_data.growth_rate.max(),
                                   100) # pay attention to this last number, we're choosing it just based on how
                                        # many sequences we get out at the end. As you may have noticed, we chose
                                        # 106 in the case of the native UTRs

selected_random_UTRs = []

for i in range(len(desired_growth_rates)):

    # with each step we select the UTR with the growth rate closest to the desired growth rate
    selected_random_UTRs.append(abs(random_data.growth_rate - desired_growth_rates[i]).argmin())

# just take the unique ones
selected_random_UTRs = pd.Series(np.unique(selected_random_UTRs))

selected_random_UTRs_data = random_data.loc[selected_random_UTRs]

# make an array of the UTRs that we want to evolve
seqs = selected_random_UTRs_data.UTR.values

# run them through our evolve_seqs function for 40 rounds
mut_random_seqs = evolve_seqs(seqs, rounds = 40)


I am working on the file "Jupyter_notebooks/Notebook_4_Generating_Model_Directed_Evolution_of_UTRs_From_100_Native_and_Random_UTRs.ipynb".
The first task is:
Create directories to save results:
(if it doesn't already exist)
Can you write Python code for it?

results_dir = '../Results/EvolvedUTRs/'

if not os.path.exists(results_dir):
    os.mkdir(results_dir)

fig_dir = '../Figures/EvolvedUTRs/'

if not os.path.exists(fig_dir):
    os.mkdir(fig_dir)


I am working on the file "Jupyter_notebooks/Notebook_4_Generating_Model_Directed_Evolution_of_UTRs_From_100_Native_and_Random_UTRs.ipynb".
The first task is:
Evolve sequences:
Here we make all possible mutations to a given sequence, ask the model to score each, and take the best one. We do this for 40 rounds.
Can you write Python code for it?

def evolve_seqs(seqs, rounds = 5):
    all_seqs = {}
    all_scores = {}
    all_seqs[0] = seqs
    all_scores[0] = model.predict(one_hot_encoding(seqs)).flatten()

    # march through every round of 'evolution'
    for i in range(rounds):
        Y_mut = {}

        # loop through the four bases
        for base in bases:

            # we'll try each base at each position of the sequences
            for position in range(50):
                mut_seqs = make_mutation(seqs, position, base)                      # make mutated sequence
                X_mut = one_hot_encoding(mut_seqs)                                  # one-hot-encoding of mutated sequence
                Y_mut[base + '_' + str(position)] = model.predict(X_mut).flatten()  # ask the model what it thinks

        # Note which mutations are the best performing for each UTR
        best_muts = pd.DataFrame(Y_mut).apply(np.argmax, axis = 1).values

        # Remake these mutations (an example value in best_muts looks would be "T_30", which tells us the nucleotide
        # and the position)
        seqs = [s[:int(m[2:])] + m[:1] + s[int(m[2:]) + 1:] for s, m in zip(seqs, best_muts)]

        all_seqs[i + 1] = seqs
        all_scores[i + 1] = model.predict(one_hot_encoding(seqs)).flatten()

        # keep track of where we are
        print i, pd.DataFrame(Y_mut).max(1).mean()

    return pd.DataFrame(all_seqs), pd.DataFrame(all_scores)


I am working on the file "Jupyter_notebooks/Notebook_4_Generating_Model_Directed_Evolution_of_UTRs_From_100_Native_and_Random_UTRs.ipynb".
The first task is:
First, evolve native UTR sequences:
Load sequences
Can you write Python code for it?

native_data = pd.read_csv('../Data/Native_UTRs.csv', index_col = 0)