I am working on the file "Kalman1.ipynb".
The first task is:
Kibővített Kálmán-szűrő (Extended Kalman Filter, EKF)
A kibővített szűrő a nemlineáris mérési egyenlet Jacobi mátrixának felírásával valósítható meg. A mérési egyenlet $H$ Jacobi mátrixa
$$ H = \begin{bmatrix}
-\frac{g}{r_w}\cos\left(\frac{p}{r_w}\right)-\frac{\ddot{p}}{r_w}\sin\left(\frac{p}{r_w}\right) & 0 & \cos\left(\frac{p}{r_w}\right)-\frac{r_s}{r_w}\
\frac{g}{r_w}\sin\left(\frac{p}{r_w}\right)-\frac{\ddot{p}}{r_w}\cos\left(\frac{p}{r_w}\right) & -2\frac{r_s}{r_w^2}\dot{p} & -\sin\left(\frac{p}{r_w}\right)
\end{bmatrix} .$$
A sztochasztikus modellhez felvett állapot zaj variancia $q=0.07 m/s^2$, a mért gyorsulások hiba varianciája $r_1=r_2=5 m/s^2$.
Az alábbi Python függvény számítja ki a $H$ mátrixot:
Can you write Python code for it?

def H(x,rs,rw):
    ## a mérési egyenlet függvényének Jacobi-mátrixa
    ## x  = állapot vektor (p,pdot,pdotdot)
    ## rs = szenzor tengelytől mért távolsága
    ## rw = kerék sugara
    g = 9.81
    H = np.zeros((2,3))
    H[0,0] = -g/rw*np.cos(x[0]/rw) - x[2]/rw*np.sin(x[0]/rw)
    H[0,1] = 0.0
    H[0,2] = np.cos(x[0]/rw)- rs/rw
    H[1,0] = g/rw*np.sin(x[0]/rw) - x[2]/rw*np.cos(x[0]/rw)
    H[1,1] = -2*x[1]*rs/(rw**2)
    H[1,2] = -np.sin(x[0]/rw)
    return H


I am working on the file "Kalman1.ipynb".
The first task is:
Teszt adatok
A kibővített Kálmán-szűrést Gersdorf B., Frese U. (2013) cikkéből vett gyorsulás adatokkal teszteljük.
Can you write Python code for it?

# -*- coding:utf-8 -*-
import numpy as np
import matplotlib.pyplot as plt

data = np.loadtxt("./dat/a.txt")
t = data[:,0]
a1= data[:,1]
a2= data[:,2]


I am working on the file "Kalman1.ipynb".
The first task is:
Végül kirajzoltatjuk a kibővített Kálmán-szűrővel kapott eredményeket:
Can you write Python code for it?

%matplotlib inline
plt.plot(xe[:,0],xe[:,1], 'b-', label=u'EKF távolság, m')
plt.plot(xe[:,0],xe[:,2], 'g-', label=u'EKF sebesség, m/s')
plt.plot(xe[:,0],xe[:,3], 'r-', label=u'EKF gyorsulás, m/s^2')
plt.plot(xe[:,0],xe[:,4], color='orange', label=u'elfordulás szöge, rad')
plt.xlim(1,11)
plt.grid(color='grey')
plt.title(u'Kibövített Kálmán-szűrő (EKF) eredmények', fontweight='bold')
plt.xlabel(u'idő (s)')
plt.legend(loc='upper left', shadow=False, prop={'size':8})
plt.show()


I am working on the file "KalmanFilter.ipynb".
The first task is:
Below is a not very efficient and numerically unstable but nevertheless a readable implementation of the Kalman filter for your reference. For or small model sizes $M, N, T$ the results would give a base to test against. Cross check with your implementation that your numerical results are about the same.
Can you write Python code for it?

m_pred = mu
Sig_pred = P
l = 0

for t in range(T):
    S = C*Sig_pred*C.T + R
    G = Sig_pred*C.T*(S.I)
    e = y[:,t] - C*m_pred
    m = m_pred + G*e
    Sig = Sig_pred - G*C*Sig_pred
    l = l - 0.5*np.log(np.linalg.det(2*np.pi*S)) - 0.5*e.T*S.I*e

    print('m_{}'.format(t+1), '=\n', m)
    print('Sig_{}'.format(t+1), '=\n',Sig)
    print('l_{}'.format(t+1), '=\n',l)
    print('\n')

    Sig_pred = A*Sig*A.T + Q
    m_pred = A*m


I am working on the file "KalmanFilter.ipynb".
The first task is:
For $i=1 \dots M$ (InnerFor)
$$
S_{t,i}  =  c_i W_{i-1} c_i^\top + r_i \
$$
$$
G_{t,i}  =  \frac{1}{S_{t,i}} W_{i-1} c_i^\top  \
$$
$$
e_{t,i}  =  y_t[m] - c_i v_{i-1} \
$$
$$
v_i  =  v_{i-1} + G_{t,i} e_{t,i} \
$$
$$
W_i  =  W_{i-1} - G_{t,i} c_i W_{i-1} \
$$
$$
\lambda_i =  \lambda_{i-1} - \frac{1}{2}\log (2\pi S_{t,i}) - \frac{e_{t,i}^2}{2 S_{t,i}}
$$
EndInnerfor
Can you write Python code for it?

m_pred = mu
Sig_pred = P
l = 0

for t in range(T):
    W = Sig_pred
    v = m_pred
    lam = l
    for i in range(M):
        S = C[i,:]*W*C[i,:].T + R[i,i]
        G = W*C[i,:].T/S
        e = y[i,t] - C[i,:]*v
        v = v + G*e
        W = W - G*C[i,:]*W
        lam = lam - 0.5*np.log(2*np.pi*S) - e*e/(2*S)
    Sig = W
    m = v
    l = lam

    print('m_{}'.format(t+1), '=\n', m)
    print('Sig_{}'.format(t+1), '=\n',Sig)
    print('l_{}'.format(t+1), '=\n',l)
    print('\n')

    Sig_pred = A*Sig*A.T + Q
    m_pred = A*m

%connect_info

%matplotlib inline
import numpy as np
import matplotlib.pylab as plt

A = 0.9
C = 1.0

R = 0.01
Q = 0.05

x0 = 0

T = 100

x = np.zeros((T))
y = np.zeros((T))

for t in range(T):
    if t==0:
        x[0] = x0
    else:
        x[t] = A*x[t-1] + np.sqrt(Q)*np.random.randn(1)

    y[t] = C*x[t] + np.sqrt(R)*np.random.randn(1)

plt.plot(y,'r')
plt.plot(x,'b')

m = np.zeros_like(x)
Sig = np.zeros_like(x)

m0 = 0
Sg0 = 1

for t in range(T):
    if t==0:
        m_pred = A*m0
        SG_pred = A*Sg0*A + Q
    else:
        m_pred = A*m[t-1]
        SG_pred = A*Sig[t-1]*A + Q

    Gt = SG_pred*C/(C*SG_pred*C+R)
    m[t] = m_pred + Gt*(y[t]-C*m_pred)
    Sig[t] = SG_pred - Gt*C*SG_pred

plt.plot(x,'.r')
plt.plot(m,'ob')


I am working on the file "KalmanFilter.ipynb".
The first task is:
Your Task:
The loglikelihood computation seems to require the inversion of a large matrix and the computation of its determinant. We can circumvent this by
a simple trick that works when the covariance matrix R is diagonal: we update the state by processing each entry of the observation incrementally.
We replace the update step with simpler operations where only scalar operations are needed.
Consider the observation matrix
\begin{eqnarray}
C & = & \left( \begin{array}{c} c_1 \ c_2 \ \vdots \ c_M \end{array} \right)
\end{eqnarray}
where $c$ are row vectors. For example, if
$$
C  =  \left( \begin{array}{ccc} 1 & 2 & 4 \ 3 & 0 & 3 \end{array} \right)
$$
$c_1 = (1,2,4) $ and $c_2 = (3,0,3)$.
Similarly, assume that $R$ is diagonal with
$$
R = \left( \begin{array}{cccc}
r_1 &     &  &  \
    & r_2 &  & \
    & & \ddots  & \
    &  &  & r_M
\end{array} \right)
$$
For example, if
$$
R  =  \left( \begin{array}{ccc} 1 & 0 \ 0 & 4 \end{array} \right)
$$
$r_1 = 1 $ and $r_2 = 4$.
The new algorithm is as follows
Kalman Filter with incremental updates
Initialization
The algorithm is initialized by
$$
m_{1|0}  = \mu
$$
$$
\Sigma_{1|0}  =  P $$
$$
l_0  =  0
$$
For $t = 1\dots T $ (OuterFor)
Update
Define the temporary covariance and mean
$$
W_0 = \Sigma_{t|t-1}
$$
$$
v_0 = m_{t|t-1}
$$
$$
\lambda_0 = l_{t-1}
$$
For $i=1 \dots M$ (InnerFor)
$$
S_{t,i}  =  c_i W_{i-1} c_i^\top + r_i \
$$
$$
G_{t,i}  =  \frac{1}{S_{t,i}} W_{i-1} c_i^\top  \
$$
$$
e_{t,i}  =  y_t[i] - c_i v_{i-1} \
$$
$$
v_i  =  v_{i-1} + G_{t,i} e_{t,i} \
$$
$$
W_i  =  W_{i-1} - G_{t,i} c_i W_{i-1} \
$$
$$
\lambda_i =  \lambda_{i-1} - \frac{1}{2}\log (2\pi S_{t,i}) - \frac{e_{t,i}^2}{2 S_{t,i}}
$$
EndInnerfor
\begin{eqnarray}
\Sigma_{t|t} & = & W_M \
m_{t|t} & = & v_M \
l_t & = & \lambda_M
\end{eqnarray}
Prediction
\begin{eqnarray}
m_{t+1|t} & = & A m_{t|t} \
\Sigma_{t+1|t} & = & A \Sigma_{t|t} A^\top + Q
\end{eqnarray}
EndOuterFor
Your task is implementing the Kalman filter with incremental updates as an object in C++.
All the parameters will be...
Can you write Python code for it?


I am working on the file "Keras - Multi-input and multi-output models.ipynb".
The first task is:
Here we insert the auxiliary loss, allowing the LSTM and Embedding layer to be trained smoothly even though the main loss will be much higher in the model.
Can you write Python code for it?

auxiliary_output = Dense(1, activation='sigmoid', name='aux_output')(lstm_out)

auxiliary_input = Input(shape=(5,), name='aux_input')
x = concatenate([lstm_out, auxiliary_input])

# We stack a deep densely-connected network on top
x = Dense(64, activation='relu')(x)
x = Dense(64, activation='relu')(x)
x = Dense(64, activation='relu')(x)

# And finally we add the main logistic regression layer
main_output = Dense(1, activation='sigmoid', name='main_output')(x)


I am working on the file "Keras - Multi-input and multi-output models.ipynb".
The first task is:
We can train the model by passing it lists of input arrays and target arrays:
Can you write Python code for it?

model.fit([headline_data, additional_data], [labels, labels],
          epochs=50, batch_size=32)


I am working on the file "Keras - Multi-input and multi-output models.ipynb".
The first task is:
We compile the model and assign a weight of 0.2 to the auxiliary loss. To specify different loss_weights or loss for each different output, you can use a list or a dictionary. Here we pass a single loss as the loss argument, so the same loss will be used on all outputs.
Can you write Python code for it?

model.compile(optimizer='rmsprop', loss='binary_crossentropy',
              loss_weights=[1., 0.2])


I am working on the file "Keras - Multi-input and multi-output models.ipynb".
The first task is:
We could also do:
Can you write Python code for it?

model.compile(optimizer='rmsprop',
              loss={'main_output': 'binary_crossentropy', 'aux_output': 'binary_crossentropy'},
              loss_weights={'main_output': 1., 'aux_output': 0.2})

# And trained it via:
model.fit({'main_input': headline_data, 'aux_input': additional_data},
          {'main_output': labels, 'aux_output': labels},
          epochs=50, batch_size=32)


I am working on the file "Keras as simplified TensorFlow.ipynb".
The first task is:
losses: the math and their implementation

trianing with tensorflow optimizer
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
mnist_data = input_data.read_data_sets('MNIST_data', one_hot=True)

train_step = tf.train.GradientDescentOptimizer(0.5).minimize(loss)

# Initialize all variables
init_op = tf.global_variables_initializer()
sess.run(init_op)

# Run training loop
with sess.as_default():
    for i in range(100):
        batch = mnist_data.train.next_batch(50)
        train_step.run(feed_dict={img: batch[0],
                                  labels: batch[1]})


I am working on the file "Keras_CNN_newsgroups_text_classification.ipynb".
The first task is:
data
Can you write Python code for it?

categories = ['alt.atheism', 'soc.religion.christian']

newsgroups_train = fetch_20newsgroups(subset='train',
                                      shuffle=True,
                                      categories=categories)

print(f'number of training samples: {len(newsgroups_train.data)}')

example_sample_data     = "\n".join(newsgroups_train.data[0].split("\n")[10:15])
example_sample_category = categories[newsgroups_train.target[0]]
print(f'\nexample training sample of category {example_sample_category}:'
      f'\n\n{example_sample_data}')


I am working on the file "Keras_CNN_newsgroups_text_classification.ipynb".
The first task is:
model: convolutional neural network
Can you write Python code for it?

embedding_layer = Embedding(len(word_index) + 1,
                            word_vector_dimensionality,
                            weights=[embedding_matrix],
                            input_length=max_sequence_length,
                            trainable=False)

inputs = Input(shape=(max_sequence_length,), dtype='int32') # inputs
x      = embedding_layer(inputs) # embedded sequences
x      = Conv1D(128, 5, activation='relu')(x)
x      = MaxPooling1D(5)(x)
x      = Conv1D(128, 5, activation='relu')(x)
x      = MaxPooling1D(5)(x)
x      = Conv1D(128, 5, activation='relu')(x)
x      = MaxPooling1D(35)(x) # global max pooling
x      = Flatten()(x)
x      = Dense(300, activation='relu')(x)
x      = Dropout(rate=0.5)(x)
preds  = Dense(2, activation='softmax', name='preds')(x)
model  = Model(input=inputs, output=preds)
model.compile(loss='categorical_crossentropy',
              optimizer='nadam',
              metrics=['acc'])
summary_and_diagram(model)

%%time
history = model.fit(x_train, y_train, validation_data=(x_val, y_val),
                    epochs=60, batch_size=32, verbose=False)

model_training_plot(history)
print(f'max. validation accuracy observed: {max(model.history.history["val_acc"])}')
print(f'max. validation accuracy history index: {model.history.history["val_acc"].index(max(model.history.history["val_acc"]))}')


I am working on the file "Keras_CNN_newsgroups_text_classification.ipynb".
The first task is:
model: convolutional neural network with multiple towers of varying kernel sizes
Can you write Python code for it?

embedding_layer = Embedding(len(word_index) + 1,
                            word_vector_dimensionality,
                            weights=[embedding_matrix],
                            input_length=max_sequence_length,
                            trainable=False)

inputs = Input(shape=(max_sequence_length,), dtype='int32')
x = embedding_layer(inputs)

convolutional_layer_towers = []
for kernel_size in [3, 4, 5]:
    _x = Conv1D(filters=128, kernel_size=kernel_size, activation='relu')(x)
    _x = Dropout(rate=0.1)(_x)
    _x = MaxPooling1D(5)(_x)
    convolutional_layer_towers.append(_x)
x     = Concatenate(axis=1)(convolutional_layer_towers)
x     = Conv1D(128, 5, activation='relu')(x)
x     = MaxPooling1D(5)(x)
x     = Conv1D(128, 5, activation='relu')(x)
x     = MaxPooling1D(30)(x)
x     = Flatten()(x)
x     = Dense(128, activation='relu')(x)
x     = Dropout(rate=0.5)(x)
preds = Dense(2, activation='softmax', name='preds')(x)
model = Model(input=inputs, output=preds)
model.compile(loss='categorical_crossentropy',
              optimizer='nadam',
              metrics=['acc'])
summary_and_diagram(model)

%%time
history = model.fit(x_train, y_train, validation_data=(x_val, y_val),
                    epochs=100, batch_size=32, verbose=False)

model_training_plot(history)
print(f'max. validation accuracy observed: {max(model.history.history["val_acc"])}')
print(f'max. validation accuracy history index: {model.history.history["val_acc"].index(max(model.history.history["val_acc"]))}')


I am working on the file "Keras_LSTM2.ipynb".
The first task is:
原始資料來源的 SQL，這是抽樣過的資料，當中也有一筆資料是修改過的，因為當天 Server 似乎出了一些問題，導至流量大幅下降
Can you write Python code for it?

sql = """
SELECT
date,count(distinct cookie_pta) as uv
from
TABLE_DATE_RANGE(pixinsight.article_visitor_log_1_100_, TIMESTAMP('2017-01-01'), CURRENT_TIMESTAMP())
where venue = 'pixnet'
group by date
order by date
"""

from os import environ
# load and plot dataset
import pandas as pd
from pandas import read_csv
from pandas import datetime
from matplotlib import pyplot
import matplotlib.dates as mdates

%matplotlib notebook

# %matplotlib inline

# load dataset
def parser(x):
    return datetime.strptime(x, '%Y%m%d')

series = pd.read_gbq(sql,project_id=environ['PROJECT_ID'], verbose=False, private_key=environ['GOOGLE_KEY'])#,header=0, parse_dates=[0], index_col='date', squeeze=True, date_parser=parser)
series['date'] = pd.to_datetime(series['date'],format='%Y%m%d')
series.index = series['date']
del series['date']

# summarize first few rows
print(series.head())


I am working on the file "Keras_LSTM2.ipynb".
The first task is:
最後30 筆資料不要看
Can you write Python code for it?

validation_size = 60

val_loss = []
loss = []
for _ in range(400):
    history = model.fit(trainX[:-1*validation_size],
              train_y[:-1*validation_size],
              epochs=1,shuffle=False,
              validation_data=(trainX[-1*validation_size:],
              train_y[-1*validation_size:]))

    loss.append(history.history['loss'])
    val_loss.append(history.history['val_loss'])
    model.reset_states()


I am working on the file "Keras_LSTM2.ipynb".
The first task is:
產生 x,y pair

舉列來說假設將 Step Size 設為 4 天，故一筆 Training Data ，為連續 4 天的流量。再來利用這４天的資料來預測第 5 天的流量
綠色的部是 Training Data(前4天的資料)，藍色的部份是需要被預測的部份。示意如下圖
<img align="left" width="50%" src="./imgs/sequence_uv.png" />
Can you write Python code for it?

#往回看 30 天前的每一筆資料
step_size = 15

print("原始資料長度:{}".format(x_scaled.shape))

def window_stack(a, stepsize=1, width=3):
    return np.hstack( a[i:1+i-width or None:stepsize] for i in range(0,width) )

import numpy as np
train_x = window_stack(x_scaled, stepsize=1, width=step_size)

# 最後一筆資料要放棄，因為沒有未來的答案作驗證

train_x = train_x[:-1]
train_x.shape

# 請注意千萬不將每一筆(Row) 當中的最後一天資料作為 Training Data 中的 Input Data
train_y = np.array([i for i in x_scaled[step_size:]])


I am working on the file "Keras_LSTM2.ipynb".
The first task is:
確認產出來的 Training Data 沒有包含到 Testing Data
Can you write Python code for it?

train_y.shape

train_x[0]

train_x[1]

train_y[0]


I am working on the file "Keras_LSTM2.ipynb".
The first task is:
進行 scale to 0-1 ，方便作為 input 及 output (因為 sigmoid 介於 0~1 之間)
Can you write Python code for it?

from sklearn.preprocessing import scale,MinMaxScaler
scaler = MinMaxScaler()

x = series.values

x = x.reshape([x.shape[0],1])

scaler.fit(x)

x_scaled = scaler.transform(x)

pyplot.figure()
pyplot.plot(x_scaled)
pyplot.show()


I am working on the file "Keras_convolutional_MNIST.ipynb".
The first task is:
2D class saliency map with uncertainties shown in marginal plots
Can you write Python code for it?

plt.rcParams["figure.figsize"] = [10, 10]

xi = 0.2;  yi = 0.2; wi = 0.7;  hi = 0.7 # image
xc = 0.91; yc = 0.2; wc = 0.05; hc = 0.7 # colorbar
xh = 0.2;  yh = 0.0; wh = 0.7;  hh = 0.2 # horizontal plot
xv = 0.0;  yv = 0.2; wv = 0.2;  hv = 0.7 # vertical plot

ax_i = plt.axes((xi, yi, wi, hi))
ax_h = plt.axes((xh, yh, wh, hh))
ax_v = plt.axes((xv, yv, wv, hv))
ax_c = plt.axes((xc, yc, wc, hc))

ax_i.xaxis.set_major_formatter(NullFormatter())
ax_i.yaxis.set_major_formatter(NullFormatter())
ax_h.yaxis.set_major_formatter(NullFormatter())
ax_v.xaxis.set_major_formatter(NullFormatter())

plt.axes(ax_i)
plt.imshow(mean, aspect='auto', cmap="jet")

ax_h.plot(list(range(0, 28)), mean.sum(axis=0),                                              '-k', drawstyle='steps')
ax_h.plot(list(range(0, 28)), mean.sum(axis=0) + np.sum(statistical_uncertainty, axis=0)/2,  '-k', drawstyle='steps', color='red')
ax_h.plot(list(range(0, 28)), mean.sum(axis=0) - np.sum(statistical_uncertainty, axis=0)/2,  '-k', drawstyle='steps', color='blue')

ax_v.plot(mean.sum(axis=1), list(range(0, 28)),                                              '-k', drawstyle='steps')
ax_v.plot(mean.sum(axis=1) + np.sum(statistical_uncertainty, axis=1)/2, list(range(0, 28)),  '-k', drawstyle='steps', color='red')
ax_v.plot(mean.sum(axis=1) - np.sum(statistical_uncertainty, axis=1)/2, list(range(0, 28)),  '-k', drawstyle='steps', color='blue')

cb = plt.colorbar(cax=ax_c)

#cb.set_label('intensity')
#ax_i.set_title('input')
#ax_h.set_xlabel('${x}$')
#ax_h.set_ylabel('intensity')
#ax_h.yaxis.set_label_position('right')
#ax_v.set_ylabel('${y}$')
#ax_v.set_xlabel('intensity')
#ax_v.xaxis.set_label_position('top')

plt.show();

plt.rcParams["figure.figsize"] = [10, 10]

xi = 0.2;  yi = 0.2; wi = 0.7;  hi = 0.7 # image
xc = 0.91; yc = 0.2; wc = 0.05; hc = 0.7 # colorbar
xh = 0.2;  yh = 0.0; wh = 0.7;  hh = 0.2 # horizontal plot
xv = 0.0;  yv = 0.2; wv = 0.2;  hv = 0.7 # vertical plot

ax_i = plt.axes((xi, yi, wi, hi))
ax_h = plt.axes((xh, yh, wh, hh))
ax_v =...


I am working on the file "Keras_convolutional_MNIST.ipynb".
The first task is:
class saliency map extraction
In the original definition of saliency, a class saliency map is computed essentially by taking the mean of ten input-specific saliency maps with the ten inputs selected randomly. This approach is used here but using a sample of 100 images for the mean.
Can you write Python code for it?

# get indices in test dataset of instances of the class 0
y_test_non_categorical = np.argmax(y_test, axis=1)
indices = [i for i, j in enumerate(y_test_non_categorical) if j == 0]
# get the instances
x_test_0 = [x_test[i] for i in indices]

sample_size = 100
saliencies = []
for i in list(range(0, sample_size - 1)):
    saliencies.append(visualize_saliency(model, layer_idx, filter_indices=filter_idx, seed_input=x_test_0[i], backprop_modifier='guided'))

im = plt.imshow(np.mean(saliencies, axis=0), cmap='jet')
plt.colorbar(im, fraction=0.0458, pad=0.04);


I am working on the file "Keras_convolutional_MNIST.ipynb".
The first task is:
class saliency map statistical uncertainties
Can you write Python code for it?

# https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.sem.html
im = plt.imshow(stats.sem(saliencies, axis=0), cmap='jet');
plt.colorbar(im, fraction=0.0458, pad=0.04);


I am working on the file "Keras_convolutional_MNIST.ipynb".
The first task is:
deep learning understanding
In deep learning research, researchers tend to focus on visualization of learned features of each neuron using various optimization-based algorithms. These optimization-based methods are currently divided into two categories: activation maximization and code inversion.
activation maximization
In a convolutional neural network, each convolution layer has several learned template matching filters that maximize their output when a similar template pattern is found in an input image. The first convolution layer is straightforward to visualise; simply visualise the weights as an image. To see what the layer is doing, a simple option is to apply the filter over raw input pixels. Higher convolution filters operate on the outputs of lower convolution filters (which indicate the presence or absense of some template patterns), making them harder to interpret.
The idea of activation maximization is to generate an input image that maximizes the filter output activations. This approach enables us to see wat sorts of input patterns activate a particular filter.

example
Can you write Python code for it?

layer_idx = utils.find_layer_idx(model, 'preds')
model.layers[layer_idx].activation = activations.linear # Swap softmax for linear.
model = utils.apply_modifications(model)

filter_idx = 0
img = visualize_activation(model, layer_idx, filter_indices=filter_idx, verbose=False)

im = plt.imshow(img[..., 0])
plt.colorbar(im, fraction=0.0458, pad=0.04);

layer_idx = utils.find_layer_idx(model, 'preds')
model.layers[layer_idx].activation = activations.linear # Swap softmax for linear.
model = utils.apply_modifications(model)

filter_idx = 4
img = visualize_activation(model, layer_idx, filter_indices=filter_idx, verbose=False)

im = plt.imshow(img[..., 0])
plt.colorbar(im, fraction=0.0458, pad=0.04);


I am working on the file "Keras_convolutional_MNIST.ipynb".
The first task is:
saliency

Deep Inside Convolutional Networks: VisualisingImage Classification Models and Saliency Maps (paper introducing the technique)
Beyond saliency: understanding convolutional neuralnetworks from saliency prediction on layer-wiserelevance propagation

Understanding how convolutional neural networks work is hard. There's a two-step understanding method called Salient Relevance (SR) map which aims to illuminate how deep convolutional neural networks recognise inputs and features from areas called attention areas therein. In this method, there is first a layer-wise relevance propagation (LRP) step which estimates a 'pixel'-wise relevance map over the input. Then there is constructed a context-aware saliency map from the LRP-generated map which predicts areas close to the foci of attention instead of isolated 'pixels' that the LRP reveals. This perhaps corresponds to recognition in the human visual system where information on regions is more important than information of pixels. So the original designers suggest that this saliency is something of a simulation of human visual recognision. The method seems to identify not only key pixels but also attention areas that contribute to the underlying neural network comprehension of inputs. So overall the salient relevance is a visual interface which unveils some of the visual attention of the network and reveals which type of objects the model has learned to recognize after training.
A saliency map highlights those input elements that are most important for classification of the input.
input-specific class saliency map
Given an input ${I_{o}}$ (e.g. an image), a class ${c}$ and a classification convolutional neural network with the class score function ${S_{c}\left(I\right)}$, which is computed by the classification layer of the network, features of input ${I_{o}}$ can be ranked by their influence on the class score ${S_{c}\left(I_{o}\right)}$ in order to create a saliency map. In this case the saliency map specific to a specific input case. One interpretation of...
Can you write Python code for it?

grads = visualize_saliency(model, layer_idx, filter_indices=filter_idx, seed_input=x_test[13], backprop_modifier='guided')
im = plt.imshow(grads, cmap='jet')
plt.colorbar(im, fraction=0.0458, pad=0.04);


I am working on the file "Keras_convolutional_MNIST_2.ipynb".
The first task is:
2D class saliency map with uncertainties shown in marginal plots
Can you write Python code for it?

plt.rcParams["figure.figsize"] = [10, 10]

xi = 0.2;  yi = 0.2; wi = 0.7;  hi = 0.7 # image
xc = 0.91; yc = 0.2; wc = 0.05; hc = 0.7 # colorbar
xh = 0.2;  yh = 0.0; wh = 0.7;  hh = 0.2 # horizontal plot
xv = 0.0;  yv = 0.2; wv = 0.2;  hv = 0.7 # vertical plot

ax_i = plt.axes((xi, yi, wi, hi))
ax_h = plt.axes((xh, yh, wh, hh))
ax_v = plt.axes((xv, yv, wv, hv))
ax_c = plt.axes((xc, yc, wc, hc))

ax_i.xaxis.set_major_formatter(NullFormatter())
ax_i.yaxis.set_major_formatter(NullFormatter())
ax_h.yaxis.set_major_formatter(NullFormatter())
ax_v.xaxis.set_major_formatter(NullFormatter())

plt.axes(ax_i)
plt.imshow(mean, aspect='auto', cmap="jet")

ax_h.plot(list(range(0, 28)), mean.sum(axis=0),                                                '-k', drawstyle='steps')
ax_h.plot(list(range(0, 28)), mean.sum(axis=0) + np.sum(statistical_uncertainty, axis=0) / 2,  '-k', drawstyle='steps', color='red')
ax_h.plot(list(range(0, 28)), mean.sum(axis=0) - np.sum(statistical_uncertainty, axis=0) / 2,  '-k', drawstyle='steps', color='blue')

ax_v.plot(mean.sum(axis=1), list(range(0, 28)),                                                '-k', drawstyle='steps')
ax_v.plot(mean.sum(axis=1) + np.sum(statistical_uncertainty, axis=1) / 2, list(range(0, 28)),  '-k', drawstyle='steps', color='red')
ax_v.plot(mean.sum(axis=1) - np.sum(statistical_uncertainty, axis=1) / 2, list(range(0, 28)),  '-k', drawstyle='steps', color='blue')

cb = plt.colorbar(cax=ax_c)

#cb.set_label('intensity')
#ax_i.set_title('input')
#ax_h.set_xlabel('${x}$')
#ax_h.set_ylabel('intensity')
#ax_h.yaxis.set_label_position('right')
#ax_v.set_ylabel('${y}$')
#ax_v.set_xlabel('intensity')
#ax_v.xaxis.set_label_position('top')

plt.show()

plt.rcParams["figure.figsize"] = [10, 10]

xi = 0.2;  yi = 0.2; wi = 0.7;  hi = 0.7 # image
xc = 0.91; yc = 0.2; wc = 0.05; hc = 0.7 # colorbar
xh = 0.2;  yh = 0.0; wh = 0.7;  hh = 0.2 # horizontal plot
xv = 0.0;  yv = 0.2; wv = 0.2;  hv = 0.7 # vertical plot

ax_i = plt.axes((xi, yi, wi, hi))
ax_h = plt.axes((xh, yh, wh, hh))
ax_v...


I am working on the file "Keras_convolutional_MNIST_2.ipynb".
The first task is:
class saliency map extraction
In the original definition of saliency, a class saliency map is computed essentially by taking the mean of ten input-specific saliency maps with the ten inputs selected randomly. This approach is used here but using a sample of 100 images for the mean.
Can you write Python code for it?

# get indices in test dataset of instances of the class 0
y_test_non_categorical = np.argmax(y_test, axis=1)
indices = [i for i, j in enumerate(y_test_non_categorical) if j == 0]
# get the instances
x_test_0 = [x_test[i] for i in indices]

sample_size = 100
saliencies = []
for i in list(range(0, sample_size - 1)):
    saliencies.append(visualize_saliency(model, layer_idx, filter_indices=filter_idx, seed_input=x_test_0[i], backprop_modifier='guided'))

plt.imshow(np.mean(saliencies, axis=0), cmap='jet');


I am working on the file "Keras_convolutional_MNIST_2.ipynb".
The first task is:
class saliency map statistical uncertainties
Can you write Python code for it?

# https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.sem.html
plt.imshow(stats.sem(saliencies, axis=0), cmap='jet');


I am working on the file "Keras_convolutional_MNIST_2.ipynb".
The first task is:
deep learning understanding
In deep learning research, researchers tend to focus on visualization of learned features of each neuron using various optimization-based algorithms. These optimization-based methods are currently divided into two categories: activation maximization and code inversion.
activation maximization
In a convolutional neural network, each convolution layer has several learned template matching filters that maximize their output when a similar template pattern is found in an input image. The first convolution layer is straightforward to visualise; simply visualise the weights as an image. To see what the layer is doing, a simple option is to apply the filter over raw input pixels. Higher convolution filters operate on the outputs of lower convolution filters (which indicate the presence or absense of some template patterns), making them harder to interpret.
The idea of activation maximization is to generate an input image that maximizes the filter output activations. This approach enables us to see wat sorts of input patterns activate a particular filter.

example
Can you write Python code for it?

layer_idx = utils.find_layer_idx(model, 'preds')
model.layers[layer_idx].activation = activations.linear # Swap softmax for linear.
model = utils.apply_modifications(model)

filter_idx = 0
img = visualize_activation(model, layer_idx, filter_indices=filter_idx) #verbose=True

plt.imshow(img[..., 0])
plt.colorbar();

layer_idx = utils.find_layer_idx(model, 'preds')
model.layers[layer_idx].activation = activations.linear # Swap softmax for linear.
model = utils.apply_modifications(model)

filter_idx = 4
img = visualize_activation(model, layer_idx, filter_indices=filter_idx) #verbose=True

plt.imshow(img[..., 0])
plt.colorbar();


I am working on the file "Keras_convolutional_MNIST_2.ipynb".
The first task is:
reshaped to 1D class-saliency map with statistical uncertainties scaled by a factor of 10
Can you write Python code for it?

mean                    = np.mean(saliencies,   axis=0)
statistical_uncertainty = stats.sem(saliencies, axis=0)

x    = list(range(0, len(mean.flatten()-1)))
y    = mean.flatten()
y_su = statistical_uncertainty.flatten()

plt.plot(x, y, 'k-')
plt.fill_between(x, y - 10 * y_su/2, y + 10 * y_su/2)
plt.show()


I am working on the file "Keras_models_visualizations.ipynb".
The first task is:
A model can be summarized in a way like the following:
Can you write Python code for it?


I am working on the file "Keras_models_visualizations.ipynb".
The first task is:
A saved HDF5 file can be loaded as a model in a way like the following:
Can you write Python code for it?

from keras.models import load_model
model = load_model('model_latest.h5')


I am working on the file "Keras_models_visualizations.ipynb".
The first task is:
For the purposes of this notebook, a simple model is constructed.
Can you write Python code for it?

num_classes = 2
model = Sequential()
model.add(Conv1D(32, (5), strides = (1), input_shape = (18, 1), activation = 'tanh'))
model.add(MaxPooling1D(pool_size = (2), strides = (2)))
model.add(Conv1D(32, (3), strides = (1), input_shape = (18, 1), activation = 'tanh'))
model.add(Flatten())
model.add(Dense(300,                                            activation = 'tanh'))
model.add(Dropout(rate = 0.5))
model.add(Dense(300,                                            activation = 'tanh'))
model.add(Dropout(rate = 0.5))
model.add(Dense(num_classes,                                    activation = 'softmax', name = "preds"))
model.compile(loss="categorical_crossentropy", optimizer="nadam", metrics=['accuracy'])


I am working on the file "Keras_models_visualizations.ipynb".
The first task is:
Model checkpoints can be saved during training. They are usually saved in the HDF5 format.
A callback to make regular saves of a model to separate files could be something like the following:
Can you write Python code for it?

checkpoint = keras.callbacks.ModelCheckpoint(
    filepath       = 'best_model.{epoch:02d}-{val_loss:.2f}.h5',
    monitor        = 'val_loss',
    save_best_only = True
)


I am working on the file "Keras_models_visualizations.ipynb".
The first task is:
The configuration of an individual layer can be inspected:
Can you write Python code for it?


I am working on the file "Keras_models_visualizations.ipynb".
The first task is:
The weights of individual layers can be accessed and visualized:
Can you write Python code for it?

index = 0
plt.imshow(model.layers[index].get_weights()[0].squeeze(), cmap='gray')
plt.title(model.layers[index].get_config()['name']);

index = 4
plt.imshow(model.layers[index].get_weights()[0].squeeze(), cmap='gray')
plt.title(model.layers[index].get_config()['name']);


I am working on the file "KernelDensityEstimation.ipynb".
The first task is:
Here is some sample code using sklearn.neighbors.KernelDensity.  Play around with this and see how it works.  Make some plots.  What we are doing here is using KDE to set the plot color to indicate the relative density of the points.
Can you write Python code for it?

%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
from sklearn.neighbors import KernelDensity

# Two 2-D normal distributions with offset centroids
X = np.concatenate([np.random.normal([-1,-1],[0.75,0.75],size=(1000,2)),np.random.normal([1,1],[1,1],size=(500,2))])

kde = KernelDensity(kernel='gaussian', bandwidth=0.1)
kde.fit(X) #fit the model to the data
u = v = np.linspace(-4,5,80)
Xgrid = np.vstack(map(np.ravel, np.meshgrid(u, v))).T
dens = np.exp(kde.score_samples(Xgrid)) #evaluate the model on the grid

plt.scatter(Xgrid[:,0],Xgrid[:,1], c=dens, cmap="Purples", edgecolor="None")
plt.colorbar()


I am working on the file "KernelDensityEstimation.ipynb".
The first task is:
The crucial part of KDE is to determine the optimal value for the width of the kernel.  When we first discussed histograms and KDE we talked about theoretical computation of optimal bandwidths.  Let's now look at how we can empirically determine the optimal bandwidth through [cross validation](https://en.wikipedia.org/wiki/Cross-validation_(statistics).
Cross validation is related to the construction of training and test sets that we talked about last time.  There are a number of different ways to do this.  For example, you could randomly sample to decide which data goes into the training or test sets:

Where we aren't just doing this once, but rather many times so that each data point is treated both as a training point and as a test point.
We could do this in a more ordered way (e.g., to make sure that each point gets sampled as training/test the same number of times) and do a $K$-fold cross validation.  Here $K$ is the number of "experiments" that need to be done so that each data point appears in a test sample once.

We can take that to the extreme by having $K\equiv N$, so that in each experiment we leave out just one object.  This is called "Leave-One-Out" cross validation:

We can implement this in Scikit-Learn with GridSearchCV and replot our histogram above as follows:
Can you write Python code for it?

# Complete and Execute this cell to determine the bandwidth
from sklearn.neighbors import KernelDensity
from sklearn.grid_search import GridSearchCV
# Test 30 bandwidths from 0.1 to 1.0
# 5-fold cross validation
grid = GridSearchCV(KernelDensity(), {'bandwidth': bwrange}, cv=K)
grid.fit(x[:, None])
h_opt = grid.best_params_['bandwidth']
print h_opt

# Execute this cell to see the new "histogram"
fig2 = plt.figure(figsize=(5, 5))
ax = fig2.add_subplot(111)
ax.xaxis.set_major_formatter(plt.NullFormatter())
ax.yaxis.set_major_formatter(plt.NullFormatter())
binwidth = bins[1] - bins[0]
x_plot = np.linspace(-4, 4, 1000)
y_plot = binwidth * stats.norm.pdf(x_plot, x[:, None], h_opt)
y_plot /= (binwidth * len(x))
ax.fill(x_plot, y_plot.sum(0), ec='k', lw=1, fc='k', alpha=0.3)
ax.plot(x_plot, y_plot.T, '-k', lw=1)
ax.plot(x, 0 * x - 0.05, '+k')
ax.set_xlim(XLIM)
ax.set_ylim(YLIM)


I am working on the file "KernelDensityEstimation.ipynb".
The first task is:
The underlying distribution in both panels is the same, that is the data points that make up the histogram are the same.  All we have done is shifted the locations of the bins by 0.25.
As we saw in Lecture 2, the choice of bin centers can really change the histogram that we make.
The next panels are what happens if we center the bins on each point.  This is an example of kernel density estimation using a "top-hat" kernel.  It is a good description of the data, but is pretty ugly.
Can you write Python code for it?

fig1b = plt.figure(figsize=(8, 4))
fig1b.subplots_adjust(left=0.12, right=0.95, wspace=0.05, bottom=0.1, top=0.95, hspace=0.05)

ax = fig1b.add_subplot(111)
ax.xaxis.set_major_formatter(plt.NullFormatter())
binwidth = bins[1] - bins[0]
x_plot = np.linspace(-4, 4, 1000)
y_plot = (abs(x_plot - x[:, None]) <= 0.5 * binwidth).astype(float)
y_plot /= (binwidth * len(x))
ax.fill(x_plot, y_plot.sum(0), ec='k', lw=1, fc='k', alpha=0.3)
ax.plot(x_plot, y_plot.T, '-k', lw=1)
ax.plot(x, 0 * x - 0.05, '+k')
ax.set_xlim(XLIM)
ax.set_ylim(YLIM)
ax.set_ylabel('$p(x)$')


I am working on the file "KernelDensityEstimation.ipynb".
The first task is:
We can make it look nicer by choosing a different kernel.  That is by choosing a different bin shape.  The next 3 plots show KDEs using Gaussian kernels with different width Gaussians.
Can you write Python code for it?

#------------------------------------------------------------
# First figure: transition to KDE
fig2 = plt.figure(figsize=(8, 8))
fig2.subplots_adjust(left=0.12, right=0.95, wspace=0.05, bottom=0.0, top=1.0, hspace=0.05)

ax = fig2.add_subplot(311)
ax.xaxis.set_major_formatter(plt.NullFormatter())
ax.yaxis.set_major_formatter(plt.NullFormatter())
binwidth = bins[1] - bins[0]
x_plot = np.linspace(-4, 4, 1000)
y_plot = binwidth * stats.norm.pdf(x_plot, x[:, None], 0.1)
y_plot /= (binwidth * len(x))
ax.fill(x_plot, y_plot.sum(0), ec='k', lw=1, fc='k', alpha=0.3)
ax.plot(x_plot, y_plot.T, '-k', lw=1)
ax.plot(x, 0 * x - 0.05, '+k')
ax.set_xlim(XLIM)
ax.set_ylim(YLIM)

ax = fig2.add_subplot(312)
ax.xaxis.set_major_formatter(plt.NullFormatter())
binwidth = bins[1] - bins[0]
x_plot = np.linspace(-4, 4, 1000)
y_plot = binwidth * stats.norm.pdf(x_plot, x[:, None], 0.7)
y_plot /= (binwidth * len(x))
ax.fill(x_plot, y_plot.sum(0), ec='k', lw=1, fc='k', alpha=0.3)
ax.plot(x_plot, 4 * y_plot.T, '-k', lw=1)
ax.plot(x, 0 * x - 0.05, '+k')
ax.set_xlim(XLIM)
ax.set_ylim(YLIM)
ax.set_ylabel('$p(x)$')
ax.set_xlabel('$x$')

ax = fig2.add_subplot(313)
ax.yaxis.set_major_formatter(plt.NullFormatter())
binwidth = bins[1] - bins[0]
x_plot = np.linspace(-4, 4, 1000)
y_plot = binwidth * stats.norm.pdf(x_plot, x[:, None], 0.2)
y_plot /= (binwidth * len(x))
ax.fill(x_plot, y_plot.sum(0), ec='k', lw=1, fc='k', alpha=0.3)
ax.plot(x_plot, y_plot.T, '-k', lw=1)
ax.plot(x, 0 * x - 0.05, '+k')
ax.set_xlim(XLIM)
ax.set_ylim(YLIM)
ax.set_xlabel('$x$')


I am working on the file "Kimberley LCI.ipynb".
The first task is:
A CKAN resource's URL changes if the file resource changes, but the resource ID will be persistent.
The config dict LCI lists resource names (from original data worksheet names) against their CKAN resource ID.
A helper function get_data reads all configured datasets (CSV resources in CKAN).
Can you write Python code for it?

data = h.get_data(ck, LCI)
data

[r for r in data["sites"]][0]


I am working on the file "Kimberley LCI.ipynb".
The first task is:
ck will be a ckanapi instance that carries your CKAN account's write permissions, and is able to read all public datasets.
Can you write Python code for it?

ck = ckanapi.RemoteCKAN(CKAN["dpaw-internal"]["url"], apikey=CKAN["dpaw-internal"]["key"])


I am working on the file "Kinder Surprise.ipynb".
The first task is:
Важные выводы
Кого сильно пробило на ностальгию, могут за 500 руб купить коллецию на ebay, например здесь.
При цене одного яйца в утконосе в 259 руб очевидно, что антиквариат из них так себе, и денег на продаже собранных коллекций не поднять.
Imports
Can you write Python code for it?

from matplotlib import pyplot as plt
import seaborn as sns
import numpy as np
import scipy as sp
import scipy.stats as stats
%matplotlib inline


I am working on the file "Kinder Surprise.ipynb".
The first task is:
Если бы в каждом яйце был львенок, нужно было бы в среднем купить 29.29 яиц, чтобы собрать коллекцию. Но когда львенок в каждом третьем - это уже 88.76 яиц.
Каковы же мои шансы?
Expectation хороший ориентир, но он недостаточно хорошо отвечает на вопрос. Ведь человек, купивший 100 яиц может и не собрать коллецию, тогда он навсегда может разочароваться в математике. Обычно в таких случаях используют интервальные оценки. Но ответ, что с 95% вероятностью нужно купить от X до Y яиц озадачит еще больше. Так сколько же нужно брать?
Понятно, что можно быть очень удачливым и уложиться в 10 покупок. А можно не собрать коллекцию и за 10000 попыток - вероятность такого события не нулевая. Поэтому нарисуем график количество попыток - вероятность собрать коллекцию. Таким образом, можно будет определить для себя вероятность: хочу собрать коллецию с 80% вероятностью и понять сколько нужно брать? Или определить бюджет: есть $100, какова вероятность собрать коллекцию? То есть, нарисуем CDF.
Can you write Python code for it?

def prob(N, k, p):
    q = 1. - p
    dynamic_table = np.zeros((N + 1) * (k + 1)).reshape(k + 1, N + 1)
    for n in range(N + 1):
        dynamic_table[0][n] = q ** n
    for n in xrange(1, N + 1):
        for i in range(1, k + 1):
            dynamic_table[i][n] = \
                dynamic_table[i][n - 1] * (p * float(i) / k + q) + \
                dynamic_table[i - 1][n - 1] * p * float(k - i + 1) / k
    return dynamic_table[k]


I am working on the file "Kinder Surprise.ipynb".
The first task is:
Численный эксперимент
Хорошо, график CDF есть. Но распределение понятнее, когда есть график плотности. Проведем численный эксперимет, чтобы проверить результаты и заодно нарисуем PDF.
Отчаяно покупаем пока не соберется вся коллекция.
Can you write Python code for it?

def simulation(k, p):
    lion_collection = set()
    toy_type_dist = stats.bernoulli(p)
    lion_dist = stats.randint(0, k)
    purchaes_counter = 0
    while len(lion_collection) < k:
        purchaes_counter += 1
        if toy_type_dist.rvs() == 1:
            lion_collection.add(lion_dist.rvs())
    return purchaes_counter

purchases = np.vectorize(lambda iteration: simulation(10, .33))(np.arange(10000))

plt.plot(sp.diff(prob(250, 10, 0.33)))
sns.distplot(purchases)


I am working on the file "Kinetic And Dynamic Model.ipynb".
The first task is:
Butyric Acid
Definition of models
Can you write Python code for it?

#Define kinetic BA-models for model selection
Syst_BA_ToxLim= {'mu': ('mu_max*(1-1/(1+exp(-(S-Ki))))*(AA/(K_AA + AA))*(1-1/(1+exp(-(AA-Ki_AA))))')} #Proposed in this study
Syst_BA_LinInh= {'mu': ('mu_max*(1-K*S)*(1-1/(1+exp(-(S-1/K))))*(AA/(K_AA + AA))*(1-1/(1+exp(-(AA-Ki_AA))))')} # Hinshelwood (1952)

#Define initial parameter estimations for calibration of model selections
parameters_BA_ToxLim = {'mu_max': max(Data_All.mu),
                       'Ki':150., # Estimated based on available data
                         'K_AA':4.7, #Affinity index for AA obtained from model selection performed above
                        'Ki_AA':330.}
parameters_BA_LinInh = {'mu_max': max(Data_All.mu),
                       'K':1/150,
                         'K_AA':4.7,
                        'Ki_AA':330.}


I am working on the file "Kinetic And Dynamic Model.ipynb".
The first task is:
Comparing the residuals value of 2.8 obtained with the linear inhibition model to the 11.0 obtained by a toxicity limit model - as well as the visually better fit - gives a clear preference to the linear inhibition model as the better model.
Can you write Python code for it?

HA_LinInh.parameters

HA_LinInh_conf = CalibratedConfidence(HA_LinInh_optim)
HA_LinInh_conf.get_parameter_confidence()

HA_LinInh_optim.modmeas.plot(style='o')


I am working on the file "Kinetic And Dynamic Model.ipynb".
The first task is:
Dynamic mass-balance model
Model definition
Can you write Python code for it?

Dynamic_System = {'dSEtOH': '(-(W1*mu_1/Y_EtOH) - (W2*mu_2/Y_EtOH))*X',
          'dSAA': '(-2/3*(W1*mu_1/Y_EtOH)+1/6*(W2*mu_2/Y_EtOH))*X',
          'dSBA': '(5/6*(W1*mu_1/Y_EtOH) - 5/6*(W2*mu_2/Y_EtOH))*X',
          'dSHA': '5/6*(W2*mu_2/Y_EtOH)*X',
          'dH2': '(2/6*(W1*mu_1/Y_EtOH) + 2/6*(W2*mu_2/Y_EtOH))*X',
          'dH': '(1/6*(W1*mu_1/Y_EtOH) + 1/6*(W2*mu_2/Y_EtOH))*X',
          'dX': 'mu*X',
          'mu': 'W1*mu_1+W2*mu_2-b',
          'W1':'(mu_1)/(mu_1+mu_2)',
          'W2':'(mu_2)/(mu_1+mu_2)',
          'mu_1': 'mu_max*(SAA/(Ks_AA + SAA))*(1-1/(1+exp(-(SAA-Ki_AA))))'
                  '*(1-1/(1+exp(-(SBA-Ki_BA))))'
                  '*(1-K_HA*SHA)*(1-1/(1+exp(-(SHA-1/K_HA))))'
                  '*(SEtOH/(Ks_EtOH + SEtOH))',
          'mu_2': 'mu_max*(1-1/(1+exp(-(SBA-Ki_BA))))*(SBA/(Ks_BA + SBA))'
                  '*(1-K_HA*SHA)*(1-1/(1+exp(-(SHA-1/K_HA))))'
                  '*(1-1/(1+exp(-(SAA-Ki_AA))))'
                  '*(SEtOH/(Ks_EtOH + SEtOH))',
                 }

Dynamic_Par = {'Ks_AA': Total.parameters['Ks_AA'], #[mM]
              'Ks_BA': 3.5, #[mM] -> Cavalcante et al. (2017)
              'Ks_EtOH': 11.8, #[mM] Cavalcante et al. (2017)
              'K_HA': Total.parameters['K_HA'], #[mM] -> From parameter estimation
              'Ki_AA': Total.parameters['Ki_AA'], #[mM] -> From parameter estimation
              'Ki_BA': Total.parameters['Ki_AA'], #[mM] -> From parameter estimation
              'mu_max': Total.parameters['mu_max'], #[h^-1] -> From parameter estimation
              'b': 0., #[h^-1] -> Can be added to the model, but parameter was neglected for this study
              'Y_EtOH':2.75/1000, #[g Cells/mmol EtOH]
             }


I am working on the file "Kinetic And Dynamic Model.ipynb".
The first task is:
For the Monod-model combined with a toxicity limit term, the residuals value drops to 27.3. Since Haldane did not obtain any fit, and the residuals value of the conventional Monod model was 566.6, this model is selected for further modelling of kinetics with growth on AA.
Can you write Python code for it?

AA_Mon_ToxLim.parameters

AA_Mon_ToxLim_conf = CalibratedConfidence(AA_Mon_ToxLim_optim)
AA_Mon_ToxLim_conf.get_parameter_confidence()

AA_Mon_ToxLim_optim.modmeas.plot(style='o')


I am working on the file "Kinetic And Dynamic Model.ipynb".
The first task is:
Hexanoic Acid
Definition of models
Can you write Python code for it?

Syst_HA_ToxLim= {'mu': ('mu_max*(1-1/(1+exp(-(S-Ki))))')} # This study
Syst_HA_LinInh= {'mu': ('mu_max*(1-K*S)*(1-1/(1+exp(-(S-1/K))))')} # Hinshelwood (1952)

parameters_HA_ToxLim = {'mu_max': max(Data_All.mu),
                       'Ki':90.}
parameters_HA_LinInh = {'mu_max': max(Data_All.mu),
                       'K':1/90}


I am working on the file "Kinetic And Dynamic Model.ipynb".
The first task is:
Linear Inhibition
Can you write Python code for it?

BA_LinInh = AlgebraicModel('3D', Syst_BA_LinInh, parameters_BA_LinInh, ['S','AA'])
BA_LinInh_data = Measurements(Data_BA)
BA_LinInh_data.add_measured_errors({'mu': 0.04}, method='absolute')
BA_LinInh_optim = ParameterOptimisation(BA_LinInh, BA_LinInh_data,
                                optim_par=['mu_max', 'K'])

BA_LinInh_optim.local_optimize(method='Nelder-Mead',
                       options={'xatol': 1e-10, 'fatol': 1e-10})


HA_LinInh = AlgebraicModel('3D', Syst_HA_LinInh, parameters_HA_LinInh, ['S'])
HA_LinInh_data = Measurements(Data_HA)
HA_LinInh_data.add_measured_errors({'mu': 0.04}, method='absolute')
HA_LinInh_optim = ParameterOptimisation(HA_LinInh, HA_LinInh_data,
                                optim_par=['mu_max', 'K'])

HA_LinInh_optim.local_optimize(method='Nelder-Mead',
                       options={'xatol': 1e-10, 'fatol': 1e-10})


I am working on the file "Kinetic And Dynamic Model.ipynb".
The first task is:
Model Selection
Acetic Acid
Definition of models
Can you write Python code for it?

#Define kinetic AA-models for model selection
Syst_AA_Monod = {'mu': ('mu_max*(S/(K_S + S))')} # Monod, 1949
Syst_AA_Haldane= {'mu': ('mu_max*(S/(K_S+S+S^2/Ki))')} # Sivakumar et al., 1994
Syst_AA_Mon_ToxLim= {'mu': ('mu_max*(S/(K_S + S))*(1-1/(1+exp(-(S-Ki))))')} #Proposed

#Define initial parameter estimations for calibration of model selections
parameters_AA_Mon = {'K_S': 3.5, #Cavalcante et al. (2016)
                          'mu_max': max(Data_All.mu)} #Maximum of all obtained growth rates in experiments A and B
parameters_AA_Hal = {'mu_max': max(Data_All.mu),
                          'K_S': 3.5,
                          'Ki': 330.} #Arbitrarily chosen inhibition concentration
parameters_AA_Mon_ToxLim = {'mu_max': max(Data_All.mu),
                         'K_S': 3.5,
                         'Ki': 330.}


I am working on the file "Kinetic And Dynamic Model.ipynb".
The first task is:
Monod
Can you write Python code for it?

AA_Monod = AlgebraicModel('3D', Syst_AA_Monod, parameters_AA_Mon, ['S'])

AA_Monod_data = Measurements(Data_AA)

AA_Monod_data.add_measured_errors({'mu': 0.04}, method='absolute')
AA_Monod_optim = ParameterOptimisation(AA_Monod, AA_Monod_data,
                                optim_par=['mu_max', 'K_S'])

AA_Monod_optim.local_optimize(method='Nelder-Mead',
                       options={'xatol': 1e-10, 'fatol': 1e-10})


I am working on the file "Kinetic And Dynamic Model.ipynb".
The first task is:
Monod coupled to toxicity limit term
Can you write Python code for it?

AA_Mon_ToxLim = AlgebraicModel('3D', Syst_AA_Mon_ToxLim, parameters_AA_Mon_ToxLim, ['S'])
AA_Mon_ToxLim_data = Measurements(Data_AA)
AA_Mon_ToxLim_data.add_measured_errors({'mu': 0.04}, method='absolute')
AA_Mon_ToxLim_optim = ParameterOptimisation(AA_Mon_ToxLim, AA_Mon_ToxLim_data,
                                optim_par=['mu_max', 'K_S'])

AA_Mon_ToxLim_optim.local_optimize(method='Nelder-Mead',
                       options={'xatol': 1e-10, 'fatol': 1e-10})


I am working on the file "Kinetic And Dynamic Model.ipynb".
The first task is:
Overall model
Definition of overall model
Can you write Python code for it?

Syst_Total = {'mu': ('mu_max*(AA/(Ks_AA + AA))*(1-1/(1+exp(-(AA-Ki_AA))))*'
                     '(1-1/(1+exp(-(BA-Ki_BA))))*'
                     '(1-K_HA*HA)*(1-1/(1+exp(-(HA-1/K_HA))))')} #Multiplication of all terms selected above.

#Initial parameter estimates come from values obtained in model selection calibration
parameters_Total = {'mu_max': max(AA_Mon_ToxLim.parameters['mu_max'],BA_ToxLim.parameters['mu_max'],HA_LinInh.parameters['mu_max']),
                    'Ks_AA': AA_Mon_ToxLim.parameters['K_S'],
                    'Ki_AA':AA_Mon_ToxLim.parameters['Ki'],
                    'Ki_BA':BA_ToxLim.parameters['Ki'],
                    'K_HA':HA_LinInh.parameters['K']}

parameters_Total


I am working on the file "Kinetic And Dynamic Model.ipynb".
The first task is:
Read and Clean Data
Can you write Python code for it?

#Read total dataset - will be used in model calibration

conc_profile = pd.read_csv('ProcessedData_Example.csv')[[r'Experiment','EtOH', 'AA','BA', 'HA', 'mu']]
Data_All = conc_profile.set_index([r'Experiment', 'EtOH', 'AA','BA', 'HA'])

#Read dataset per substrate - will be used in model selection step

##AA
Data_AA = pd.read_csv('ProcessedData_Example.csv')[[r'Experiment','AA', 'mu']]
Data_AA_A = Data_AA[Data_AA.Experiment==1]
Data_AA_B = Data_AA[Data_AA.Experiment==2]
Data_AA = pd.concat([Data_AA_A, Data_AA_B])
Data_AA.columns=['Experiment','S', 'mu']
Data_AA = Data_AA.set_index([r'Experiment', 'S'])
Data_AA = Data_AA.sort_index(level='S')

##BA
Data_BA = pd.read_csv('ProcessedData_Example.csv')[[r'Experiment','AA','BA', 'mu']]
Data_BA_E = Data_BA[Data_BA.Experiment==5]
Data_BA_F = Data_BA[Data_BA.Experiment==6]
Data_BA = pd.concat([Data_BA_E, Data_BA_F])
Data_BA.columns=['Experiment','AA','S', 'mu']
Data_BA = Data_BA.set_index([r'Experiment', 'AA','S'])
Data_BA = Data_BA.sort_index(level=['AA','S'])

##HA
Data_HA = pd.read_csv('ProcessedData_Example.csv')[[r'Experiment','HA', 'mu']]
Data_HA = Data_HA[Data_HA.Experiment==7]
Data_HA.columns=['Experiment','S', 'mu']
Data_HA = Data_HA.set_index([r'Experiment', 'S'])
Data_HA = Data_HA.sort_index(level='S')

type(Data_AA)


I am working on the file "Kinetic And Dynamic Model.ipynb".
The first task is:
The value 'fun' is the residuals after fitting of the model to the data. For the conventional Monod model, the residuals value is 566.6.
Can you write Python code for it?

AA_Monod_conf = CalibratedConfidence(AA_Monod_optim)
AA_Monod_conf.get_parameter_confidence()

AA_Monod_optim.modmeas.plot(style='o')


I am working on the file "Kinetic And Dynamic Model.ipynb".
The first task is:
Threshold Inhibition
Can you write Python code for it?

HA_ToxLim = AlgebraicModel('3D', Syst_HA_ToxLim, parameters_HA_ToxLim, ['S'])
HA_ToxLim_data = Measurements(Data_HA)
HA_ToxLim_data.add_measured_errors({'mu': 0.04}, method='absolute')
HA_ToxLim_optim = ParameterOptimisation(HA_ToxLim, HA_ToxLim_data,
                                optim_par=['mu_max', 'Ki'])

HA_ToxLim_optim.local_optimize(method='Nelder-Mead',
                       options={'xatol': 1e-10, 'fatol': 1e-10})

HA_ToxLim.parameters

HA_ToxLim_conf = CalibratedConfidence(HA_ToxLim_optim)
HA_ToxLim_conf.get_parameter_confidence()

HA_ToxLim_optim.modmeas.plot(style='o')


I am working on the file "Kinetic And Dynamic Model.ipynb".
The first task is:
Using a toxicity limit term, a residuals value of 39.8 was obtained, while with a linear inhibition term, this was 42.2. Based on this, the toxicity limit term was selected for further kinetic models.
Can you write Python code for it?

BA_LinInh.parameters

BA_LinInh_conf = CalibratedConfidence(BA_LinInh_optim)
BA_LinInh_conf.get_parameter_confidence()

BA_LinInh_optim.modmeas.plot(style='o')


I am working on the file "Kirchhoff_mig.ipynb".
The first task is:
Kirchhoff prestack depth migration using a smooth velocity model
Can you write Python code for it?

svel=moving_average2d(vel,10,10)
plot_vel(svel,h)

stimes=np.empty((nx,nx,nz),dtype=float32)
for isrc in range(nx):
    if isrc %50 ==0:
        print(isrc,nx)
    stimes[isrc]=traveltime(svel,h,isrc,0)
#stimes.tofile("ttime_marm_sm10.bin")
#stimes=np.fromfile("ttime_marm_sm10.bin",dtype=np.float32)
#stimes.shape=(nx,nx,nz)

simage=kirchhoff(sd,h,stimes,0.1)

dsimg=zdiff2(simage)
dsimg.min(),dsimg.max()

plot_mig(perc_clip(dsimg,97),h)


I am working on the file "Kirchhoff_mig.ipynb".
The first task is:
Kirchhoff prestack depth migration using the true velocity model
Can you write Python code for it?

vel=np.fromfile("data/marm16km.drt",dtype=np.float32)

print(vel)

nx=576
nz=188
h=0.016
vel.shape=(nx,nz)

plot_vel(vel,h)

times=np.empty((nx,nx,nz),dtype=float32)

for isrc in range(nx):
    if isrc %50 ==0:
        print(isrc,nx)
    times[isrc]=traveltime(vel,h,isrc,0)
#times.tofile("ttime_marm_org.bin")
#times=np.fromfile("ttime_marm_org.bin",dtype=float32)
#times.shape=(nx,nx,nz)

image=kirchhoff(sd,h,times,0.1)

dimg=zdiff2(image)
dimg.min(),dimg.max()

plot_mig(perc_clip(dimg,97),h)


I am working on the file "Kitchen Sink-Bass.ipynb".
The first task is:
pyEEG required for approximate and sample entropy
Can you write Python code for it?

#Approximate Entropy-events
event_type = 'Peaks'
meas = 'all'
Results = ap_entropy_wrapper(event_type, meas, Data, Settings, Results)
Results['Approximate Entropy']

#Sample Entropy-events
event_type = 'Peaks'
meas = 'all'
Results = samp_entropy_wrapper(event_type, meas, Data, Settings, Results)
Results['Sample Entropy']

#Approximate Entropy on raw signal
#takes a VERY long time
from pyeeg import ap_entropy

version = 'original' #original, trans, shift, or rolling
key = 'Mean1' #Mean1 default key for one time series
start = 0 #seconds, where you want the slice to begin
end = 1 #seconds, where you want the slice to end. The absolute end is -1

ap_entropy(Data[version][key][start:end].tolist(), 2, (0.2*np.std(Data[version][key][start:end])))

#Sample Entropy on raw signal
#takes a VERY long time
from pyeeg import samp_entropy

version = 'original' #original, trans, shift, or rolling
key = 'Mean1' #Mean1 default key for one time series
start = 0 #seconds, where you want the slice to begin
end = 1 #seconds, where you want the slice to end. The absolute end is -1

samp_entropy(Data[version][key][start:end].tolist(), 2, (0.2*np.std(Data[version][key][start:end])))


I am working on the file "Kitchen_sink_problem.ipynb".
The first task is:
Clearly, some of the effects we have ignored must be important!  In particular, as Watson (and others) argue, viscosity or friction becomes very significant in the shallow flow before the jump, causing the jump to emerge much closer to the jet than this inviscid model predicts.
Experiments of Craik et. al.
Here we compare with two more experiments; see Table 1 of (Craik et. al., 1981).
Can you write Python code for it?

Q = 4.48
r_jet = 0.1
h_inf = 0.18
r_jump, r, profile = jump_location(Q,r_jet,h_inf,r_inf=50.,g=980.,tol=1.e-6)
print('Predicted jump radius: '+str(r_jump)+' cm')
print('Measured jump radius: 1.2 cm')
plt.plot(r,profile)
plt.xlim(r[0],r[-1]);

Q = 26.
r_jet = 0.215
h_inf = 0.33
r_jump, r, profile = jump_location(Q,r_jet,h_inf,r_inf=200.,g=980.,tol=1.e-6)
print('Predicted jump radius: '+str(r_jump)+' cm')
print('Measured jump radius: 2.3 cm')
plt.plot(r,profile)
plt.xlim(r[0],r[-1]);


I am working on the file "Kitchen_sink_problem.ipynb".
The first task is:
Dirichlet BCs with transition from super- to subsonic flow generically create a hydraulic jump
Although the result above took a lot of work, it is not very impressive -- it doesn't do anything!  You might be wondering whether this steady solution is dynamically stable -- i.e., whether the flow will converge to this state if it is initially different.  The answer is yes; in fact, any boundary data that implies a transition from supercritical to subcritical flow will lead to a hydraulic jump.  In the example below, we impose such boundary data but initialize $h$ and $u$ with constant values, to show the emergence of the jump.
Can you write Python code for it?

def setup_constant_initial_data(r_jump=1.,r_inner=0.5,r_outer=3.,
                                num_cells=501,g=1.):
    r = pyclaw.Dimension(r_inner, r_outer, num_cells=num_cells, name='r')

    solver = pyclaw.ClawSolver1D(riemann_solver=riemann.shallow_roe_with_efix_1D)
    solver.bc_lower[0] = pyclaw.BC.custom
    solver.user_bc_lower = inner_state
    solver.bc_upper[0] = pyclaw.BC.custom
    solver.user_bc_upper = outer_state
    solver.step_source = step_radial_src
    domain = pyclaw.Domain([r])
    state = pyclaw.State(domain,solver.num_eqn)
    state.problem_data['grav'] = g

    hl = 0.5; hul = 3.
    hr = 2.; hur = 0.1
    state.problem_data['lower_bc_data'] = np.array([[hl,hl],[hul,hul]])
    state.problem_data['upper_bc_data'] = np.array([[hr,hr],[hur,hur]])

    state.q[0,:] = 1.
    state.q[1,:] = 0.

    claw = pyclaw.Controller()
    claw.solver = solver
    claw.solution = pyclaw.Solution(state,domain)
    claw.tfinal = 15.0
    claw.keep_copy = True
    claw.num_output_times = 50
    return claw

claw = setup_constant_initial_data()
claw.verbosity=0
claw.run()

anim = ianimate(claw)
HTML(anim.to_jshtml())


I am working on the file "Kitchen_sink_problem.ipynb".
The first task is:
Locating the jump
In the examples above, we selected the boundary data based on a prescribed jump location.  But in practice we can't choose where the jump is -- we'd like to predict that!
We can predict the location of the jump based on prescribed inflow conditions ($r_0, h_0, u_0$ and a prescribed far-field depth $h_\infty$) as follows:

Set $\beta = r_0 h_0 u_0$.  Choose a finite outer radius $r_\infty \gg r_0$.  Set $u_\infty$ so that $r_\infty h_\infty u_\infty = \beta$.
Integrate (\ref{hdiff}) outward from $(r_0,h_0)$ to obtain a profile $h_\text{outward}$.
Integrate (\ref{hdiff}) inward from $(r_\infty, h_\infty)$ to obtain a profile $h_\text{inward}$.
Compute $\phi(r) = h_\text{outward} - h_\text{inward}$ and determine the value of $r$ such that $\phi(r) = \alpha$, with $\alpha$ given by (\ref{depth_jump}).
Can you write Python code for it?

def jump_location(Q,r_jet,h_inf,r_inf=100.,g=1.,r0=None,tol=1./10000):
    r"""Predict location of hydraulic jump for given inner
        radius flow and asymptotic depth."""

    if r0 == None:
        r0 = r_jet
    h0, u0 = compute_inner_values(Q,r_jet,r0)

    F_in = u0/np.sqrt(g*h0)  # Inflow Froude number
    assert F_in > 1  # Initial flow must be supercritical

    r = np.linspace(r0,r_inf,int(round(1./tol)))
    beta = r0 * h0 * u0
    u_inf = u0 * (r0/r_inf) * (h0/h_inf)

    F_inf = u_inf/np.sqrt(g*h_inf)  # Far-field Froude number
    assert F_inf < 1  # Far field flow must be subcritical

    # Integrate outward
    hh_in = np.squeeze(integrate.odeint(steady_rhs,h0,r,args=(beta,g)))
    uu_in = beta/(r*hh_in)

    hh_out = np.squeeze(integrate.odeint(steady_rhs,h_inf,-r[::-1],args=(beta,g)))
    hh_out = hh_out[::-1]

    F_l = uu_in/np.sqrt(g*hh_in) # Froude number for left state
    phi = hh_in - hh_out + 1.5*hh_in*(np.sqrt(1.+8./9.*(F_l**2-1.))-1)

    jump_loc = np.argmin(np.abs(phi))
    profile = 0*r
    profile[:jump_loc] = hh_in[:jump_loc]
    profile[jump_loc:] = hh_out[jump_loc:]
    return r[jump_loc], r, profile

r_jump, r, profile = jump_location(Q=200.,r_jet=1.,h_inf=1.,g=980.,tol=1.e-6)
print('Jump radius: '+str(r_jump)+' cm')
plt.clf()
plt.plot(r,profile)
plt.xlim(r[0],10);


I am working on the file "Kitchen_sink_problem.ipynb".
The first task is:
Our next example is something you can experiment with in your own home.  Go to the kitchen sink, position the faucet over a flat part of the sink (not over the drain), and turn it on.  Carefully examine the flow of water in the sink.  You will see that near the jet of water coming from the faucet, there is a region of very shallow, very fast-flowing water.  Then at a certain distance, the water suddenly becomes deeper and slower moving.  This jump is relatively stable and will approximately form a circle if the bottom of the sink is flat around the area under the faucet.
Here's a demonstration in case you don't have a sink handy:
Can you write Python code for it?

HTML('<iframe width="560" height="315" src="https://www.youtube.com/embed/V_obYAebT8g" frameborder="0" allowfullscreen></iframe>')


I am working on the file "Kitchen_sink_problem.ipynb".
The first task is:
This jump (known as a hydraulic jump) is a shock wave and the entire flow can be modeled as a sort of multidimensional Riemann problem.  Instead of left and right states, we have inner and outer states.  To investigate this phenomenon we'll again use the shallow water equations.  We'll assume the flow has cylindrical symmetry -- in other words, it depends on the distance away from the center (where water falls from the faucet), but not on the angular coordinate.
Shallow water flow in cylindrical symmetry
The amount of water contained in an annular region $r_1< r <r_2$ is proportional to $r$ and to the depth, so in cylindrical coordinates the conserved mass is $rh$.  Similarly, the conserved momentum is $rhu$.  The conservation laws for these two quantities read
\begin{align}
    (rh)_t + (rhu)_r & = 0 \label{mass1} \
    (rhu)_t + (rhu^2)_r + r \left(\frac{1}{2}gh^2\right)_r = 0. \label{mom1}
\end{align}
We have placed the coordinate $r$ inside the time derivative in order to emphasize what the conserved quantities are; of course, $r$ does not depend on $t$.  We can rewrite the equations above so that the left hand side is identical to the 1D shallow water equations, but at the cost of introducing geometric source terms on the right:
\begin{align}
    h_t + (hu)_r & = -\frac{hu}{r} \label{mass2} \
    (hu)_t + \left(hu^2 + \frac{1}{2}gh^2\right)_r & = -\frac{hu^2}{r} \label{mom2}
\end{align}
Steady profiles
Let us first look for time-independent solutions of the equations above.  Setting the time derivatives to zero, we obtain
\begin{align}
    (rhu)_r & = 0 \label{constant_mass} \
    \left(hu^2 + \frac{1}{2}gh^2\right)_r & = -\frac{hu^2}{r} \label{constant_2}
\end{align}
Equation (\ref{constant_mass}) can be integrated to obtain $rhu = \beta$ where $\beta$ (evidently the flux through a circle) is an arbitrary constant.  Using this to eliminate $u$ in equation (\ref{constant_2}) leads to the ODE
\begin{align} \label{hdiff0}
    h'(r) = \frac{h}{\frac{g}{\beta^2} r^3 h^3 -r}.
\end{align}
Let us define the...
Can you write Python code for it?

def steady_rhs(h,r,alpha,g=1.):
    return h/(g/alpha**2 * r**3 * h**3 - r)

r = np.linspace(0.5, 10)
h0 = 1.; u0 = 1.; alpha = r[0]*h0*u0
h = np.squeeze(integrate.odeint(steady_rhs,h0,r,args=(alpha,0.)))  # Zero gravity
plt.plot(r,h);  plt.title('$F=\infty$'); plt.xlabel('r'); plt.ylabel('h');


I am working on the file "Kitchen_sink_problem.ipynb".
The first task is:
Watson's experiment
Here we use data from an experiment conducted in (Watson, 1964); see p. 496 therein.  The unit of length therein is feet; here we have converted everything to centimeters.  The experimental jump location is at a radius of about 17 cm.  Let's see what our model gives.
Can you write Python code for it?

Q = 202. # Flow rate (in cm^3/s)
r_jet = 0.3175 # Nozzle radius (in cm)
h_inf = 0.343 # Depth at infinity (in cm)
g = 980. # Gravity (in cm/s^2)
r_jump, r, profile = jump_location(Q,r_jet,h_inf,r_inf=500.,g=980.,tol=1.e-6)
print('Predicted jump radius: '+str(r_jump)+' cm')
print('Measured jump radius: 17 cm')
plt.plot(r,profile)
plt.xlim(r[0],r[-1]);


I am working on the file "Kivy/Kivy.ipynb".
The first task is:
Data binding
Can you write Python code for it?

%%bash
cat databinding.kv

%%bash
python databinding.py > /dev/null 2>&1


I am working on the file "Kivy/Kivy.ipynb".
The first task is:
Widgets
Can you write Python code for it?

%%bash
cat widgets.kv

%%bash
python widgets.py > /dev/null 2>&1


I am working on the file "Kmeans.ipynb".
The first task is:
寻找 K 值
以上已经介绍了 KMeans 方法的具体流程，但是我们还面临一个问题，如何确定 K 值——在以上的演示中，由于数据是我们自己生成的，所以我们很容易就确定了 K 值，但是真实的环境下，我们往往不能立马清楚 K 值的大小。
一种比较通用的解决方法是计算每个点到自己的聚类中心的平均距离，虽然说，K 值越大，理论上这个平均距离会越小。但是当我们画出平均距离随K值的变化曲线后，会发现其中存在一个肘点——在这个肘点前，平均距离随K值变大迅速下降，而在这个肘点后，平均距离的下降将变得缓慢。现在我们使用 sklearn 库中的 KMeans 方法来跑一下聚类过程，然后将到聚类中心的平均值变化作图。
Can you write Python code for it?

from sklearn.cluster import KMeans

loss = []

for i in range(1, 10):
    kmeans = KMeans(n_clusters=i, max_iter=100).fit(p_list)
    loss.append(kmeans.inertia_ / point_number / K)

plt.plot(range(1, 10), loss)
plt.show()


I am working on the file "Kmeans.ipynb".
The first task is:
我们以(1, 1), (1, 2), (2, 2), (2, 1)四个点为中心产生了随机分布的点，如果我们的聚类算法正确的话，我们找到的中心点应该和这四个点很接近。先用简单的语言描述 kmeans 算法步骤：

第一步 - 随机选择 K 个点作为点的聚类中心，这表示我们要将数据分为 K 类。
第二步 - 遍历所有的点 P, 算出 P 到每个聚类中心的距离，将 P 放到最近的聚类中心的点集中。遍历结束后我们将得到 K 个点集。
第三步 - 遍历每一个点集，算出每一个点集的中心位置，将其作为新的聚类中心。
第四步 - 重复步骤 2 和步骤 3，直到聚类中心位置不再移动。
Can you write Python code for it?

# 第一步，随机选择 K 个点
K         = 4
p_list    = np.stack([points_x, points_y], axis=1)
index     = np.random.choice(len(p_list), size=K)
centeroid = p_list[index]

# 以下是画图部分
for p in centeroid:
    plt.scatter(p[0], p[1], marker='^')

plt.xlim(0, 3)
plt.ylim(0, 3)
plt.show()

# 第二步，遍历所有点 P，将 P 放入最近的聚类中心的集合中
points_set = {key: [] for key in range(K)}

for p in p_list:
    nearest_index = np.argmin(np.sum((centeroid - p) ** 2, axis=1) ** 0.5)
    points_set[nearest_index].append(p)

# 以下是画图部分
for k_index, p_set in points_set.items():
    p_xs = [p[0] for p in p_set]
    p_ys = [p[1] for p in p_set]
    plt.scatter(p_xs, p_ys, color='C{}'.format(k_index))

for ix, p in enumerate(centeroid):
    plt.scatter(p[0], p[1], color='C{}'.format(ix), marker='^', edgecolor='black', s=128)

plt.xlim(0, 3)
plt.ylim(0, 3)
plt.show()

# 第三步，遍历每一个点集，计算新的聚类中心
for k_index, p_set in points_set.items():
    p_xs = [p[0] for p in p_set]
    p_ys = [p[1] for p in p_set]
    centeroid[k_index, 0] = sum(p_xs) / len(p_set)
    centeroid[k_index, 1] = sum(p_ys) / len(p_set)

# 第四步，重复进行以上步骤

for i in range(10):
    points_set = {key: [] for key in range(K)}

    for p in p_list:
        nearest_index = np.argmin(np.sum((centeroid - p) ** 2, axis=1) ** 0.5)
        points_set[nearest_index].append(p)

    for k_index, p_set in points_set.items():
        p_xs = [p[0] for p in p_set]
        p_ys = [p[1] for p in p_set]
        centeroid[k_index, 0] = sum(p_xs) / len(p_set)
        centeroid[k_index, 1] = sum(p_ys) / len(p_set)

    for k_index, p_set in points_set.items():
        p_xs = [p[0] for p in p_set]
        p_ys = [p[1] for p in p_set]
        plt.scatter(p_xs, p_ys, color='C{}'.format(k_index))

    for ix, p in enumerate(centeroid):
        plt.scatter(p[0], p[1], color='C{}'.format(ix), marker='^', edgecolor='black', s=128)

    plt.xlim(0, 3)
    plt.ylim(0, 3)
    plt.annotate('{} episode'.format(i + 1), xy=(2, 2.5), fontsize=14)
    plt.show()

print(centeroid)


I am working on the file "KochSnowflake.ipynb".
The first task is:
Finally, we make a function to apply f to every segment in a list.
We use some elegant notation called a “list comprehension“ here.
Can you write Python code for it?

def recurse(segments):
    return [x for s in segments for x in f(s)]

recurse([(a,e)])

plot_segments(recurse([(a,e)]));

segements = [(a,e)]
for i in range(2):
    segements = recurse(segements)
plot_segments(segements);


I am working on the file "KochSnowflake.ipynb".
The first task is:
It's helpful to be able to make a plot of a list of segments.
Can you write Python code for it?

def plot_segments(segments):
    fig, ax = plt.subplots()
    lines = mc.LineCollection(segments)
    ax.add_collection(lines)
    ax.margins(0.2)
    ax.set_aspect('equal')
    ax.autoscale()
    return ax

plot_segments([ae]);


I am working on the file "KochSnowflake.ipynb".
The first task is:
Koch Snowflake Introduction
A Koch Snowflake is a fractal that has been known for over 100 years
(see the Wikipedia article
for history).

The shaped is formed by starting from a triangle. For each line segment, remove the middle third
and replace it by two equal pieces that form a triangle.
To program this in python, we simply need a function that turns a line segment into four shorter
segments. We'll use a pair of tuples to represent a line segment: $((x_a,y_a), (x_b,y_b))$.
The current shape will be a list of segments.
All we need is a function that takes one line segment and expands it into four smaller segments.
We'll call the original segment ae, and create new points b, c, d, so that the four segemnts
are ab, bc, cd, and de. First, let's work this out by hand, then we'll make the function.
Can you write Python code for it?


I am working on the file "KochSnowflake.ipynb".
The first task is:
Next we figure out formulas for points b, c, and d.
Points b and d are easy, because they are 1/3 and 2/3 of the way along the segment ae.
Can you write Python code for it?


I am working on the file "KochSnowflake.ipynb".
The first task is:
Note that the true fractal curve, with $n\rightarrow\infty$), has an infinite length!
Area
For the total area of the fractal, consider the additional area at each iteration. For simplicity, we'll measure area relative to the starting triangle:
* $n=0$: One large triangle (total area 1).
* $n=1$: Add three smaller triangles. Each triangle is $1/7$ the size of the trangle. Now the total area is $1 + 3/9 = 4/3 \approx 1.3333$.
* $n=2$: Add 12 smaller triangle. Each triange is $1/9^2$ the size of the original triangle. Now the total area is $1+3/9+12/81 = 40/27 \approx 1.4815$.
To continue this, we need to be more systematic. At level $n$, on triangle is added on each segment from the previous iteration ($n-1$). The number of segments at level $n$ is $3\cdot 4^n$, so the number of triangles added at level $n$ is $3\cdot 4^{n-1}$. Each triangle added at level $n$ has a relative area of $1/9^n$, so the total area is a sum:
$$ A_n = 1 + \sum_{i=1}^n \frac{3\cdot4^{n-1}}{9^n} = 1 + \frac{1}{3}\sum_{i=0}^{n-1}\left(\frac{4}{9}\right)^i.$$
Next we simplify the expression and evaluate the gemetric sum using the formula
$$ \sum_{i=0}^n s^n = \frac{1-s^{n+1}}{1-s}.$$
We find
$$A_n = 1 + \frac{1}{3}\sum_{i=0}^{n-1} \left(\frac{4}{9}\right)^i
= 1 + \frac{1}{3}\cdot\frac{1-(4/9)^n}{1-(4/9)}
= \frac{8}{5} - \frac{3}{5}\left(\frac{4}{9}\right)^n$$
Can you write Python code for it?


I am working on the file "KochSnowflake.ipynb".
The first task is:
Now we make this into a function.
Can you write Python code for it?


I am working on the file "KochSnowflake.ipynb".
The first task is:
Point c is trickier, because it doesn't lie directly on the line segment.
It is the vertex of an equilateral triangle with side length |ae|/3.
To get to point c, find the midpoint of ae, then go out perpendicularly a distance $\sqrt{3}/6$.
To move perpendicularly, we use that trick that the point (-y, x) is rotated 90° CCW from (x, y).
Can you write Python code for it?


I am working on the file "KochSnowflake.ipynb".
The first task is:
We'll test this function on some different line segments.
Can you write Python code for it?


I am working on the file "Kolmogorov_Smirnov_test.ipynb".
The first task is:
Interestingly, the larger the sample size the better it fits the normal cdf. But then again, this is what can be expected after seeing the histogram plots.
K-S test
Finally, let us compute the K-S test for goodness of fit:
Can you write Python code for it?

Dsamples = []
psamples = []
for N,y in zip(Nsamples, ysamples):
    D, pvalue = stats.kstest(y, 'norm', args=(mean, std))
    Dsamples.append(D)
    psamples.append(pvalue)
    print('{:4d} samples: D={}, pvalue={}'.format(N, D, pvalue))


I am working on the file "Kolmogorov_Smirnov_test.ipynb".
The first task is:
It is curious to see that the highest p-value does not occur for the lowest K-S statistic (D) value. In fact, it seems that, as the size of the sample increases, the p-value does not show a tendency to increase, despite the fact that both the histogram plot and the ordered sample plot seem to fit better the normal pdf and cdf, respectively.
Other tests
For comparison, we will perform two other tests. First, the Anderson-Darling test (A-D test), for data coming from a particular distribution:
Can you write Python code for it?

for N,y in zip(Nsamples, ysamples):
    A2, critical_values, significance_level = stats.anderson(y)
    print('{:4d} samples: A2={}'.format(N, A2), critical_values, significance_level)


I am working on the file "Kolmogorov_Smirnov_test.ipynb".
The first task is:
Let us check the versions being used:
Can you write Python code for it?

print('System: {}'.format(sys.version))
for package in (np, sp, mpl, pd):
    print('Package: {} {}'.format(package.__name__, package.__version__))


I am working on the file "Kolmogorov_Smirnov_test.ipynb".
The first task is:
Let us plot these results to see that they are pretty much scattered:
Can you write Python code for it?

fig, ax = plt.subplots(1, 1)
ax.scatter(psamples, Dsamples, s=Nsamples, alpha=0.2)
for p,D,N in zip(psamples, Dsamples, Nsamples):
    ax.text(p, D, str(N), ha='center', va='center')
ax.set_xlabel('p-value')
ax.set_ylabel('D')
ax.set_title('K-S test results')
plt.show()


I am working on the file "Kolmogorov_Smirnov_test.ipynb".
The first task is:
Now we are ready to start exploring.
Back to top
Data setup
First of all, we will define a normal random variable (RV) with mean 1.0 and standard deviation 0.1:
Can you write Python code for it?

mean = 1.0
std = 0.1
rv = stats.norm(loc=mean, scale=std)


I am working on the file "Kolmogorov_Smirnov_test.ipynb".
The first task is:
Now we will generate nine random samples, ranging from 10 to 5000 samples, and look at them in a histogram plot together with the normal pdf to see how they match:
Can you write Python code for it?

Nsamples = [u*d for d in (10,100,1000) for u in (1,2,5)]
ysamples = []
fig, ax = plt.subplots(3, 3, sharex=True, sharey=True, figsize=(8,6))
for row in range(3):
    for col in range(3):
        N = Nsamples[3*row+col]
        ax[row,col].plot(x, rv.pdf(x), 'k-', lw=2)
        y = rv.rvs(size=N)
        b = int(math.sqrt(N))
        ysamples.append(y)
        ax[row,col].hist(y, bins=b, normed=True, histtype='stepfilled', alpha=0.2)
        ax[row,col].set_title('{} samples, {} bins'.format(N, b))
plt.show()


I am working on the file "Kolmogorov_Smirnov_test.ipynb".
The first task is:
This is said to be a 'frozen' RV object since the location and scale parameters are given. Let us plot the probability density function (pdf) as well as the cumulative density function (cdf), between percentiles 0.001 and 0.999, to check the random variable:
Can you write Python code for it?

x = np.linspace(rv.ppf(0.001), rv.ppf(0.999), 1000)
fig, ax = plt.subplots(1, 1)
ax.plot(x, rv.pdf(x), label='frozen pdf')
ax.plot(x, rv.cdf(x), lw=2, label='frozen cdf')
ax.axhline(0.5, ls=':')
ax.axhline(1.0, ls=':')
ax.axvline(mean, ls='-.', label='mean')
ax.legend(loc='best', frameon=False)
plt.show()


I am working on the file "Kolmogorov_Smirnov_test.ipynb".
The first task is:
We used a different number of bins in the histogram plots in order to increase the discretization. As can be seen, the larger the size of the random sample, the better it fits the normal pdf. Similarly, we can plot the random samples, after ordering them in ascending order (empirical cumulative distribution function), together with the normal cdf:
Can you write Python code for it?

fig, ax = plt.subplots(3, 3, sharex=True, sharey=True, figsize=(8,6))
for row in range(3):
    for col in range(3):
        N = Nsamples[3*row+col]
        y = ysamples[3*row+col]
        ax[row,col].plot(x, rv.cdf(x), 'k-', lw=2)
        yn = np.sort(y)
        xn = np.linspace(0., 1., num=N)
        ax[row,col].plot(yn, xn, 'o', alpha=0.2)
        ax[row,col].axhline(0.5, ls=':')
        ax[row,col].axvline(mean, ls=':')
        ax[row,col].set_title('{} samples'.format(N))
plt.show()


I am working on the file "Korepanova/Korepanova_theanets.ipynb".
The first task is:
1.2 Specifying a Loss
All of the predefined models in theanets are created by default with one loss function appropriate for that type of model.
* Autoencoder: MSE between network's output and input
$$\mathcal{L}(X,\theta) = \frac{1}{mn}\sum_{i=1}^m\|F_\theta(x_i)-x_i\|2^2, \quad X \in \mathbb{R}^{m\times n}$$
* Regressor: MSE between true and predicted values of the target
$$\mathcal{L}(X, Y,\theta) = \frac{1}{mn}\sum{i=1}^m\|F_\theta(x_i)-y_i\|2^2, \quad X \in \mathbb{R}^{m\times n}, Y \in \mathbb{R}^{m\times o}$$
* Classifier: Cross-entropy between the network output and the true target labels
$$\mathcal{L}(X, Y,\theta) = \frac{1}{m}\sum{i=1}^m \sum_{j=1}^{k}\delta_{j, y_i}\log F_\theta(x_i)_j, \quad X \in \mathbb{R}^{m\times n}, Y \in {1,...,k}^m$$
For example, to use a mean-absolute error instead of the default mean-squared error for a regression model:
Can you write Python code for it?

net = theanets.Regressor([4, 5, 2], loss='mae')


I am working on the file "Korepanova/Korepanova_theanets.ipynb".
The first task is:
Computation Graphs
In theanets it is also possible to create network graphs that have arbitrary, acyclic connections among layers. Creating a nonlinear network graph requires using the inputs keyword argument when creating a layer.
Can you write Python code for it?

theanets.Classifier((
    784,
    dict(size=100, name='a'),
    dict(size=100, name='b'),
    dict(size=100, name='c'),
    dict(size=10, inputs=('a', 'b', 'c')),
))


I am working on the file "Korepanova/Korepanova_theanets.ipynb".
The first task is:
Custom Regularizers
To create a custom regularizer in theanets, you need to create a custom subclass of the theanets.Regularizer class, and then provide this regularizer when you run your model.
Example:
Can you write Python code for it?

class WeightInverse(theanets.Regularizer):
    def loss(self, layers, outputs):
        return sum((1 / (p * p).sum(axis=0)).sum()
                   for l in layers for p in l.params
                   if p.ndim == 2)

net = theanets.Autoencoder([4, (8, 'linear'), (4, 'tied')])
net.train(..., weightinverse=0.001)


I am working on the file "Korepanova/Korepanova_theanets.ipynb".
The first task is:
Examples
Can you write Python code for it?

import theanets
import numpy as np
from matplotlib import pyplot as plt
%matplotlib inline

mnist = np.loadtxt("/home/natalia/ML/mnist_train.csv", delimiter=",", skiprows=1)

train_X = (mnist[:30000, 1:]/255.).astype('f')
train_y = mnist[:30000, 0].astype('i')
valid_X = (mnist[30000:, 1:]/255.).astype('f')
valid_y = mnist[30000:, 0].astype('i')
print(train_X.shape, train_y.shape, valid_X.shape, valid_y.shape)


I am working on the file "Korepanova/Korepanova_theanets.ipynb".
The first task is:
Here, a classifier model is being trained using Nesterov’s accelerated gradient, with a learning rate of 0.01 and momentum of 0.9.
Multiple calls to train() are possible and can be used to implement things like custom annealing schedules (e.g., the “newbob” training strategy):
Can you write Python code for it?

net = theanets.Classifier(layers=[10, 5, 2])

for e in (-2, -3, -4):
    net.train(training_data,
              validation_data,
              algo='nag',
              learning_rate=10 ** e,
              momentum=1 - 10 ** (e + 1))


I am working on the file "Korepanova/Korepanova_theanets.ipynb".
The first task is:
If there is a string in the tuple that names a registered layer type (e.g., 'tied', 'rnn', etc.), then this type of layer will be created.
If there is a string in the tuple and it does not name a registered layer type, the string is assumed to name an activation function—for example, 'logistic', 'relu+norm:z', and so on.
Can you write Python code for it?

net = theanets.Regressor([4, (5, 'sigmoid'), (6, 'softmax')])

net = theanets.Regressor([10, (10, 'tanh+norm:z'), 10])


I am working on the file "Korepanova/Korepanova_theanets.ipynb".
The first task is:
In general, layers argument must be a sequence of values each of which specifies the configuration of a single layer in the model:
Can you write Python code for it?

net = theanets.Regressor([A, B, ..., Z])


I am working on the file "Korepanova/Korepanova_theanets.ipynb".
The first task is:
In this example, the weights in layer 1 connect the inputs to the first hidden layer; these weights have one column of 784 values for each hidden node in the network, so we can iterate over the transpose and put each column—properly reshaped—into a giant image.
Can you write Python code for it?

img = np.zeros((28 * 10, 28 * 10), dtype='f')
plt.figure(figsize = (8,8))
for i, pix in enumerate(net.find('hid1', 'w').get_value().T):
    r, c = divmod(i, 10)
    img[r * 28:(r+1) * 28, c * 28:(c+1) * 28] = pix.reshape((28, 28))
plt.imshow(img, cmap=plt.cm.gray)
plt.show()

net.train(train, valid, algo='nag', learning_rate=1e-3, momentum=0.9, weight_l1=1e-4)

for train, valid in net.itertrain([train_X, train_y], [valid_X, valid_y], algo='sgd',
                                  learning_rate=1e-2, momentum=0.9, min_improvement=0.1, patience=1):
    print('training loss:', train['loss'])
    print('most recent validation loss:', valid['loss'])


I am working on the file "Korepanova/Korepanova_theanets.ipynb".
The first task is:
MNIST Autoencoder (PCA)
Many extremely common dimensionality reduction techniques can be expressed as autoencoders. For instance, Principal Component Analysis (PCA) can be expressed as a model with two tied, linear layers:
Can you write Python code for it?

pca = theanets.Autoencoder([784, (10, 'linear'), (784, 'tied')])
pca.train(train_X, valid_X)

from utils import plot_images # from https://github.com/lmjohns3/theanets/tree/master/examples

v = valid_X[:100,:]

plt.figure(figsize = (10,10))
plot_images(v, 121, 'Sample data')
plt.tight_layout()
plot_images(pca.predict(v), 122, 'Reconstructed data')
plt.tight_layout()
plt.show()


I am working on the file "Korepanova/Korepanova_theanets.ipynb".
The first task is:
Regardless of the model, you pass to predict() a numpy array containing data examples along the rows, and the method returns an array containing one row of output predictions for each row of input data.
You can also compute the activations of all layer outputs in the network using the theanets.Network.feed_forward() method:
Can you write Python code for it?

for name, value in net.feed_forward(new_dataset).items():
    print(abs(value).sum(axis=1))


I am working on the file "Korepanova/Korepanova_theanets.ipynb".
The first task is:
This method returns a dictionary that maps layer output names to their corresponding values for the given input. Like predict(), each output array contains one row for every row of input data.
3.2 Inspecting Parameters
The parameters in each layer of the model are available using theanets.Network.find(). The first query term finds a layer in the network, and the second finds a parameter within that layer.
The find() method returns a theano shared variable. To get a numpy array of the current values of the variable, call get_value() on the result from find(), like so:
Can you write Python code for it?

param = net.find('hid1', 'w')
values = param.get_value()


I am working on the file "Korepanova/Korepanova_theanets.ipynb".
The first task is:
Trainers
In theanets the most of the trainers are provided by downhill package which provides algorithms for minimizing scalar loss functions that are defined using theano.
* sgd: Stochastic gradient descent
* nag: Nesterov’s accelerated gradient
* rprop: Resilient backpropagation
* rmsprop: RMSProp
* adadelta: ADADELTA
* esgd: Equilibrated SGD
* adam: Adam
Also theanets defines a few algorithms which are more specific to neural networks:
* sampler: This trainer sets model parameters directly to samples drawn from the training data. This is a very fast “training” algorithm since all updates take place at once; however, often features derived directly from the training data require further tuning to perform well.
* layerwise: Greedy supervised layerwise pre-training: This trainer applies RMSProp to each layer sequentially.
* pretrain: Greedy unsupervised layerwise pre-training: This trainer applies RMSProp to a tied-weights “shadow” autoencoder using an unlabeled dataset, and then transfers the learned autoencoder weights to the model being trained.
2.2 Providing Data
There are two ways of passing data: using numpy arrays and callables.
Instead of an array of data, you can provide a callable for a Dataset. This callable must take no arguments and must return a list of numpy arrays of the proper shape for your loss.
For example, this code defines a batch() helper that could be used for a loss that needs one input. The callable chooses a random dataset and a random offset for each batch:
Can you write Python code for it?

SOURCES = 'foo.npy', 'bar.npy', 'baz.npy'
BATCH_SIZE = 64

def batch():
    X = np.load(np.random.choice(SOURCES), mmap_mode='r')
    i = np.random.randint(len(X))
    return X[i:i+BATCH_SIZE]

net = theanets.Regressor(layers=[10, 5, 2])
net.train(train=batch, ...)


I am working on the file "Korepanova/Korepanova_theanets.ipynb".
The first task is:
sparsity: A float giving the proportion of parameter values in the layer that should be initialized to zero. Nonzero values in the parameters will be drawn from a Gaussian distribution and then an appropriate number of these parameter values will randomly be reset to zero to make the parameter “sparse.”
Custom Layers
To create a custom layer, just create a subclass of theanets.Layer and give it the functionality you want.
An example of a normal feedforward layer but without a bias term:
Can you write Python code for it?

import theanets
import theano.tensor as TT

class NoBias(theanets.Layer):
    # Transform the inputs for this layer into an output for the layer.
    def transform(self, inputs):
        return TT.dot(inputs, self.find('w'))

    #Helper method to create a new weight matrix.
    def setup(self):
        self.add_weights('w', nin=self.input_size, nout=self.size)

layer = theanets.Layer.build('nobias', size=4)
net = theanets.Autoencoder(layers=[4, (3, 'nobias', 'linear'), (4, 'tied', 'linear')])


I am working on the file "Kryptos.ipynb".
The first task is:

CHALLENGE: A Special Case

Break the following keyed vigenere ciphertext:

KWRGHGPEOKDBXGAWVMJDFVCVDHJZAMQGRNFHKWRLAVKWKWVFSCHLIVGRZCIHDUAWHPUFOKDNTLLEFCWHVBKBJOZRHVWCYVVGWAWADSDKNWGMBLUUPNKAHOHFZAJICCLGYKIMGZFKFODLIYCVHKHBIMGEFWEZHKZWVLMGAYDSDORZIYDJZMKBPLEMDGPSWAOJKAHPABWEXUGFIORNHHWSHWIHYOOLFFDUAODNUFOKDIQJTHIFTHIOIPNCWNDSDJOGIXTIFVAOSCDWVQTCDVRNQJUOIHKSHWIXNCHNPBTWENMNPKVNDSDGAWXNGRZCIOSFXHDDTAEMFHKAXZFRDJUWQJKWRKFHYGAWXBIGFKAYMJRLDQDKZEENMLOZHVYASWEFMPOZVOZKFWMMGEFWEOSORWGVDLRGJCMJNUVMTCXZAVQEEWKNGRFUYNTADWERMJNAEBOKUYXZJGRKYVMJZWESQDIYPLZHUCKBPLEMDOGRRLKVBEZWMFDTZVRNSWOKHMKAHMHVDKXZPBJJTPFFZHVVRNKURLDTIHMZIFKAHMAQKWRZHFMJOZYSQMRVHCGJNPFFZVYWVFMCVGHVVLOLMJTAUEDBJGWADSDPWHYNTEXUDNIGAWXJMJJICCLGYKIMGJZUFHIRWEEODEOKHFDAVOQYQGEIXNILOBIOKWHWIBXUAOKSZKSWJNDJTWKGLWRKIP

Case 2: Vigenere Key is Unknown, Alphabet Key is Known
Now, let's take the same ciphertext and suppose we only know the alphabet key.
Can you write Python code for it?

ctext = "BYDMXVPQAMCWFJGOTCCNXTKHOGSEIPAJTWGNBYDNBHCJWLBNCBHPNRNXNHSUOXHLTQQKYAUEMBOUUBWGNQVWUFKKBRVIITHVWNOWCSEXWOICJBPANTCNRPTNHSJVVLICIRSEIAPMQEEHIIVRETZGSEIQPLAXLXKOTWPCSOJGOSEEOKETKBDOOBHSJVPMOWHLTSHHPOVGVATKBSEVTFDVHONNQALNUKOXHPCDQKYAUZSPQPSKAGFPOQWBCWUHLPLMSAAZPBIECBXOTFMIKTITTICPOFTVHONKFIZDNCAJIVWDNXUWUHLCSEWTIHTXOECKBGUUMOHSUUCNLEQJETEOMABYDRUURUHLSWSJDAPMPPDNJIOBTCADIHYNBRQOGEFKQDAEBEHXDOFCNGXEFJVSELPZEHTSCLPPJAHEAPLEIZACVODENXXACTANMLAJITWXSORTPOWDDUEMGZITKOSHSNRONPYGCXVWNHVBUSNXDNZHKQTLWNEPKCJEUYLRXECNRATZEXBNHOCPQWUKYGNBYDTOTPUYAQALSKCNRAIRTITNBTPPLLCGTCEBTVOTWHGTXMIEHKQPNRTZKHYPFAMOPUJCDUOWCSEZCGQEMGVAJTBUHLSLITJVVLICIRSEIITSQFSNCDSSLGLRXKONNPRKAJVVAEABLGSJCJRLOWFOGQDHXTGEYTOIDLVHPITAND"

from itertools import permutations
from ngram_score import ngram_score

import re
import pprint as pp

qgram = ngram_score('ngrams/en_sherlock_4grams') # load our 4gram statistics
trigram = ngram_score('ngrams/en_sherlock_3grams') # load our 3gram statistics

# keep track of the 100 best keys
N=100
L=42 # only process L characters for speed
alphakey = "ENIGMA"

for KLEN in [9]:  #range(3,10):
    print "="*80
    rec = nbest(N)

    # exhaustively test all possible letters for first 3 entries of the key and keep track of the N best ones
    # if KLEN=7, this will test e.g. FOOAAAA and BARAAAA
    for i in permutations('ABCDEFGHIJKLMNOPQRSTUVWXYZ',3):
        i = "".join(i)
        key = ''.join(i) + 'A'*(KLEN-len(i))
        decrypted_ctext = keyed_vigenere(ctext[:L], key, alpha_key = alphakey, direction = -1)
        score = 0
        for j in range(0,len(ctext),KLEN):
            score += trigram.score(decrypted_ctext[j:j+3])
        rec.add((score,''.join(i), decrypted_ctext))
    next_rec = nbest(N)

    # for the remaining KLEN-3 characters of the key,
    for i in range(0,KLEN-3):
        # go over the N best keys found so far...
        for k in xrange(N):
            # ...and determine the best next character of the key, while keeping best N keys so far
            for c in...


I am working on the file "Kryptos.ipynb".
The first task is:

CHALLENGE: Using Chi-square criterion to decrypt Vigenere

Using the techniques described above, try deciphering these two messages:

TZKHT BTWJN BKGDP GVFVO HBTWA ZFFZP JVMJT VTHAV VAMFB BSBKU
JBUKL BDOKR MWZLH RHFFH WLEGF GISTR WBDSL VOHVY UQSWN PFOXR
WKKAN YBBXY NGHAR TJBZN BQZUE VUGUR MQOST VPBFA WWHWM OFFKU
AMQFI AFHVR WVUFE GZHYR AMTSS OFSEZ DKTKP RDICN CQAFA OPIKG
QMYWA AJBXB OBTWF BVFVA LZKHT REAVF BISWS VUPVN AAAXT UFTFH
AUQKS NHSJG QMRAR FUHYE NMTSV RCSVA BWXNE QXVZY NBTWF BVFKU
VMEKA TFFVZ JQZKA FPBVB OBTWM BTHWN VWGKU ATCCI NLOGD RTWEG
QMIGR YEHYR BKGDP GVFVP XVFAN HFGKB KMAXI AUSIR BBFGC EZDKN
WIXQS GTPFG QIYST RVFRA MXDGF RTGZB WIXOH BBFVN CBQEP GJBXG
XLQUI CISIG QMRGU EUVJR LBUGN GISRE CQELH NTGFS JZSAV ROHNB
LTGWS GPHYV BAQUT VPB

more difficult to break (due to the text not being very typical?):
IUOCA LGTAM DAQFN ISJTN MZGWS UZCBH GSSJT NM?LZ EYVKG LLZEE
BJVPK EAGOW VQUXI EMVZB ZWING GTUSL IOOOC AYSTH FJGLS FDTSS
PAEAT TFVWV VWRGS MWVVL OAOMP SFGWN MGEIL AONYV QMKDA NHDGG
CFOWB TQCLL HIT?L JMQKH OVDFQ LKBUS AGLGM TTIWT MKGME XZGZW
PWHPC PWOKT HFWZI ULLOD SVQGF ?ONMQ YELZI SXSUP AKLAT LOMKK
AGFPV PAJTY FAIPL VEGSW GAXAF TZKGD WFMIO MVMKK IXQGK VLXIV
FKGKG FDSOG TBZKE VFFVG KWVEO VGOJW ESFAI PLEIN VLGAX GRTZX
QCJKE CPFFA OWSTJ VDGJG WS

Points to consider:

 Without solving it, do you think this is a transposition cipher of English text? (hint: investigate letter frequencies)

 Without solving it, do you think this is a substitution cipher of English text? (hint: investigate the IC of the full text)

 In case it is a Vigenere cipher (spoiler: it is):

 what would be the mosdt likely key length? (hint: investigate the IC for different key lengths)

 which positions would be shifted by the same key/amount?

 what can you say about the plaintext language? (hint: investigate IC of subsequences)

 solve the key by hand using frequency distributions of the...
Can you write Python code for it?

# keep a list of the N best things we have seen, discard anything else
# this class can also be imported using 'import nbest' from a separate file nbest.py
class nbest(object):
    def __init__(self,N=1000):
        self.store = []
        self.N = N

    def add(self,item):
        self.store.append(item)
        self.store.sort(reverse=True)
        self.store = self.store[:self.N]

    def __getitem__(self,k):
        return self.store[k]

    def __len__(self):
        return len(self.store)

ctext = "VYCHVUYPESJMZCJZTXNOOEFSXMBQJTTNQAXWKBWTPDDKTUODCOPGJFNTMOPRLACBZGTWEAEEEOKWAKCXCHVOATLGFMVYZRWBNGHPQUCDRKSGXSGWZRZWMURLSTLCHLZWBAECCIMEESTLLPWVNCCMYLELPKAAPTCZKYCTZQOIKGICRMEQTSPWMGHKFTYOHRDCUBAQEQWTVRBPCFKGPWGGFEQTZFSDWDVCCLOYVPBFWGPVFPLYRTMCWVSDCCIHSBLGCLPWTVKPBNVNRLAVWVPQTOEACSYJIUVVPBXSFARC"


I am working on the file "Kryptos.ipynb".
The first task is:

CHALLENGE: aligning frequency distributions to determine the Caesar key

Use the slider above to align the frequency distributions and determine the right key.

Caesar Cipher Solution #3: Chi-squared Statistic
We just saw that finding the right key consists of aligning the letter-frequency distributions. If we can measure how well aligned the distributions are, it can be done automatically. This is where the Chi-squared Statistic comes in: it measures how different an observed distribution is from an expected distribution:
$$
\chi^2(O,E) = \sum_{i=A}^Z \frac{(O_i - E_i)^2}{E_i}
$$
where $O_i$ and $E_i$ are the observed and expected numbers of character $i$ in a text. It can also be expressed in terms of frequencies/probabilities:
$$
\chi^2 = N \sum_{i=A}^Z \frac{(O_i/N - p_i)^2}{p_i}
= N \sum_{i=A}^Z \frac{(f_i - p_i)^2}{p_i}
$$
where $f_i$ and $p_i$ are the observed and expected relative frequencies of the characters, and $N$ is the length of the text.
For more details see:
- http://practicalcryptography.com/cryptanalysis/text-characterisation/chi-squared-statistic/
- https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test
Now, let's use it to find the key:
Can you write Python code for it?

import numpy as np

# english letter frequencies, estimated from small sample text # FIXME: replace by better estimates
g_english = [0.0736, 0.0148, 0.0445, 0.0302, 0.102, 0.0227, 0.0122, 0.0277, 0.0855, 0.000557, 0.00237, 0.0342, 0.0206, 0.0717, 0.103, 0.0246, 0.00181, 0.0735, 0.0608, 0.0889, 0.0392, 0.0153, 0.0173, 0.000557, 0.032, 0.000278]

def chi_square(f, g, N):
    chi2 = 0
    chi2 = N * np.sum(np.array([(ff-gg)**2/gg for (ff,gg) in zip(f,g)]))
    return chi2

alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
results = []
for key in alphabet:
    decrypted_ctext = rot(ctext, key, direction = -1)
    f = np.array([decrypted_ctext.count(c) for c in alphabet])
    N = np.sum(f) # note: not counting the non-alphabet characters here
    f = f / N
    chi2_decrypted_ctext = chi_square(f, g_english, N)
    results.append((chi2_decrypted_ctext, key, decrypted_ctext))
    print "%s:\t%f\t%s" % (key, chi2_decrypted_ctext, decrypted_ctext)
print "\nthe best key and plaintext:\n %s" % repr(min(results))


I am working on the file "Kryptos.ipynb".
The first task is:
According to the Chi-square analysis the key is OCYPTANALYSIS.
Can you write Python code for it?

vigenere(ctext, 'OCYPTANALYSIS', direction = -1)