I am working on the file "07_autoencoder/Convolutional_Autoencoder.ipynb".
The first task is:
Network Architecture
The encoder part of the network will be a typical convolutional pyramid. Each convolutional layer will be followed by a max-pooling layer to reduce the dimensions of the layers. The decoder though might be something new to you. The decoder needs to convert from a narrow representation to a wide reconstructed image. For example, the representation could be a 4x4x8 max-pool layer. This is the output of the encoder, but also the input to the decoder. We want to get a 28x28x1 image out from the decoder so we need to work our way back up from the narrow decoder input layer. A schematic of the network is shown below.
<img src='assets/convolutional_autoencoder.png' width=500px>
Here our final encoder layer has size 4x4x8 = 128. The original images have size 28x28 = 784, so the encoded vector is roughly 16% the size of the original image. These are just suggested sizes for each of the layers. Feel free to change the depths and sizes, but remember our goal here is to find a small representation of the input data.
What's going on with the decoder
Okay, so the decoder has these "Upsample" layers that you might not have seen before. First off, I'll discuss a bit what these layers aren't. Usually, you'll see transposed convolution layers used to increase the width and height of the layers. They work almost exactly the same as convolutional layers, but in reverse. A stride in the input layer results in a larger stride in the transposed convolution layer. For example, if you have a 3x3 kernel, a 3x3 patch in the input layer will be reduced to one unit in a convolutional layer. Comparatively, one unit in the input layer will be expanded to a 3x3 path in a transposed convolution layer. The TensorFlow API provides us with an easy way to create the layers, tf.nn.conv2d_transpose.
However, transposed convolution layers can lead to artifacts in the final images, such as checkerboard patterns. This is due to overlap in the kernels which can be avoided by setting the stride and kernel size equal. In this...
Can you write Python code for it?

learning_rate = 0.001
# Input and target placeholders
inputs_ = tf.placeholder(tf.float32, [None, 28, 28, 1], name = 'input')
targets_ = tf.placeholder(tf.float32, [None, 28, 28, 1], name = 'targets')

### Encoder
conv1 = tf.layers.conv2d(inputs_, 16, 3, 1, padding='same', activation=tf.nn.relu)
# Now 28x28x16
maxpool1 = tf.layers.max_pooling2d(conv1, 2, 2, padding='valid')
# Now 14x14x16
conv2 = tf.layers.conv2d(maxpool1, 8, 3, 1, padding='same', activation=tf.nn.relu)
# Now 14x14x8
maxpool2 = tf.layers.max_pooling2d(conv2, 2, 2, padding='valid')
# Now 7x7x8
conv3 = tf.layers.conv2d(maxpool2, 8, 3, 1, padding='same', activation=tf.nn.relu)
# Now 7x7x8
encoded = tf.layers.max_pooling2d(conv3, 2, 2, padding='same')
# Now 4x4x8

### Decoder
upsample1 = tf.image.resize_nearest_neighbor(encoded, [7, 7])
# Now 7x7x8
conv4 = tf.layers.conv2d(upsample1, 8, 3, 1, padding='same', activation=tf.nn.relu)
# Now 7x7x8
upsample2 = tf.image.resize_nearest_neighbor(conv4, [14, 14])
# Now 14x14x8
conv5 = tf.layers.conv2d(upsample2, 8, 3, 1, padding='same', activation=tf.nn.relu)
# Now 14x14x8
upsample3 = tf.image.resize_nearest_neighbor(conv5, [28, 28])
# Now 28x28x8
conv6 = tf.layers.conv2d(upsample3, 16, 3, 1, padding='same', activation=tf.nn.relu)
# Now 28x28x16

logits = tf.layers.conv2d(upsample3, 1, 3, 1, padding='same')
#Now 28x28x1

# Pass logits through sigmoid to get reconstructed image
decoded = tf.sigmoid(logits)

# Pass logits through sigmoid and calculate the cross-entropy loss
loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=targets_, logits=logits)

# Get cost and define the optimizer
cost = tf.reduce_mean(loss)
opt = tf.train.AdamOptimizer(learning_rate).minimize(cost)


I am working on the file "07_autoencoder/Convolutional_Autoencoder.ipynb".
The first task is:
Training
As before, here we'll train the network. Instead of flattening the images though, we can pass them in as 28x28x1 arrays.
Can you write Python code for it?

sess = tf.Session()

epochs = 20
batch_size = 200
sess.run(tf.global_variables_initializer())
for e in range(epochs):
    for ii in range(mnist.train.num_examples//batch_size):
        batch = mnist.train.next_batch(batch_size)
        imgs = batch[0].reshape((-1, 28, 28, 1))
        batch_cost, _ = sess.run([cost, opt], feed_dict={inputs_: imgs,
                                                         targets_: imgs})

        print("Epoch: {}/{}...".format(e+1, epochs),
              "Training loss: {:.4f}".format(batch_cost))

fig, axes = plt.subplots(nrows=2, ncols=10, sharex=True, sharey=True, figsize=(20,4))
in_imgs = mnist.test.images[:10]
reconstructed = sess.run(decoded, feed_dict={inputs_: in_imgs.reshape((10, 28, 28, 1))})

for images, row in zip([in_imgs, reconstructed], axes):
    for img, ax in zip(images, row):
        ax.imshow(img.reshape((28, 28)), cmap='Greys_r')
        ax.get_xaxis().set_visible(False)
        ax.get_yaxis().set_visible(False)

fig.tight_layout(pad=0.1)

sess.close()


I am working on the file "07_autoencoder/Simple_Autoencoder.ipynb".
The first task is:
Below I'm plotting an example image from the MNIST dataset. These are 28x28 grayscale images of handwritten digits.
Can you write Python code for it?

img = mnist.train.images[0]
plt.imshow(img.reshape((28, 28)), cmap='Greys_r')

img


I am working on the file "07_autoencoder/Simple_Autoencoder.ipynb".
The first task is:
Here I'll write a bit of code to train the network. I'm not too interested in validation here, so I'll just monitor the training loss.
Calling mnist.train.next_batch(batch_size) will return a tuple of (images, labels). We're not concerned with the labels here, we just need the images. Otherwise this is pretty straightfoward training with TensorFlow. We initialize the variables with sess.run(tf.global_variables_initializer()). Then, run the optimizer and get the loss with batch_cost, _ = sess.run([cost, opt], feed_dict=feed).
Can you write Python code for it?

epochs = 20
batch_size = 200
sess.run(tf.global_variables_initializer())
for e in range(epochs):
    for ii in range(mnist.train.num_examples//batch_size):
        batch = mnist.train.next_batch(batch_size)
        feed = {inputs_: batch[0], targets_: batch[0]}
        batch_cost, _ = sess.run([cost, opt], feed_dict=feed)

        print("Epoch: {}/{}...".format(e+1, epochs),
              "Training loss: {:.4f}".format(batch_cost))


I am working on the file "07_autoencoder/Simple_Autoencoder.ipynb".
The first task is:
We'll train an autoencoder with these images by flattening them into 784 length vectors. The images from this dataset are already normalized such that the values are between 0 and 1. Let's start by building basically the simplest autoencoder with a single ReLU hidden layer. This layer will be used as the compressed representation. Then, the encoder is the input layer and the hidden layer. The decoder is the hidden layer and the output layer. Since the images are normalized between 0 and 1, we need to use a sigmoid activation on the output layer to get values matching the input.

Exercise: Build the graph for the autoencoder in the cell below. The input images will be flattened into 784 length vectors. The targets are the same as the inputs. And there should be one hidden layer with a ReLU activation and an output layer with a sigmoid activation. Feel free to use TensorFlow's higher level API, tf.layers. For instance, you would use tf.layers.dense(inputs, units, activation=tf.nn.relu) to create a fully connected layer with a ReLU activation. The loss should be calculated with the cross-entropy loss, there is a convenient TensorFlow function for this tf.nn.sigmoid_cross_entropy_with_logits (documentation). You should note that tf.nn.sigmoid_cross_entropy_with_logits takes the logits, but to get the reconstructed images you'll need to pass the logits through the sigmoid function.
Can you write Python code for it?

tf.reset_default_graph()

# Size of the encoding layer (the hidden layer)
encoding_dim = 32 # feel free to change this value

# Input and target placeholders
inputs_ = tf.placeholder(tf.float32, [None, 28*28], 'input')
targets_ =  tf.placeholder(tf.float32, [None, 28*28], 'target')

# Output of hidden layer, single fully connected layer here with ReLU activation
encoded = tf.layers.dense(inputs_, encoding_dim, activation=tf.nn.relu, name='encoded')

# Output layer logits, fully connected layer with no activation
logits = tf.layers.dense(encoded, 28*28, activation=None, name='logits')

# Sigmoid output from logits
decoded = tf.sigmoid(logits, name='decoded')

# Sigmoid cross-entropy loss
loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=targets_, logits=logits, name='loss')

# Mean of the loss
cost = tf.reduce_mean(loss, name='cost')

# Adam optimizer
opt = tf.train.AdamOptimizer(0.0005).minimize(cost)


I am working on the file "07_duplicates/simhash.ipynb".
The first task is:
Как видим, с течением времени в размере групп не наблюдается какого-либо тренда.
Can you write Python code for it?

plt.figure(figsize=(12,8))
plt.hist(groups_sizes, bins=100, log=True)
plt.xlabel("Group size")
plt.ylabel("Number of groups");


I am working on the file "07_duplicates/simhash.ipynb".
The first task is:
Отработав 6 часов, скрипт обещал работать еще около 9 суток, что слишком долго. Поэтому проведём анализ уже полученных данных как репрезентативной выборки.
Can you write Python code for it?

groups_sizes = np.array(groups_sizes)

plt.figure(figsize=(12,8))
plt.plot(groups_sizes);
plt.xlabel("Group ID")
plt.ylabel("Group size");


I am working on the file "07_duplicates/simhash.ipynb".
The first task is:
Поделим simhash-и на 4 части для индексирования.
Can you write Python code for it?

MAX_DISTANCE = 3
NUM_PARTS = MAX_DISTANCE + 1
PART_SIZE = SIMHASH_SIZE / NUM_PARTS

neg_part_mask = "0" * PART_SIZE
pos_part_mask = "1" * PART_SIZE
masks = [neg_part_mask * part_id + pos_part_mask + neg_part_mask * (NUM_PARTS - part_id - 1)\
         for part_id in range(NUM_PARTS)]
masks = np.array([int(mask, 2) for mask in masks], dtype=np.uint64)

def get_part(simhash, part_id):
    return int(simhash & masks[part_id]) >> (PART_SIZE * (NUM_PARTS - part_id - 1))

%%time
simhashes_parts = np.zeros((len(simhashes), NUM_PARTS), dtype=np.int32)
for simhash_id, simhash in enumerate(simhashes):
    for part_id in xrange(NUM_PARTS):
        simhashes_parts[simhash_id][part_id] = get_part(simhash, part_id)


I am working on the file "07_duplicates/simhash.ipynb".
The first task is:
Построим индексы.
Can you write Python code for it?

%%time
indices = [[list() for __ in xrange(2 ** PART_SIZE)] for _ in xrange(NUM_PARTS)]
for simhash_id in xrange(num_samples):
    simhash_parts = simhashes_parts[simhash_id]
    for part_id in xrange(NUM_PARTS):
        indices[part_id][simhash_parts[part_id]].append(simhash_id)


I am working on the file "07_duplicates/simhash.ipynb".
The first task is:
Прокластеризуем хеши.
Can you write Python code for it?

def ones_positions(num_ones, size=SIMHASH_SIZE):
    if num_ones == 0:
        yield []
        return
    for position in range(size):
        for positions in ones_positions(num_ones - 1, size):
            yield [position] + positions

accepted_xors = set()

for num_ones in xrange(MAX_DISTANCE + 1):
    for positions in ones_positions(num_ones):
        xor = ["0"] * SIMHASH_SIZE
        for pos in positions:
            xor[pos] = "1"
        accepted_xors.add(np.uint64(int("".join(xor), 2)))

print len(accepted_xors)

def similar(hash1, hash2):
    return (hash1 ^ hash2) in accepted_xors

%%time
groups_sizes = []
assigned = [False] * num_samples  # indicators of simhashes assigned to any of the considered groups
num_assigned = 0

start = time()

for simhash_id, simhash in enumerate(simhashes):
    if assigned[simhash_id]:
        continue

    group_size = 0
    simhash_parts = simhashes_parts[simhash_id]
    for part_id, part in enumerate(simhash_parts):
        for candidate_id in indices[part_id][part]:
            if assigned[candidate_id]:
                continue
            if similar(simhash, simhashes[candidate_id]):
                group_size += 1
                assigned[candidate_id] = True #.add(candidate_id)
                num_assigned += 1

    groups_sizes.append(group_size)

    if simhash_id % 3000 == 0:
        spent = time() - start
        clear_output()
        print "assigned: {}\tRemained time: {:.2f} days".format(
            num_assigned,(float(num_samples) / num_assigned - 1) * spent / 60 / 60 / 24)


I am working on the file "07_dx_portfolio_risk.ipynb".
The first task is:
Derivatives Positions
Two different kinds of derivatives make up the portfolio---an American put option and a European maximum call option. Both types of derivatives populate three positions, respectively.
Can you write Python code for it?

positions = {}
half = 3  # 2 times that many options
for i in range(half):
    name = 'am_put_pos_%s' %i  # same name for position key and name
    positions[name] = dx.derivatives_position(
                        name=name,
                        quantity=1,
                        underlyings=['gbm_1'],
                        mar_env=me_option,
                        otype='American single',
                        payoff_func='np.maximum(instrument_values - 40., 0)')

multi_payoff = "np.maximum(np.maximum(maturity_value['gbm_1'], maturity_value['gbm_2']) - 40., 0)"
for i in range(half, 2 * half):
    name = 'multi_pos_%s' %i  # same name for position key and name
    positions[name] = dx.derivatives_position(
                        name=name,
                        quantity=1,
                        underlyings=['gbm_1', 'gbm_2'],
                        mar_env=me_option,
                        otype='European multi',
                        payoff_func=multi_payoff)


I am working on the file "07_dx_portfolio_risk.ipynb".
The first task is:
Derivatives Positions
We are going to model total of 6 derivatives positions.
Market Environment
All derivatives instruments (positions) share the same market_environment object.
Can you write Python code for it?

# market environment for the options
me_option = dx.market_environment('put', dt.datetime(2015, 1, 1))
me_option.add_constant('maturity', dt.datetime(2015, 12, 31))
me_option.add_constant('currency', 'EUR')
me_option.add_environment(val_env)


I am working on the file "07_dx_portfolio_risk.ipynb".
The first task is:
For example, increasing the initial value of the first risk factor (gbm_1) by 10% increases the portfolio value by about 11 currency units.
Can you write Python code for it?

deltas

deltas.loc(axis=0)[:, 'value'] - benchvalue


dx.risk_report(deltas)

dx.risk_report(deltas.ix[:, :, 'value'] - benchvalue)


I am working on the file "07_dx_portfolio_risk.ipynb".
The first task is:
Here, the value estimates from the Monte Carlo simulation and valuation.
Can you write Python code for it?


I am working on the file "07_dx_portfolio_risk.ipynb".
The first task is:
Of course, you can generate the same risk report for the portfolio initial value sensitivities.
Can you write Python code for it?

%time deltas, benchvalue = portfolio.get_port_risk(Greek='Delta', fixed_seed=True)


I am working on the file "07_dx_portfolio_risk.ipynb".
The first task is:
Portfolio Risk Reports
Portfolio risk reports are meant to provide a broad overview of how sensitive the value of a portfolio is with regard to the value of certain input parameters (market data, model parameters). While Greeks provide the same information with regard to marginal changes in the input paramters, risk reports provide a wider range input-output (parameter-portfolio value) combinations.
No Correlation
First, consider the portfolio from before, i.e. without correlation.
Can you write Python code for it?


I am working on the file "07_dx_portfolio_risk.ipynb".
The first task is:
Risk Factors
The example is based on two risk factors, both modeled as geometric Brownian motions.
Can you write Python code for it?

# constant short rate
r = dx.constant_short_rate('r', 0.01)

# market environment
me_gbm_1 = dx.market_environment('gbm_1', dt.datetime(2015, 1, 1))

# geometric Brownian motion
me_gbm_1.add_constant('initial_value', 40.)
me_gbm_1.add_constant('volatility', 0.2)
me_gbm_1.add_constant('currency', 'EUR')
me_gbm_1.add_constant('model', 'gbm')

me_gbm_2 = dx.market_environment('gbm_2', me_gbm_1.pricing_date)

# valuation environment
val_env = dx.market_environment('val_env', dt.datetime(2015, 1, 1))
val_env.add_constant('paths', 25000)
    # 25,000 paths
val_env.add_constant('frequency', 'W')
    # weekly frequency
val_env.add_curve('discount_curve', r)
val_env.add_constant('starting_date', dt.datetime(2015, 1, 1))
val_env.add_constant('final_date', dt.datetime(2015, 12, 31))

# add valuation environment to market environments
me_gbm_1.add_environment(val_env)
me_gbm_2.add_environment(me_gbm_1)
me_gbm_2.add_constant('initial_value', 40.)
me_gbm_2.add_constant('volatility', 0.5)
  # higher volatility

risk_factors = {'gbm_1' : me_gbm_1, 'gbm_2' : me_gbm_2}
  # market with two risk factors


I am working on the file "07_dx_portfolio_risk.ipynb".
The first task is:
Since the value of the European maximum call option is dependent on the risk factor correlation you see a significant change in this derivative's value estimate.
Can you write Python code for it?


I am working on the file "07_dx_portfolio_risk.ipynb".
The first task is:
The return object is a pandas Panel object.
Can you write Python code for it?


I am working on the file "07_dx_portfolio_risk.ipynb".
The first task is:
Using the helper funtion risk_report allows the easy, readable printout of the results, i.e. the portfolio volatility sensitivities. In this case you can see that, for example, the increase in the first risk fator's (gbm_1) volatility by 10% leads to a portfolio value increase bya bit less than 1 currency unit. Decreasing the same input parameter by 10% reduces the portfolio value by a bit less than 1 currency unit.
Can you write Python code for it?


I am working on the file "07_dx_portfolio_risk.ipynb".
The first task is:
Via the step parameter, you can influence the  granularity of the risk report.
Can you write Python code for it?

%%time
deltas, benchvalue = portfolio.get_port_risk(Greek='Delta',
                                 fixed_seed=True,
                                 step=0.05)


I am working on the file "07_dx_portfolio_risk.ipynb".
The first task is:
With Correlation
Consider now a highly negative correlation case.
Can you write Python code for it?

correlations = [['gbm_1', 'gbm_2', -0.9]]

portfolio = dx.derivatives_portfolio(
                        'portfolio', positions, val_env,
                        risk_factors, correlations, parallel=False)

portfolio.val_env.get_list('cholesky_matrix')


I am working on the file "07_ensemble_learning_and_random_forests.ipynb".
The first task is:
A bit better! The SVM was hurting performance. Now let's try using a soft voting classifier. We do not actually need to retrain the classifier, we can just set voting to "soft":
Can you write Python code for it?

voting_clf.voting = "soft"

voting_clf.score(X_val, y_val)


I am working on the file "07_ensemble_learning_and_random_forests.ipynb".
The first task is:
Bagging ensembles
Can you write Python code for it?

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

bag_clf = BaggingClassifier(
    DecisionTreeClassifier(random_state=42), n_estimators=500,
    max_samples=100, bootstrap=True, n_jobs=-1, random_state=42)
bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)

from sklearn.metrics import accuracy_score
print(accuracy_score(y_test, y_pred))

tree_clf = DecisionTreeClassifier(random_state=42)
tree_clf.fit(X_train, y_train)
y_pred_tree = tree_clf.predict(X_test)
print(accuracy_score(y_test, y_pred_tree))

from matplotlib.colors import ListedColormap

def plot_decision_boundary(clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.5, contour=True):
    x1s = np.linspace(axes[0], axes[1], 100)
    x2s = np.linspace(axes[2], axes[3], 100)
    x1, x2 = np.meshgrid(x1s, x2s)
    X_new = np.c_[x1.ravel(), x2.ravel()]
    y_pred = clf.predict(X_new).reshape(x1.shape)
    custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
    plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap, linewidth=10)
    if contour:
        custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
        plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)
    plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", alpha=alpha)
    plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", alpha=alpha)
    plt.axis(axes)
    plt.xlabel(r"$x_1$", fontsize=18)
    plt.ylabel(r"$x_2$", fontsize=18, rotation=0)

plt.figure(figsize=(11,4))
plt.subplot(121)
plot_decision_boundary(tree_clf, X, y)
plt.title("Decision Tree", fontsize=14)
plt.subplot(122)
plot_decision_boundary(bag_clf, X, y)
plt.title("Decision Trees with Bagging", fontsize=14)
save_fig("decision_tree_without_and_with_bagging_plot")
plt.show()


from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

bag_clf = BaggingClassifier(
    DecisionTreeClassifier(random_state=42), n_estimators=500,
    max_samples=100, bootstrap=True, n_jobs=-1, random_state=42)
bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)

from sklearn.metrics import accuracy_score
print(accuracy_score(y_test, y_pred))

tree_clf = DecisionTreeClassifier(random_state=42)
tree_clf.fit(X_train, y_train)
y_pred_tree = tree_clf.predict(X_test)
print(accuracy_score(y_test, y_pred_tree))

from matplotlib.colors import ListedColormap

def plot_decision_boundary(clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.5, contour=True):
    x1s = np.linspace(axes[0], axes[1], 100)
    x2s = np.linspace(axes[2], axes[3], 100)
    x1, x2 = np.meshgrid(x1s, x2s)
    X_new = np.c_[x1.ravel(), x2.ravel()]
    y_pred = clf.predict(X_new).reshape(x1.shape)
    custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
    plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
    if contour:
        custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
        plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)
    plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", alpha=alpha)
    plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", alpha=alpha)
    plt.axis(axes)
    plt.xlabel(r"$x_1$", fontsize=18)
    plt.ylabel(r"$x_2$", fontsize=18, rotation=0)

plt.figure(figsize=(11,4))
plt.subplot(121)
plot_decision_boundary(tree_clf, X, y)
plt.title("Decision Tree", fontsize=14)
plt.subplot(122)
plot_decision_boundary(bag_clf, X, y)
plt.title("Decision Trees with Bagging", fontsize=14)
save_fig("decision_tree_without_and_with_bagging_plot")
plt.show()


I am working on the file "07_ensemble_learning_and_random_forests.ipynb".
The first task is:
Exercise solutions
1. to 7.
See Appendix A.
8. Voting Classifier
Exercise: Load the MNIST data and split it into a training set, a validation set, and a test set (e.g., use 50,000 instances for training, 10,000 for validation, and 10,000 for testing).
The MNIST dataset was loaded earlier.
Can you write Python code for it?

from sklearn.model_selection import train_test_split

X_train_val, X_test, y_train_val, y_test = train_test_split(
    mnist.data, mnist.target, test_size=10000, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val, y_train_val, test_size=10000, random_state=42)


I am working on the file "07_ensemble_learning_and_random_forests.ipynb".
The first task is:
Feature importance
Can you write Python code for it?

try:
    from sklearn.datasets import fetch_openml
    mnist = fetch_openml('mnist_784', version=1, as_frame=False)
    mnist.target = mnist.target.astype(np.int64)
except ImportError:
    from sklearn.datasets import fetch_mldata
    mnist = fetch_mldata('MNIST original')

rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
rnd_clf.fit(mnist["data"], mnist["target"])

def plot_digit(data):
    image = data.reshape(28, 28)
    plt.imshow(image, cmap = mpl.cm.hot,
               interpolation="nearest")
    plt.axis("off")

plot_digit(rnd_clf.feature_importances_)

cbar = plt.colorbar(ticks=[rnd_clf.feature_importances_.min(), rnd_clf.feature_importances_.max()])
cbar.ax.set_yticklabels(['Not important', 'Very important'])

save_fig("mnist_feature_importance_plot")
plt.show()


I am working on the file "07_ensemble_learning_and_random_forests.ipynb".
The first task is:
Gradient Boosting
Can you write Python code for it?

np.random.seed(42)
X = np.random.rand(100, 1) - 0.5
y = 3*X[:, 0]**2 + 0.05 * np.random.randn(100)

from sklearn.tree import DecisionTreeRegressor

tree_reg1 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg1.fit(X, y)

y2 = y - tree_reg1.predict(X)
tree_reg2 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg2.fit(X, y2)

y3 = y2 - tree_reg2.predict(X)
tree_reg3 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg3.fit(X, y3)

X_new = np.array([[0.8]])

y_pred = sum(tree.predict(X_new) for tree in (tree_reg1, tree_reg2, tree_reg3))

y_pred

def plot_predictions(regressors, X, y, axes, label=None, style="r-", data_style="b.", data_label=None):
    x1 = np.linspace(axes[0], axes[1], 500)
    y_pred = sum(regressor.predict(x1.reshape(-1, 1)) for regressor in regressors)
    plt.plot(X[:, 0], y, data_style, label=data_label)
    plt.plot(x1, y_pred, style, linewidth=2, label=label)
    if label or data_label:
        plt.legend(loc="upper center", fontsize=16)
    plt.axis(axes)

plt.figure(figsize=(11,11))

plt.subplot(321)
plot_predictions([tree_reg1], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="$h_1(x_1)$", style="g-", data_label="Training set")
plt.ylabel("$y$", fontsize=16, rotation=0)
plt.title("Residuals and tree predictions", fontsize=16)

plt.subplot(322)
plot_predictions([tree_reg1], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="$h(x_1) = h_1(x_1)$", data_label="Training set")
plt.ylabel("$y$", fontsize=16, rotation=0)
plt.title("Ensemble predictions", fontsize=16)

plt.subplot(323)
plot_predictions([tree_reg2], X, y2, axes=[-0.5, 0.5, -0.5, 0.5], label="$h_2(x_1)$", style="g-", data_style="k+", data_label="Residuals")
plt.ylabel("$y - h_1(x_1)$", fontsize=16)

plt.subplot(324)
plot_predictions([tree_reg1, tree_reg2], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="$h(x_1) = h_1(x_1) + h_2(x_1)$")
plt.ylabel("$y$", fontsize=16, rotation=0)

plt.subplot(325)
plot_predictions([tree_reg3], X, y3, axes=[-0.5, 0.5, -0.5, 0.5], label="$h_3(x_1)$", style="g-",...


I am working on the file "07_ensemble_learning_and_random_forests.ipynb".
The first task is:
However, it did not update the list of trained estimators:
Can you write Python code for it?


I am working on the file "07_ensemble_learning_and_random_forests.ipynb".
The first task is:
Let's remove the SVM to see if performance improves. It is possible to remove an estimator by setting it to None using set_params() like this:
Can you write Python code for it?


I am working on the file "07_ensemble_learning_and_random_forests.ipynb".
The first task is:
Now let's evaluate the VotingClassifier again:
Can you write Python code for it?


I am working on the file "07_ensemble_learning_and_random_forests.ipynb".
The first task is:
Out-of-Bag evaluation
Can you write Python code for it?

bag_clf = BaggingClassifier(
    DecisionTreeClassifier(random_state=42), n_estimators=500,
    bootstrap=True, n_jobs=-1, oob_score=True, random_state=40)
bag_clf.fit(X_train, y_train)
bag_clf.oob_score_

bag_clf.oob_decision_function_

from sklearn.metrics import accuracy_score
y_pred = bag_clf.predict(X_test)
accuracy_score(y_test, y_pred)


I am working on the file "07_ensemble_learning_and_random_forests.ipynb".
The first task is:
Random Forests
Can you write Python code for it?

bag_clf = BaggingClassifier(
    DecisionTreeClassifier(splitter="random", max_leaf_nodes=16, random_state=42),
    n_estimators=500, max_samples=1.0, bootstrap=True, n_jobs=-1, random_state=42)

bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)

from sklearn.ensemble import RandomForestClassifier

rnd_clf = RandomForestClassifier(n_estimators=500, max_leaf_nodes=16, n_jobs=-1, random_state=42)
rnd_clf.fit(X_train, y_train)

y_pred_rf = rnd_clf.predict(X_test)

np.sum(y_pred == y_pred_rf) / len(y_pred)  # almost identical predictions

from sklearn.datasets import load_iris
iris = load_iris()
rnd_clf = RandomForestClassifier(n_estimators=500, n_jobs=-1, random_state=42)
rnd_clf.fit(iris["data"], iris["target"])
for name, score in zip(iris["feature_names"], rnd_clf.feature_importances_):
    print(name, score)

rnd_clf.feature_importances_

plt.figure(figsize=(6, 4))

for i in range(15):
    tree_clf = DecisionTreeClassifier(max_leaf_nodes=16, random_state=42 + i)
    indices_with_replacement = np.random.randint(0, len(X_train), len(X_train))
    tree_clf.fit(X[indices_with_replacement], y[indices_with_replacement])
    plot_decision_boundary(tree_clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.02, contour=False)

plt.show()


I am working on the file "07_ensemble_learning_and_random_forests.ipynb".
The first task is:
So we can either fit the VotingClassifier again, or just remove the SVM from the list of trained estimators:
Can you write Python code for it?


I am working on the file "07_ensemble_learning_and_random_forests.ipynb".
The first task is:
That's a significant improvement, and it's much better than each of the individual classifiers.
Once you have found one, try it on the test set. How much better does it perform compared to the individual classifiers?
Can you write Python code for it?

voting_clf.score(X_test, y_test)

[estimator.score(X_test, y_test) for estimator in voting_clf.estimators_]


I am working on the file "07_ensemble_learning_and_random_forests.ipynb".
The first task is:
The voting classifier reduced the error rate from about 4.0% for our best model (the MLPClassifier) to just 3.1%. That's about 22.5% less errors, not bad!
9. Stacking Ensemble
Exercise: Run the individual classifiers from the previous exercise to make predictions on the validation set, and create a new training set with the resulting predictions: each training instance is a vector containing the set of predictions from all your classifiers for an image, and the target is the image's class. Train a classifier on this new training set.
Can you write Python code for it?

X_val_predictions = np.empty((len(X_val), len(estimators)), dtype=np.float32)

for index, estimator in enumerate(estimators):
    X_val_predictions[:, index] = estimator.predict(X_val)

X_val_predictions

rnd_forest_blender = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=42)
rnd_forest_blender.fit(X_val_predictions, y_val)

rnd_forest_blender.oob_score_


I am working on the file "07_ensemble_learning_and_random_forests.ipynb".
The first task is:
Using XGBoost
Can you write Python code for it?

try:
    import xgboost
except ImportError as ex:
    print("Error: the xgboost library is not installed.")
    xgboost = None

if xgboost is not None:  # not shown in the book
    xgb_reg = xgboost.XGBRegressor(random_state=42)
    xgb_reg.fit(X_train, y_train)
    y_pred = xgb_reg.predict(X_val)
    val_error = mean_squared_error(y_val, y_pred)
    print("Validation MSE:", val_error)

if xgboost is not None:  # not shown in the book
    xgb_reg.fit(X_train, y_train,
                eval_set=[(X_val, y_val)], early_stopping_rounds=2)
    y_pred = xgb_reg.predict(X_val)
    val_error = mean_squared_error(y_val, y_pred)
    print("Validation MSE:", val_error)

%timeit xgboost.XGBRegressor().fit(X_train, y_train) if xgboost is not None else None

%timeit GradientBoostingRegressor().fit(X_train, y_train)


I am working on the file "07_ensemble_learning_and_random_forests.ipynb".
The first task is:
Voting classifiers
Can you write Python code for it?

heads_proba = 0.51
coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)
cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)

plt.figure(figsize=(8,3.5))
plt.plot(cumulative_heads_ratio)
plt.plot([0, 10000], [0.51, 0.51], "k--", linewidth=2, label="51%")
plt.plot([0, 10000], [0.5, 0.5], "k-", label="50%")
plt.xlabel("Number of coin tosses")
plt.ylabel("Heads ratio")
plt.legend(loc="lower right")
plt.axis([0, 10000, 0.42, 0.58])
save_fig("law_of_large_numbers_plot")
plt.show()

from sklearn.model_selection import train_test_split
from sklearn.datasets import make_moons

X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC

log_clf = LogisticRegression(random_state=42)
rnd_clf = RandomForestClassifier(random_state=42)
svm_clf = SVC(random_state=42)

voting_clf = VotingClassifier(
    estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
    voting='hard')
voting_clf.fit(X_train, y_train)

from sklearn.metrics import accuracy_score

for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    print(clf.__class__.__name__, accuracy_score(y_test, y_pred))

log_clf = LogisticRegression(random_state=42)
rnd_clf = RandomForestClassifier(random_state=42)
svm_clf = SVC(probability=True, random_state=42)

voting_clf = VotingClassifier(
    estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
    voting='soft')
voting_clf.fit(X_train, y_train)

from sklearn.metrics import accuracy_score

for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    print(clf.__class__.__name__, accuracy_score(y_test, y_pred))


heads_proba = 0.51
coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)
cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)

plt.figure(figsize=(8,3.5))
plt.plot(cumulative_heads_ratio)
plt.plot([0, 10000], [0.51, 0.51], "k--", linewidth=2, label="51%")
plt.plot([0, 10000], [0.5, 0.5], "k-", label="50%")
plt.xlabel("Number of coin tosses")
plt.ylabel("Heads ratio")
plt.legend(loc="lower right")
plt.axis([0, 10000, 0.42, 0.58])
save_fig("law_of_large_numbers_plot")
plt.show()

from sklearn.model_selection import train_test_split
from sklearn.datasets import make_moons

X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)


I am working on the file "07_ensemble_learning_and_random_forests.ipynb".
The first task is:
Warning: In Scikit-Learn 0.20, some hyperparameters (solver, n_estimators, gamma, etc.) start issuing warnings about the fact that their default value will change in Scikit-Learn 0.22. To avoid these warnings and ensure that this notebooks keeps producing the same outputs as in the book, I set the hyperparameters to their old default value.  In your own code, you can simply rely on the latest default values instead.
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC

log_clf = LogisticRegression(solver="liblinear", random_state=42)
rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
svm_clf = SVC(gamma="auto", random_state=42)

voting_clf = VotingClassifier(
    estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
    voting='hard')

voting_clf.fit(X_train, y_train)

from sklearn.metrics import accuracy_score

for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    print(clf.__class__.__name__, accuracy_score(y_test, y_pred))

log_clf = LogisticRegression(solver="liblinear", random_state=42)
rnd_clf = RandomForestClassifier(n_estimators=10, random_state=42)
svm_clf = SVC(gamma="auto", probability=True, random_state=42)

voting_clf = VotingClassifier(
    estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
    voting='soft')
voting_clf.fit(X_train, y_train)

from sklearn.metrics import accuracy_score

for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    print(clf.__class__.__name__, accuracy_score(y_test, y_pred))


I am working on the file "07_ivp.ipynb".
The first task is:
A similar method can be derived if we consider instead using the second order accurate central difference:
$$\frac{U^{n+1} - U^{n-1}}{2\Delta t} = f(t_{n}, U^{n})$$
this method is known as the midpoint or leap-frog method.  Note that the way we have written this method requires a previous function evaluation and technically is a "multi-step" method although we do not actually use the current evaluation.
Can you write Python code for it?

t = numpy.linspace(0.0, 1.6e3, 100)
c_0 = 1.0
decay_constant = numpy.log(2.0) / 1600.0

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, c_0 * numpy.exp(-decay_constant * t), label="True Solution")

# Plot Euler step
dt = 1e3
u_np = c_0 + dt * (-decay_constant * c_0 * numpy.exp(-decay_constant * dt / 2.0))
axes.plot((0.0, dt), (c_0, u_np), 'k')
axes.plot((dt, dt), (u_np, c_0 * numpy.exp(-decay_constant * dt)), 'k--')
axes.plot((0.0, 0.0), (c_0, u_np), 'k--')
axes.plot((0.0, dt), (u_np, u_np), 'k--')
axes.text(400, u_np - 0.05, '$\Delta t$', fontsize=16)

axes.set_title("Radioactive Decay with $t_{1/2} = 1600$ years")
axes.set_xlabel('t (years)')
axes.set_ylabel('$c$')
axes.set_xlim(-1e2, 1.6e3)
axes.set_ylim((0.5,1.0))
plt.show()

c_0 = 1.0
decay_constant = numpy.log(2.0) / 1600.0
f = lambda t, u: -decay_constant * u

t_exact = numpy.linspace(0.0, 1.6e3, 100)
u_exact = c_0 * numpy.exp(-decay_constant * t_exact)

# Implement leap-frog
t_leapfrog = numpy.linspace(0.0, 1.6e3, 10)
delta_t = t_leapfrog[1] - t_leapfrog[0]
u_leapfrog = numpy.empty(t_leapfrog.shape)
u_leapfrog[0] = c_0
# First evaluation use Euler to get us going
u_leapfrog[1] = u_leapfrog[0] + delta_t * f(t_leapfrog[0], u_leapfrog[0])
for n in xrange(1, t_leapfrog.shape[0] - 1):
    u_leapfrog[n + 1] = u_leapfrog[n - 1] + 2.0 * delta_t * f(t[n], u_leapfrog[n])

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(t_leapfrog, u_leapfrog, 'or', label="Leap-Frog")
axes.plot(t_exact, u_exact, 'k--', label="True Solution")

axes.set_title("Leap-Frog")
axes.set_xlabel("t (years)")
axes.set_xlabel("$c(t)$")
axes.set_ylim((0.4,1.1))
axes.legend()
plt.show()


I am working on the file "07_ivp.ipynb".
The first task is:
Adams-Moulton Methods
The Adams-Moulton methods are the implicit versions of the Adams-Bashforth methods.  Since this gives one additional parameter to use $\beta_r$ these methods are generally one order of accuracy greater than their counterparts.
$$\begin{aligned}
&\text{1-step:}&  &U^{n+1} = U^n &+& \frac{\Delta t}{2}(f(U^n) + f(U^{n+1})) \
&\text{2-step:}&  &U^{n+2} = U^{n+1} &+& \frac{\Delta t}{12} (-f(U^n) + 8f(U^{n+1}) + 5f(U^{n+2})) \
&\text{3-step:}&  &U^{n+3} = U^{n+2} &+& \frac{\Delta t}{24} (f(U^n) - 5f(U^{n+1}) + 19f(U^{n+2}) + 9f(U^{n+3})) \
&\text{3-step:}&  &U^{n+4} = U^{n+3} &+& \frac{\Delta t}{720} (-19 f(U^n) + 106 f(U^{n+1}) -264 f(U^{n+2}) + 646 f(U^{n+3})  + 251 f(U^{n+4}))
\end{aligned}$$
Can you write Python code for it?

# Use 2-step Adams-Moulton to compute solution
# u' = - decay u
decay_constant = 1.0
f = lambda t, u: -decay_constant * u

t_exact = numpy.linspace(0.0, 10.0, 100)
u_exact = numpy.exp(-t_exact)

N = 20
t = numpy.linspace(0, 10.0, N)
delta_t = t[1] - t[0]
U = numpy.empty(t.shape)
U[0] = 1.0
U[1] = U[0] + 0.5 * delta_t * f(t[0], U[0])
U[1] = U[0] + delta_t * f(t[0], U[1])
integration_constant = 1.0 / (1.0 + 5.0 * decay_constant * delta_t / 12.0)
for n in xrange(t.shape[0] - 2):
    U[n+2] = (U[n+1] + decay_constant * delta_t / 12.0 * (U[n] - 8.0 * U[n+1])) * integration_constant

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

axes.plot(t_exact, u_exact, 'k', label="True")
axes.plot(t, U, 'ro', label="2-step A-M")

axes.set_title("Adams-Moulton Method")
axes.set_xlabel("t")
axes.set_xlabel("u(t)")
axes.legend(loc=1)

plt.show()


I am working on the file "07_ivp.ipynb".
The first task is:
Another simple implicit method is based on integration using the trapezoidal method.  The scheme is
$$
    \frac{U^{n+1} - U^{n}}{\Delta t} = \frac{1}{2} (f(U^n) + f(U^{n+1}))
$$
Can you write Python code for it?

c_0 = 1.0
decay_constant = numpy.log(2.0) / 1600.0
t_exact = numpy.linspace(0.0, 1.6e3, 100)
u_exact = c_0 * numpy.exp(-decay_constant * t_exact)

# Implement trapezoidal method
t = numpy.linspace(0.0, 1.6e3, 10)
delta_t = t[1] - t[0]
u = numpy.empty(t.shape)
u[0] = c_0
integration_constant = (1.0 - decay_constant * delta_t / 2.0) / (1.0 + decay_constant * delta_t / 2.0)
for n in xrange(t.shape[0] - 1):
    u[n + 1] = u[n] * integration_constant

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, u, 'or', label="Trapezoidal")
axes.plot(t_exact, u_exact, 'k--', label="True Solution")

axes.set_title("Trapezoidal")
axes.set_xlabel("t (years)")
axes.set_xlabel("$c(t)$")
axes.set_ylim((0.4,1.1))
axes.legend()
plt.show()


I am working on the file "07_ivp.ipynb".
The first task is:
Error Analysis
Truncation Errors
We can define truncation errors the same as we did before where we insert the true solution of the ODE into the difference equation and use Taylor series expansions.
$$
    \tau^n = \frac{1}{\Delta t} [U^{n+1} - u(t + \Delta t)]
$$
Similarly if we know
$$
    \lim_{\Delta t \rightarrow 0} \tau^n = 0
$$
then the discretized equation is considered consistent.
Order of accuracy is also defined the same way as before.  If
$$
    || \tau || \leq C \Delta t^p
$$
uniformly on $t \in [0, T]$ then the discretization is $p$th order accurate.  Note that a method is consistent if $p > 0$.
Error Analysis of Forward Euler
We can analyze the error and convergence order of forward Euler by considering the Taylor series centered at $t_n$:
$$
    u(t) = u(t_n) + (t - t_n) u'(t_n) + \frac{u''(t_n)}{2} (t - t_n)^2 + \mathcal{O}((t-t_n)^3)
$$
Compute the LTE for forward Euler's method.
Evaluating this series at $t_{n+1}$ gives
$$\begin{aligned}
u(t_{n+1}) &= u(t_n) + (t_{n+1} - t_n) u'(t_n) + \frac{u''(t_n)}{2} (t_{n+1} - t_n)^2 + \mathcal{O}((t_{n+1}-t_n)^3)\
&=u(t_n) + \Delta t f(t_n, u(t_n)) + \frac{u''(t_n)}{2} \Delta t^2 + \mathcal{O}(\Delta t^3)
\end{aligned}$$
From the definition of truncation error we can use our Taylor series expression and find the truncation error to be
$$\begin{aligned}
    \tau^n &= \frac{u(t_{n+1}) - u(t_n)}{\Delta t} - f(t_n, u(t_n)) \
    &= \frac{1}{\Delta t} \left [u(t_n) + \Delta t ~ f(t_n, u(t_n)) + \frac{u''(t_n)}{2} \Delta t^2 + \mathcal{O}(\Delta t^3) - u(t_n) - \Delta t ~ f(t_n, u(t_n)) \right ]\
    &= \frac{1}{\Delta t} \left [ \frac{u''(t_n)}{2} \Delta t^2 + \mathcal{O}(\Delta t^3) \right ] \
    &= \frac{u''(t_n)}{2} \Delta t + \mathcal{O}(\Delta t^2)
\end{aligned}$$
This implies that forwar Euler is first order accurate and therefore consistent.
Error Analysis of Leap-Frog Method
To easily analyze this method we will expand the Taylor series from before to another order and evaluate at both the needed positions:
$$
    u(t) = u(t_n) + (t - t_n) u'(t_n)...
Can you write Python code for it?

# Compare accuracy between Euler and Leap-Frog
f = lambda t, u: -u
u_exact = lambda t: numpy.exp(-t)
u_0 = 1.0

t_f = 10.0
num_steps = [2**n for n in xrange(4,10)]
delta_t = numpy.empty(len(num_steps))
error_euler = numpy.empty(len(num_steps))
error_trap = numpy.empty(len(num_steps))
error_leapfrog = numpy.empty(len(num_steps))

for (i, N) in enumerate(num_steps):
    t = numpy.linspace(0, t_f, N)
    delta_t[i] = t[1] - t[0]

    # Compute Euler solution
    u_euler = numpy.empty(t.shape)
    u_euler[0] = u_0
    for n in xrange(t.shape[0] - 1):
        u_euler[n+1] = u_euler[n] + delta_t[i] * f(t[n], u_euler[n])

    # Compute trapezoidal
    u_trap = numpy.empty(t.shape)
    u_trap[0] = u_0
    integration_constant = (1.0 - delta_t[i] / 2.0) / (1.0 + delta_t[i] / 2.0)
    for n in xrange(t.shape[0] - 1):
        u_trap[n + 1] = u_trap[n] * integration_constant

    # Compute Leap-Frog
    u_leapfrog = numpy.empty(t.shape)
    u_leapfrog[0] = 1.0
    u_leapfrog[1] = u_euler[1]
    for n in xrange(1, t.shape[0] - 1):
        u_leapfrog[n+1] = u_leapfrog[n-1] + 2.0 * delta_t[i] * f(t[n], u_leapfrog[n])

    # Compute error for each
    error_euler[i] = numpy.linalg.norm(delta_t[i] * (u_euler - u_exact(t)), ord=1)
    error_trap[i] = numpy.linalg.norm(delta_t[i] * (u_trap - u_exact(t)), ord=1)
    error_leapfrog[i] = numpy.linalg.norm(delta_t[i] * (u_leapfrog - u_exact(t)), ord=1)

# Plot error vs. delta_t
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))
axes.loglog(delta_t, error_euler, 'bo', label='Forward Euler')
axes.loglog(delta_t, error_trap, 'go', label='Trapezoidal')
axes.loglog(delta_t, error_leapfrog, 'ro', label="Leap-Frog")

axes.loglog(delta_t, order_C(delta_t[2], error_euler[2], 1.0) * delta_t**1.0, '--b')
axes.loglog(delta_t, order_C(delta_t[2], error_trap[2], 2.0) * delta_t**2.0, '--r')
axes.loglog(delta_t, order_C(delta_t[2], error_leapfrog[2], 2.0) *...


I am working on the file "07_ivp.ipynb".
The first task is:
Linear Multi-Step Methods
Multi-step methods (as introduced via the leap-frog method) are ODE methods that require multiple time step evaluations to work.  Some of the advanatages of using a multi-step method rather than one-step method included

Taylor series methods require differentiating the given equation which can be cumbersome and difficult to impelent
One-step methods at higher order often require the evaluation of the function $f$ many times

Disadvantages

Methods are not self-starting, i.e. they require other methods to find the initial values
The time step $\Delta t$ in one-step methods can be changed at any time while multi-step methods this is much more complex

General Linear Multi-Step Methods
All linear multi-step methods can be written as the linear combination of past, present and future solutions:
$$
    \sum^r_{j=0} \alpha_j U^{n+j} = \Delta t \sum^r_{j=0} \beta_j f(U^{n+j}, t_{n+j})
$$
If $\beta_r = 0$ then the method is explicit (only requires previous time steps).  Note that the coefficients are not unique as we can multiply both sides by a constant.  In practice a normalization of $\alpha_r = 1$ is used.
Example: Adams Methods
$$
    U^{n+r} = U^{n+r-1} + \Delta t \sum^r_{j=0} \beta_j f(U^{n+j})
$$
All these methods have $\alpha_r = 1$, $\alpha_{r-1} = -1$ and $\alpha_j=0$ for $j < r - 1$ leaving the method to be specified by how the evaluations of $f$ is done determining the $\beta_j$.
Adams-Bashforth Methods
The Adams-Bashforth methods are explicit solvers that maximize the order of accuracy given a number of steps $r$.  This is accomplished by looking at the Taylor series and picking the coefficients $\beta_j$ to elliminate as many terms in the Taylor series as possible.
$$\begin{aligned}
    &\text{1-step:}& &U^{n+1} = U^n &+& \Delta t ~ f(U^n) \
    &\text{2-step:}& &U^{n+2} = U^{n+1} &+& \frac{\Delta t}{2} ~ (-f(U^n) + 3 f(U^{n+1})) \
    &\text{3-step:}& &U^{n+3} = U^{n+2} &+& \frac{\Delta t}{12} ~ (5 f(U^n) - 16 f(U^{n+1}) + 23 f(U^{n+2})) \
    &\text{4-step:}& &U^{n+4} =...
Can you write Python code for it?

# Use 2-step Adams-Bashforth to compute solution
f = lambda t, u: -u

t_exact = numpy.linspace(0.0, 10.0, 100)
u_exact = numpy.exp(-t_exact)

N = 100
t = numpy.linspace(0, 10.0, N)
delta_t = t[1] - t[0]
U = numpy.empty(t.shape)

# Use RK-2 to start the method
U[0] = 1.0
U[1] = U[0] + 0.5 * delta_t * f(t[0], U[0])
U[1] = U[0] + delta_t * f(t[0], U[1])
for n in xrange(0,len(t)-2):
    U[n+2] = U[n + 1] + delta_t / 2.0 * (-f(t[n], U[n]) + 3.0 * f(t[n+1], U[n+1]))

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

axes.plot(t_exact, u_exact, 'k', label="True")
axes.plot(t, U, 'ro', label="2-step A-B")

axes.set_title("Adams-Bashforth Method")
axes.set_xlabel("t")
axes.set_xlabel("u(t)")
axes.legend(loc=1)

plt.show()


I am working on the file "07_ivp.ipynb".
The first task is:
Numerical Methods for Initial Value Problems
We now turn towards time dependent PDEs.  Before moving to the full PDEs we will explore numerical methods for systems of ODEs that are initial value problems of the general form
$$
    \frac{\text{d} \vec{u}}{\text{d}t} = \vec{f}(t, \vec{u}) ~~~~ \vec{u}(0) = \vec{u}_0
$$
where
 - $\vec{u}(t)$ is the state vector
 - $\vec{f}(t, \vec{u})$ is a vector-valued function that controls the growth of $\vec{u}$ with time
 - $\vec{u}(0)$ is the initial condition at time $t = 0$
Note that the right hand side function $f$ could in actuality be the discretization in space of a PDE, i.e. a system of equations.
Examples:  Simple radioactive decay
$\vec{u} = [c]$
$$\frac{\text{d} c}{\text{d}t} = -\lambda c ~~~~ c(0) = c_0$$
which has solutions of the form $c(t) = c_0 e^{-\lambda t}$
Can you write Python code for it?

t = numpy.linspace(0.0, 1.6e3, 100)
c_0 = 1.0
decay_constant = numpy.log(2.0) / 1600.0

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, 1.0 * numpy.exp(-decay_constant * t))

axes.set_title("Radioactive Decay with $t_{1/2} = 1600$ years")
axes.set_xlabel('t (years)')
axes.set_ylabel('$c$')
axes.set_ylim((0.5,1.0))
plt.show()


I am working on the file "07_ivp.ipynb".
The first task is:
Predictor-Corrector Methods
One way to simplify the Adams-Moulton methods so that implicit evaluations are not needed is by estimating the required implicit function evaluations with an explicit method.  These are often called predictor-corrector methods as the explicit method provides a prediction of what the solution might be and the not explicit corrector step works to make that estimate more accurate.
Example: One-Step Adams-Bashforth-Moulton
Use the One-step Adams-Bashforth method to predict the value of $U^{n+1}$ and then use the Adams-Moulton method to correct that value:
$\hat{U}^{n+1} = U^n + \Delta t f(U^n)$
$U^{n+1} = U^n + \frac{1}{2} \Delta t (f(U^n) + f(\hat{U}^{n+1})$
This method is second order accurate.
Can you write Python code for it?

# One-step Adams-Bashforth-Moulton
f = lambda t, u: -u

t_exact = numpy.linspace(0.0, 10.0, 100)
u_exact = numpy.exp(-t_exact)

N = 100
t = numpy.linspace(0, 10.0, N)
delta_t = t[1] - t[0]
U = numpy.empty(t.shape)

U[0] = 1.0
for n in xrange(t.shape[0] - 1):
    U[n+1] = U[n] + delta_t * f(t[n], U[n])
    U[n+1] = U[n] + 0.5 * delta_t * (f(t[n], U[n]) + f(t[n+1], U[n+1]))

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

axes.plot(t_exact, u_exact, 'k', label="True")
axes.plot(t, U, 'ro', label="2-step A-B")

axes.set_title("Adams-Bashforth-Moulton P/C Method")
axes.set_xlabel("t")
axes.set_xlabel("u(t)")
axes.legend(loc=1)

plt.show()

# Compare accuracy between RK-2, AB-2 and AM-2
f = lambda t, u: -u
u_exact = lambda t: numpy.exp(-t)

t_f = 10.0
num_steps = [2**n for n in xrange(4,10)]
delta_t = numpy.empty(len(num_steps))
error_ab = numpy.empty(len(num_steps))
error_am = numpy.empty(len(num_steps))
error_pc = numpy.empty(len(num_steps))

for (i, N) in enumerate(num_steps):
    t = numpy.linspace(0, t_f, N)
    delta_t[i] = t[1] - t[0]

    # RK-2 bootstrap for AB and AM
    U_rk = numpy.empty(2)
    U_rk[0] = 1.0
    U_rk[1] = U_rk[0] + 0.5 * delta_t[i] * f(t[0], U_rk[0])
    U_rk[1] = U_rk[0] + delta_t[i] * f(t[0], U_rk[1])

    # Compute Adams-Bashforth 2-stage
    U_ab = numpy.empty(t.shape)
    U_ab[:2] = U_rk[:2]
    for n in xrange(t.shape[0] - 2):
        U_ab[n+2] = U_ab[n + 1] + delta_t[i] / 2.0 * (-f(t[n], U_ab[n]) + 3.0 * f(t[n+1], U_ab[n+1]))

    # Compute Adams-Moulton 2-stage
    U_am = numpy.empty(t.shape)
    U_am[:2] = U_ab[:2]
    decay_constant = 1.0
    integration_constant = 1.0 / (1.0 + 5.0 * decay_constant * delta_t[i] / 12.0)
    for n in xrange(t.shape[0] - 2):
        U_am[n+2] = (U_am[n+1] + decay_constant * delta_t[i] / 12.0 * (U_am[n] - 8.0 * U_am[n+1])) * integration_constant

    # Compute Adams-Bashforth-Moulton
    U_pc = numpy.empty(t.shape)
    U_pc[0] = 1.0
    for n in xrange(t.shape[0] - 1):
        U_pc[n+1] = U_pc[n] + delta_t[i] *...


I am working on the file "07_ivp.ipynb".
The first task is:
Similar to forward Euler is the backward Euler method which, as you may have guessed, evaluates the function $f$ at the updated time so that
$$
    U^{n+1} = U^n + \Delta t f(t_{n+1}, U^{n+1}).
$$
Schemes where the function $f$ is evaluated at the unknown time are called implicit methods.
Can you write Python code for it?

t = numpy.linspace(0.0, 1.6e3, 100)
c_0 = 1.0
decay_constant = numpy.log(2.0) / 1600.0

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, c_0 * numpy.exp(-decay_constant * t), label="True Solution")

# Plot Euler step
dt = 1e3
u_np = c_0 + dt * (-decay_constant * c_0 * numpy.exp(-decay_constant * dt))
axes.plot((0.0, dt), (c_0, u_np), 'k')
axes.plot((dt, dt), (u_np, c_0 * numpy.exp(-decay_constant * dt)), 'k--')
axes.plot((0.0, 0.0), (c_0, c_0 * numpy.exp(-decay_constant * dt)), 'k--')
axes.plot((0.0, dt), (c_0 * numpy.exp(-decay_constant * dt), c_0 * numpy.exp(-decay_constant * dt)), 'k--')
axes.text(400, u_np - 0.05, '$\Delta t$', fontsize=16)

axes.set_title("Radioactive Decay with $t_{1/2} = 1600$ years")
axes.set_xlabel('t (years)')
axes.set_ylabel('$c$')
axes.set_xlim(-1e2, 1.6e3)
axes.set_ylim((0.5,1.0))
plt.show()

c_0 = 1.0
decay_constant = numpy.log(2.0) / 1600.0
f = lambda t, u: -decay_constant * u

t_exact = numpy.linspace(0.0, 1.6e3, 100)
u_exact = c_0 * numpy.exp(-decay_constant * t_exact)

# Implement backwards Euler
t_backwards = numpy.linspace(0.0, 1.6e3, 10)
delta_t = t_backwards[1] - t_backwards[0]
u_backwards = numpy.empty(t_backwards.shape)
u_backwards[0] = c_0
for n in xrange(0, t_backwards.shape[0] - 1):
    u_backwards[n + 1] = u_backwards[n] / (1.0 + decay_constant * delta_t)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(t_backwards, u_backwards, 'or', label="Backwards Euler")
axes.plot(t_exact, u_exact, 'k--', label="True Solution")

axes.set_title("Backwards Euler")
axes.set_xlabel("t (years)")
axes.set_xlabel("$c(t)$")
axes.set_ylim((0.4,1.1))
axes.legend()
plt.show()


I am working on the file "07_model_3_4_1.ipynb".
The first task is:
$\phi(r) = 10^{b}r^{a}$として最小2乗法でフィッティング
Can you write Python code for it?

param = [('a', 1.5), ('b', 0.)]
xmin, xmax = 0., 0.07
x = {r'$r$': r}
y = {r'$\phi$': phi1}
def fit_func(parameter, x):
    a = parameter[0]
    b = parameter[1]
    return np.power(x, a)*np.power(10, b)
myfit(fit_func, param, x, y, xmin, xmax)


I am working on the file "07_model_3_4_1.ipynb".
The first task is:
$r$を固定して$N$を変更したときのクラスター数と点の総数の間の関係
横軸を$X_{i}$の数$N$、縦軸を$1-(\text{クラスタ数}/\text{点の総数})$としたときのグラフを書いてみる。
Can you write Python code for it?

trial = 100

N = np.arange(1, 20)
phi6 = []
for _N in N:
    _phi = 0.
    for t in range(trial):
        meeting = Meeting(K=50, N=_N, r=0.07, draw=False)
        meeting.init()
        _phi += len(uniq_list([x[1][1] for x in meeting.ideas]))/float(len(meeting.ideas))
    phi6.append(1 - _phi/trial)

myplot1({r'$N$': N}, {r'$\phi$': phi6})


I am working on the file "07_model_3_4_1.ipynb".
The first task is:
S字型の曲線であるので、
$$\phi (r) = 1 - \exp \left[ -  \left( \frac{r}{\omega} \right)^{a} \right]$$
としてパラメータ$\omega$に関して最小2乗法でフィッティングを行った場合。
Can you write Python code for it?

omega = 0.06
a = 2.0
param = [omega, a]
rmin, rmax = 0.01, 0.2

def fit_func(parameter, x):
    omega = parameter[0]
    a = parameter[1]
    return 1 - np.exp(-(x/omega)**a)

def fit(parameter, x, y):
    return y - fit_func(parameter, x)

i = 0
while r[i] < rmin:
    i += 1
imin, imax = i, i
while r[i] < rmax:
    i += 1
imax = i - 1

res = leastsq(fit, param, args=(r[imin:imax], phi1[imin:imax]))
print u"omega: " + str(res[0][0])
print u"a: " + str(res[0][1])
R3 = fit_func(res[0], r[imin:imax])

myplot1({r'$r$': r}, {r'$\phi$': phi1}, r[imin:imax], R3, param={'\omega': res[0][0], 'a': res[0][1]})


I am working on the file "07_model_3_4_1.ipynb".
The first task is:
このとき、意見の総数と参加者の数、一人あたりの意見の数の間には比例の関係が成り立っており、この数のみに依存して、どちらを変えるかは問題ではない。したがって、より刻みを多く取ることのできる一人あたりの意見の数$S$を変えて計算した場合を見てみることにする。
Can you write Python code for it?

trial = 100

S = np.arange(10, 70)
phi7 = []
for _S in S:
    _phi = 0.
    for t in range(trial):
        meeting = Meeting(K=50, S=_S, N=6, r=0.07, draw=False)
        meeting.init()
        _phi += len(uniq_list([x[1][1] for x in meeting.ideas]))/float(len(meeting.ideas))
    phi7.append(1 - _phi/trial)

myplot1({r'$S$': S}, {r'$\phi$': phi7})


I am working on the file "07_model_3_4_1.ipynb".
The first task is:
フィッティング用関数
Can you write Python code for it?

def myfit(fit_func, parameter, x, y, xmin, xmax):
    """my fitting and plotting function.

    fit_func: function (parameter(type:list), x)
    parameter: list of tuples: [('param1', param1), ('param2', param2), ...]
    x, y: dict
    xmin, xmax: float
    """
    xkey, xdata = x.items()[0]
    ykey, ydata = y.items()[0]

    def fit(parameter, x, y):
        return y - fit_func(parameter, x)

    # use x : xmin < x < xmax
    i = 0
    while xdata[i] < xmin:
        i += 1
    imin, imax = i, i
    while xdata[i] < xmax:
        i += 1
    imax = i - 1

    paramdata = [b for a, b in parameter]
    paramkey = [a for a, b in parameter]
    res = leastsq(fit, paramdata, args=(xdata[imin:imax], ydata[imin:imax]))
    for p in res[0]:
        print xkey + ": " + str(p)
    fitted = fit_func(res[0], xdata[imin:imax])

    fittedparam = dict([(k, v) for k, v in zip(paramkey, res[0])])
    myplot1(x, y, xdata[imin:imax], fitted, param=fittedparam)


I am working on the file "07_model_3_4_1.ipynb".
The first task is:
両変数を対数にした状態で直線としてフィットしてみる。得られたパラメータによるフィッティング関数のプロットは、元の状態に戻してから行う。後に示す直接べき関数として求めた場合に比べて、$r$の小さい領域での直線の傾きがよく合っているように見える。
Can you write Python code for it?

a = 1.5
b = 0.
param = [a, b]
rmin, rmax = 0., 0.07

def fit_func(parameter, x):
    a = parameter[0]
    b = parameter[1]
    return a*np.log10(x) + b

def fit(parameter, x, y):
    return np.log10(y) - fit_func(parameter, x)

i = 0
while r[i] < rmin:
    i += 1
imin, imax = i, i
while r[i] < rmax:
    i += 1
imax = i - 1

res = leastsq(fit, param, args=(r[imin:imax], phi1[imin:imax]))
print u"傾き: " + str(res[0][0])
print u"切片: " + str(res[0][1])
R1 = np.power(10, fit_func(res[0], r[imin:imax]))

myplot1({r'$r$': r}, {r'$\phi$': phi1}, r[imin:imax], R1, param={'a': res[0][0], 'b': res[0][1]})


I am working on the file "07_model_3_4_1.ipynb".
The first task is:
閾値$r$を決めたときに、領域$\Omega$内の任意の点を一様に選んだとき、その中に点が存在する確率の期待値は、解析的計算によって
$$p'(r) = \frac{1}{2}r^{4} -\frac{8}{3}r^{3} + \pi r^{2}$$
$r$を定めたとき、すべての点の個数が$M$個であるとすると、一つの点の点がもつ次数の期待値$l$は
$$l = p'(r)(M-1) = \left( \frac{1}{2}r^{4} -\frac{8}{3}r^{3} + \pi r^{2} \right)(M-1)$$
となる。これを実際のシミュレーションの結果と照らして確かめる。
Can you write Python code for it?

trial = 100

r = np.linspace(0.01, 0.5, num=50)
phi3 = []
for _r in r:
    _phi = 0.
    for t in range(trial):
        meeting = Meeting(K=50, N=6, r=_r, draw=False)
        meeting.init()
        _phi += meeting.ave_l
    phi3.append(_phi/trial)

fig = plt.figure(figsize=(8,6))
ax = fig.add_subplot(111)

r = np.linspace(0.01, 0.5, num=50)
def func(x):
    return (1./2*x**4 - 8/3.*x**3 + np.pi*x**2)*(120-1)
y = func(r)
def func2(x):
    return np.sqrt((-0.25*x**8 + 8/3.*x**7 - (64/9.+np.pi)*x**6 + 16/3.*np.pi*x**5
                   + (0.5-np.pi**2)*x**4 - 8/3.*x**3 + np.pi*x**2)*(120-1)/(trial))

delta = func2(r)
y1 = y + delta
y2 = y - delta
y3 = np.zeros(50)
y3[y2>0] = y2[y2>0]
ax.fill_between(r, y1, y3, facecolor='green', alpha=0.2)
ax.plot(r, phi3)
ax.plot(r, y)
ax.set_xlabel(r'$r$')
ax.set_ylabel(r"Average number of edges for each time: $l$")
plt.show()


I am working on the file "07_modelling/20151201_ZombieApocalypse-Assignment.ipynb".
The first task is:
In the second step, we write a small function f, that receives a list of the current values of our variables x and the current time t. The function has to evaluate the equations of our system or $\frac{\text{d}\vec{x}}{\text{d}t}$, respectively. Afterwards, it returns the values of the equations as another list.
Important
Since this function f is used by the solver, we are not allowed to change the input (arguments) or output (return value) of this function.
Can you write Python code for it?

# function 'f' or 'dxdt' to evaluate the changes of the system dy/dt = f(y, t)


I am working on the file "07_modelling/20151201_ZombieApocalypse-Assignment.ipynb".
The first task is:
Last but not least, we need to import and call our solver. The result will be a matrix with our time courses as columns and the values at the specified time points. Since we have a values for every time point and every species, we can directly plot the results using matplotlib.
Can you write Python code for it?

# zombie apocalypse modeling
import matplotlib.pyplot as plt  # for plotting
# plots inside the notebook
%matplotlib inline
from scipy.integrate import odeint  # the integrator

# solve the DEs
result = odeint(f, y0, t)
S = result[:, 0]
Z = result[:, 1]
R = result[:, 2]

# plot results
plt.figure()
plt.plot(t, S, label='Humans')
plt.plot(t, Z, label='Zombies')
plt.plot(t, R, label='Dead Humans')
plt.xlabel('Days from outbreak')
plt.ylabel('Population')
plt.title('Zombie Apocalypse - No Init. Dead Pop.; No New Births.')
plt.ylim([0,500])
plt.legend(loc=0)


I am working on the file "07_modelling/20160527_NumericalSolver.ipynb".
The first task is:
And now we compare the results. I marked the amplitude and position of the maxima with red dotted lines.
Can you write Python code for it?


I am working on the file "07_modelling/20160527_NumericalSolver.ipynb".
The first task is:
As you can see, the Heun's method seems to have already a remarkable accuracy, even if it is a very simple method. Let's compare the results of odeint and the Heun's method directly:
Can you write Python code for it?

plt.plot(timegrid, result.T[0], label='X odeint')
plt.plot(timegrid, result.T[1], label='Y odeint')
plt.legend(loc=2)

plt.plot(timegrid_e4, my_heun_result_e4[0], label="X Heun")
plt.plot(timegrid_e4, my_heun_result_e4[1], label="Y Heun")
plt.plot([0,20], [13.67, 13.67], 'r--')
plt.plot([4.32,4.32], [0,14], 'r--')
plt.plot([8.9,8.9], [0,14], 'r--')
plt.plot([13.46,13.46], [0,14], 'r--')
plt.plot([18.06,18.06], [0,14], 'r--')
plt.legend(loc=2)
plt.title('Comparison odeint & Heun method')


I am working on the file "07_modelling/20160527_NumericalSolver.ipynb".
The first task is:
Heun's method
If you want to increase the accuracy of your method, you could use the trapezoidal rule you know from the approximation of integrals. The second point is of course missing, but here you could use Euler's method!
As we will see, this method is a huge improvement compared to Euler's method!
$$\Phi (t,x,h) = x + \frac{h}{2}\Bigl(f(t,x)+f\bigl(t+h,\underbrace{x+h\cdot f(t,x)}_{Euler's\ method}\bigr)\Bigr)$$
Runge - Kutta method
The idea of Runge and Kutta were quite straight forward: Why not using Heun's method recursive? To get the second point you do not use Euler's method but again the trapezoidal rule... and again... and again. This method is still used and very good for most of the ODE systems!
Can you write Python code for it?

def my_heun_solver(dxdt, s0, timegrid):
    """
    Implementation of the Heun method (constant stepsize)
    """
    # first species values are s0
    s = s0
    # do timesteps
    for j, time in enumerate(timegrid):
        # first time step, just save initial values
        if j == 0:
            result = [[value] for value in s0]
            continue
        # next time step, calculate values and save them
        for i, species in enumerate(s):
            hi = (timegrid[j] - timegrid[j-1])
            species = species + (hi/2)*(dxdt(s,time)[i]+dxdt([s[k]+hi*dxdt(s,time)[k] for k in range(len(s))], time+hi)[i])
            result[i].append(species)
        # update species with new values
        s[0] = result[0][-1]
        s[1] = result[1][-1]
    return result

import matplotlib.pyplot as plt
%matplotlib inline

# timegrids
timegrid_e3 = np.linspace(0,20,2000)
timegrid_e4 = np.linspace(0,20,20000)

# plot results
s0=[5,10]
my_heun_result_e3 = my_heun_solver(my_dxdt, s0, timegrid_e3)
s0=[5,10]
my_heun_result_e4 = my_heun_solver(my_dxdt, s0, timegrid_e4)


I am working on the file "07_modelling/20160527_NumericalSolver.ipynb".
The first task is:
Let's simulate the same system also with odeint, the standard ODE solver of the scipy Python package.
Can you write Python code for it?

import scipy.integrate

timegrid = np.linspace(0,20,2000)
s0 = [5,10]
result = scipy.integrate.odeint(my_dxdt, s0, timegrid)


I am working on the file "07_modelling/20160606_SSAcomplete.ipynb".
The first task is:
Deterministic, continuous solution
Can you write Python code for it?

import scipy.integrate
import numpy as np
import math
from matplotlib import pyplot as plt
plt.style.use('ggplot')
%matplotlib inline

k1 = 0.1
k2 = 0.02
k3 = 0.4
k4 = 0.02

def dSdt( S, t):
    X, Y = S
    dxdt = k1*X - k2*X*Y
    dydt = k4*X*Y - k3*Y
    return [dxdt, dydt]

time = np.arange(0,300,1)
S0 = np.array([10., 10.] )

result = scipy.integrate.odeint( dSdt, S0, time  )
plt.plot( time, result )


I am working on the file "07_modelling/20160606_SSAcomplete.ipynb".
The first task is:
Stochastic, discrete solution
Can you write Python code for it?

na = 6.022e23
# species
X = 100
Y = 100
# parameters
k1 = 0.1   # birth X
k2 = 0.02  # eaten X
k3 = 0.4   # death Y
k4 = 0.02  # reproduce Y
# time
t = 0
t_end = 50
# timecourses
timecourse = [0]
X_tc = [X]
Y_tc = [Y]
# loop
while t < t_end:
    h_k1 = X*(X-1)/2
    h_k2 = X*Y
    h_k3 = Y
    h_k4 = X*Y

    R_sum = sum([h_k1*k1, h_k2*k2, h_k3*k3, h_k4*k4])

    a_k1 = h_k1*k1/R_sum
    a_k2 = h_k2*k2/R_sum
    a_k3 = h_k3*k3/R_sum
    a_k4 = h_k4*k4/R_sum
    a = [a_k1, a_k2, a_k3, a_k4]
    a_sum = sum(a)

    r1 = np.random.uniform()
    t += - (1./R_sum)*math.log(r1)
    timecourse.append(t)

    r2 = np.random.uniform()

    if r2 < sum(a[:1]):  # k1
        X += 1
        #print('k1')
    elif r2 > sum(a[:1]) and r2 < sum(a[:2]):  # k2
        X -= 1
        #print('k2')
    elif r2 > sum(a[:2]) and r2 < sum(a[:3]):  # k3
        Y -= 1
        #print('k3')
    else:  # k4
        Y += 1
        #print('k4')

    X_tc.append(X)
    Y_tc.append(Y)

plt.plot( timecourse, X_tc )
plt.plot( timecourse, Y_tc )
print(Y_tc[-10:])
print(X_tc[-10:])

import math
r1 = [(i+1)/10 for i in range(10)]
R_sum = 30
t = [(-(1./R_sum)*math.log(r)) for r in r1]

import matplotlib.pyplot as pyp
%matplotlib inline
pyp.plot(r1, t)
pyp.show()


I am working on the file "07_modelling/Modelling_Hard.ipynb".
The first task is:
Create Plot
Can you write Python code for it?

import matplotlib.pyplot as plt
plt.title('My Gillespie solution')
plt.plot(solution_SSA['time'], solution_SSA['A'], label='[A]')
plt.plot(solution_SSA['time'], solution_SSA['B'], label='[B]')
plt.plot(solution_SSA['time'], solution_SSA['C'], label='[C]')
plt.xlabel('time [sec]')
plt.ylabel('number of molecules')
plt.legend(loc='best', prop={'size': 10})
plt.show()


I am working on the file "07_object_oriented.ipynb".
The first task is:
Building something more useful
Can you write Python code for it?

import datetime

class Person:

    def __init__(self, first_name, last_name, birthday, email_address):
        self.first_name = first_name
        self.last_name = last_name
        self.birthday = birthday
        self.email_address = email_address

    def age(self):
        today = datetime.date.today()
        born = datetime.date(*[int(field) for field in self.birthday.split('/')])
        return today.year - born.year - ((today.month, today.day) < (born.month, born.day))

    def email(self, message):
        print('Sending email to {} {} ({})'.format(self.first_name, self.last_name, self.email_address))
        print(message)

harry = Person('Harry', 'Potter', '1980/7/31', 'harry@hogwarts.uk')

harry.email('Hello Harry')

harry.age()


I am working on the file "07_object_oriented.ipynb".
The first task is:
Contact Book
Can you write Python code for it?

contact_book = []

contact_book.append(Person('Harry', 'Potter', '1980/7/31', 'harry@hogwarts.uk'))
contact_book.append(Person('Albert', 'Einstein', '1879/03/14', 'albert@einstein.com'))
contact_book.append(Person('Ernst', 'August', '1771/6/5', 'ernst@koeningreich_hannover.de'))
contact_book.append(Person('Karl', 'Marx', '1818/5/5', 'marx@revolution.su'))


I am working on the file "07_object_oriented.ipynb".
The first task is:
File objects:
Can you write Python code for it?


I am working on the file "07_object_oriented.ipynb".
The first task is:
Objects are instances of classes
Can you write Python code for it?

a = MyObject()
b = MyObject()
c = MyObject()

a, b, c


I am working on the file "07_recurrent/keras_recurrent_LM.ipynb".
The first task is:
Read dataset
Good corpus to start with is WIKI. A easy starting version can be downloaded from http://www.cs.upc.edu/~nlp/wikicorpus/.
Can you write Python code for it?

with open('./en.wiki.txt', 'r') as f:
    data = f.read()
data = data[:50000000]
pos = np.random.randint(len(data)-500)
print(data[pos:pos+1000])

dataT = np.asarray(bytearray(data), dtype=np.uint8)


I am working on the file "07_responsible_ai/explainability.ipynb".
The first task is:
Explaining simpler models
Here we'll use the learned coefficients from a linear regression model as an explainability approach. Note: be cautious when drawing conclusions from learned weights, see the Explainability section in the book for more details.
Can you write Python code for it?

!gsutil cp gs://ml-design-patterns/auto-mpg.csv .

data = pd.read_csv('auto-mpg.csv', na_values='?')
data = data.dropna()

data = data.drop(columns=['car name'])

data = pd.get_dummies(data, columns=['origin'])

data.head()

labels = data['mpg']
data = data.drop(columns=['mpg', 'cylinders'])

x,y = data,labels
x_train,x_test,y_train,y_test = train_test_split(x,y)


I am working on the file "07_responsible_ai/explainability.ipynb".
The first task is:
Feature attributions with SHAP
Using the same dataset, we'll train a deep neural net with TensorFlow and use the SHAP library to get feature attributions.
Can you write Python code for it?

model = tf.keras.Sequential([
  tf.keras.layers.Dense(16, activation='relu', input_shape=[len(x_train.iloc[0])]),
  tf.keras.layers.Dense(16, activation='relu'),
  tf.keras.layers.Dense(1)
])

optimizer = tf.keras.optimizers.RMSprop(0.001)

model.compile(loss='mse',
              optimizer=optimizer,
              metrics=['mae', 'mse'])

model.fit(x_train, y_train, epochs=1000)

!pip install shap

import shap

# Create an explainer object and get feature attributions for the first 10 examples in our test dataset
explainer = shap.DeepExplainer(model, x_train[:200])
shap_values = explainer.shap_values(x_test.values[:10])

# Print the feature attributions for the first example in our test set
shap_values[0][0]

# This is the baseline value shap is using
explainer.expected_value.numpy()

shap.initjs()
shap.force_plot(explainer.expected_value[0].numpy(), shap_values[0][0,:], x_test.iloc[0,:])

shap.summary_plot(shap_values, feature_names=data.columns.tolist(), class_names=['MPG'])


I am working on the file "07_responsible_ai/heuristic_benchmark.ipynb".
The first task is:
2. Classification on poorly understood features
Problem: Whether or not an accepted answer will be edited.
Benchmark: Probability distribution of accepted answers that are edited.
Can you write Python code for it?

%%bigquery
SELECT
  AVG(IF(a.last_edit_date IS NULL, 0, 1)) AS prob_edited
FROM `bigquery-public-data.stackoverflow.posts_questions` q
JOIN `bigquery-public-data.stackoverflow.posts_answers` a
ON q.accepted_answer_id = a.id


I am working on the file "07_responsible_ai/heuristic_benchmark.ipynb".
The first task is:
4. Regression with one or two important features
Problem: Predict duration of bicycle rental.
Benchmark: Lookup table
Can you write Python code for it?

%%bigquery
CREATE TEMPORARY FUNCTION is_peak_hour(start_date TIMESTAMP) aS
(EXTRACT(DAYOFWEEK FROM start_date) BETWEEN 2 AND 6 -- weekday
    AND (
       EXTRACT(HOUR FROM start_date) BETWEEN 6 AND 10
       OR
       EXTRACT(HOUR FROM start_date) BETWEEN 15 AND 18))
;

SELECT
   start_station_name,
   is_peak_hour(start_date) AS is_peak,
   AVG(duration) AS predicted_duration,
FROM `bigquery-public-data.london_bicycles.cycle_hire`
GROUP BY 1, 2
ORDER BY predicted_duration DESC
LIMIT 10


I am working on the file "07_responsible_ai/heuristic_benchmark.ipynb".
The first task is:
Now, use this benchmark to compute the overall RMSE, so that you can compare with the model.
Can you write Python code for it?

%%bigquery
CREATE TEMPORARY FUNCTION is_peak_hour(start_date TIMESTAMP) aS
(EXTRACT(DAYOFWEEK FROM start_date) BETWEEN 2 AND 6 -- weekday
    AND (
       EXTRACT(HOUR FROM start_date) BETWEEN 6 AND 10
       OR
       EXTRACT(HOUR FROM start_date) BETWEEN 15 AND 18))
;

WITH benchmark AS (
SELECT
   start_station_name,
   is_peak_hour(start_date) AS is_peak,
   AVG(duration) AS predicted_duration,
FROM `bigquery-public-data.london_bicycles.cycle_hire`
GROUP BY 1, 2
)

SELECT
   SQRT( SUM( (duration - predicted_duration)*(duration - predicted_duration)) / COUNT(duration) ) AS rmse
FROM `bigquery-public-data.london_bicycles.cycle_hire` c
JOIN benchmark b
ON c.start_station_name = b.start_station_name AND is_peak_hour(c.start_date) = b.is_peak


I am working on the file "07_responsible_ai/heuristic_benchmark.ipynb".
The first task is:
Problem: Country from which a Stack Overflow question will be answered.
Benchmark: Fractions of answers written by people from France, India, and so on.
Can you write Python code for it?

%%bigquery
SELECT
  COUNTIF(ENDS_WITH(u.location, 'France')) / COUNT(u.location) AS from_france,
  COUNTIF(ENDS_WITH(u.location, 'India')) / COUNT(u.location) AS from_india
FROM `bigquery-public-data.stackoverflow.posts_questions` q
JOIN `bigquery-public-data.stackoverflow.posts_answers` a
ON q.accepted_answer_id = a.id
JOIN `bigquery-public-data.stackoverflow.users` u
ON u.id = a.owner_user_id


I am working on the file "07_sparkml/logistic_regression.ipynb".
The first task is:
<h2> Clean up </h2>
Can you write Python code for it?

trainquery = """
SELECT
  f.*
FROM flights f
JOIN traindays t
ON f.FL_DATE == t.FL_DATE
WHERE
  t.is_train_day == 'True'
"""
traindata = spark.sql(trainquery)

print(traindata.head(2))  # if this is empty, try changing the shard you are using.

trainquery = """
SELECT
  DEP_DELAY, TAXI_OUT, ARR_DELAY, DISTANCE
FROM flights f
JOIN traindays t
ON f.FL_DATE == t.FL_DATE
WHERE
  t.is_train_day == 'True'
"""
traindata = spark.sql(trainquery)

traindata.describe().show()


I am working on the file "07_sparkml/logistic_regression.ipynb".
The first task is:
<h2> Evaluate model </h2>

Evaluate on the test data
Can you write Python code for it?

inputs = 'gs://{}/flights/tzcorr/all_flights-00001-*'.format(BUCKET) # 1/30th
flights = spark.read.json(inputs)
flights.createOrReplaceTempView('flights')

testquery = trainquery.replace("t.is_train_day == 'True'","t.is_train_day == 'False'")
print(testquery)

testdata = spark.sql(testquery)
examples = testdata.rdd.map(to_example)

testdata.describe().show()  # if this is empty, change the shard you are using

def eval(labelpred):
    '''
        data = (label, pred)
            data[0] = label
            data[1] = pred
    '''
    cancel = labelpred.filter(lambda data: data[1] < 0.7)
    nocancel = labelpred.filter(lambda data: data[1] >= 0.7)
    corr_cancel = cancel.filter(lambda data: data[0] == int(data[1] >= 0.7)).count()
    corr_nocancel = nocancel.filter(lambda data: data[0] == int(data[1] >= 0.7)).count()

    cancel_denom = cancel.count()
    nocancel_denom = nocancel.count()
    if cancel_denom == 0:
        cancel_denom = 1
    if nocancel_denom == 0:
        nocancel_denom = 1
    return {'total_cancel': cancel.count(), \
            'correct_cancel': float(corr_cancel)/cancel_denom, \
            'total_noncancel': nocancel.count(), \
            'correct_noncancel': float(corr_nocancel)/nocancel_denom \
           }

# Evaluate model
lrmodel.clearThreshold() # so it returns probabilities
labelpred = examples.map(lambda p: (p.label, lrmodel.predict(p.features)))
print('All flights:')
print(eval(labelpred))

# keep only those examples near the decision threshold
print('Flights near decision threshold:')
labelpred = labelpred.filter(lambda data: data[1] > 0.65 and data[1] < 0.75)
print(eval(labelpred))


I am working on the file "07_sparkml/logistic_regression.ipynb".
The first task is:
<h2> Examine the model behavior </h2>

For dep_delay=20 and taxiout=10, how does the distance affect prediction?
Can you write Python code for it?

lrmodel.clearThreshold() # to make the model produce probabilities
print(lrmodel.predict([20, 10, 500]))

import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np
dist = np.arange(10, 2000, 10)
prob = [lrmodel.predict([20, 10, d]) for d in dist]

sns.set_style("whitegrid")
ax = plt.plot(dist, prob)
plt.xlabel('distance (miles)')
plt.ylabel('probability of ontime arrival')

delay = np.arange(-20, 60, 1)
prob = [lrmodel.predict([d, 10, 500]) for d in delay]
ax = plt.plot(delay, prob)
plt.xlabel('departure delay (minutes)')
plt.ylabel('probability of ontime arrival')


I am working on the file "07_sparkml/logistic_regression.ipynb".
The first task is:
<h2> Predict with the model </h2>

First save the model
Can you write Python code for it?

!gsutil -m rm -r gs://$BUCKET/flights/sparkmloutput/model

MODEL_FILE='gs://' + BUCKET + '/flights/sparkmloutput/model'

lrmodel.save(sc, MODEL_FILE)
print('{} saved'.format(MODEL_FILE))

lrmodel = 0
print(lrmodel)


I am working on the file "07_sparkml/logistic_regression.ipynb".
The first task is:
<h2> Read dataset </h2>
Can you write Python code for it?

traindays = spark.read \
    .option("header", "true") \
    .csv('gs://{}/flights/trainday.csv'.format(BUCKET))
traindays.createOrReplaceTempView('traindays')

spark.sql("SELECT * from traindays LIMIT 5").show()

inputs = 'gs://{}/flights/tzcorr/all_flights-00000-*'.format(BUCKET) # 1/30th
#inputs = 'gs://{}/flights/tzcorr/all_flights-*'.format(BUCKET)  # FULL
flights = spark.read.json(inputs)

# this view can now be queried ...
flights.createOrReplaceTempView('flights')


I am working on the file "07_sparkml/logistic_regression.ipynb".
The first task is:
Now retrieve the model
Can you write Python code for it?

from pyspark.mllib.classification import LogisticRegressionModel
lrmodel = LogisticRegressionModel.load(sc, MODEL_FILE)
lrmodel.setThreshold(0.7)

print(lrmodel.predict([36.0,12.0,594.0]))

print(lrmodel.predict([8.0,4.0,594.0]))


I am working on the file "07_spectral_elements/se_hetero_1d.ipynb".
The first task is:
1. Initialization of setup
Can you write Python code for it?

# Initialization of setup
# ---------------------------------------------------------------
nt    = 10000         # number of time steps
xmax  = 8000.         # Length of domain [m]
N     = 3             # Order of Lagrange polynomials
ne    = 250           # Number of elements
Tdom  = .2            # Dominant period of Ricker source wavelet
iplot = 20            # Plotting each iplot snapshot
vs    = 2500.         # S velocity [m/s]
rho   = 2000          # Density [kg/m^3]

# variables for elemental matrices
Me = np.zeros(N+1, dtype =  float)
Ke = np.zeros((N+1, N+1), dtype =  float)
# ----------------------------------------------------------------

# Initialization of GLL points integration weights
[xi, w] = gll(N)    # xi, N+1 coordinates [-1 1] of GLL points
                    # w Integration weights at GLL locations
# Space domain
le = xmax/ne        # Length of elements
# Vector with GLL points
k = 0
xg = np.zeros((N*ne)+1)
xg[k] = 0
for i in range(1,ne+1):
    for j in range(0,N):
        k = k+1
        xg[k] = (i-1)*le + .5*(xi[j+1]+1)*le

# ---------------------------------------------------------------
dxmin = min(np.diff(xg))
eps = 0.1           # Courant value
dt = eps*dxmin/vs   # Global time step

# Mapping - Jacobian
J = le/2
Ji = 1/J    # Inverse Jacobian

# 1st derivative of Lagrange polynomials
l1d = lagrange1st(N)   # Array with GLL as columns for each N+1 polynomial


I am working on the file "07_spectral_elements/se_hetero_1d.ipynb".
The first task is:
2. Low velocity zone
The introduction of an specific velocity model is done after allowing space-dependent elastic parameters. i.e.
\begin{equation}
\mu(x) = \rho(x) v_s(x)
\end{equation}
Exercise 1
Introduce a low-velocity zone (-40%) at the center of the model spanning 50 elements. Then, visualize your model. Additionally, you can try different velocity and density models by defining new python methods.
Can you write Python code for it?


I am working on the file "07_spectral_elements/se_hetero_1d.ipynb".
The first task is:
3. The Mass Matrix
Now we initialize the mass and stiffness matrices. In general, the mass matrix at the elemental level is given
\begin{equation}
M_{ji}^e \ = \ w_j  \ \rho (\xi)  \ \frac{\mathrm{d}x}{\mathrm{d}\xi} \delta_{ij} \vert_ {\xi = \xi_j}
\end{equation}
Exercise 2
Implements the mass matrix at each element using the integration weights at GLL locations $w$, the jacobian $J$, and density $\rho$. Then, perform the global assembly of the mass matrix, compute its inverse, and display the inverse mass matrix to visually inspect how it looks like.
Can you write Python code for it?


I am working on the file "07_spectral_elements/se_hetero_1d.ipynb".
The first task is:
4. The Stiffness matrix
On the other hand, the general form of the stiffness matrix at the elemtal level is
\begin{equation}
K_{ji}^e \ = \ \sum_{k = 1}^{N+1} w_k \mu (\xi) \partial_\xi \ell_j (\xi) \partial_\xi \ell_i (\xi) \left(\frac{\mathrm{d}\xi}{\mathrm{d}x} \right)^2 \frac{\mathrm{d}x}{\mathrm{d}\xi} \vert_{\xi = \xi_k}
\end{equation}
Exercise 3
Implements the stiffness matrix at each element using the integration weights at GLL locations $w$, the jacobian $J$, and shear stress $\mu$. Then, perform the global assembly of the mass matrix and display the matrix to visually inspect how it looks like.
Can you write Python code for it?


I am working on the file "07_spectral_elements/se_hetero_1d.ipynb".
The first task is:
6. Displaying the wave-field
Can you write Python code for it?

# Solution in space-time
x_t = np.asanyarray(x_t)

# Initialize plot
plt.figure()
plt.subplot(1,1,1)
plt.imshow(x_t, cmap='hot', aspect='auto',
          extent =[0, xmax, nt*dt, 0])
plt.title('u(x,t) Wavefield')
plt.ylabel('Time [s]')
plt.xlabel('Space [m]')
plt.show()


I am working on the file "07_spectral_elements/se_hetero_1d_solution.ipynb".
The first task is:
5. Finite element solution
Finally we implement the spectral element solution using the computed mass $M$ and stiffness $K$ matrices together with a finite differences extrapolation scheme
\begin{equation}
\mathbf{u}(t + dt) = dt^2 (\mathbf{M}^T)^{-1}[\mathbf{f} - \mathbf{K}^T\mathbf{u}] + 2\mathbf{u} - \mathbf{u}(t-dt).
\end{equation}
Can you write Python code for it?

# SE Solution, Time extrapolation
# ---------------------------------------------------------------

# initialize source time function and force vector f
src  = ricker(dt,Tdom)
isrc = int(np.floor(ng/2))   # Source location

# Initialization of solution vectors
u = np.zeros(ng)
uold = u
unew = u
f = u

# Initialize animated plot
# ---------------------------------------------------------------
plt.figure(figsize=(10,6))

# ploting low velocity zone
half = np.floor(xmax/2)
plt.axvspan(half - el_span/2*le,
            half + el_span/2*le,
            alpha=0.2, facecolor='b')
plt.title('SEM 1D Animation')
plt.xlabel(' x (m)')
plt.ylabel(' Amplitude ')
font = {'family': 'serif',
        'color':  'black',
        'weight': 'normal',
        'size': 16,
        }

lines = plt.plot(xg, u, lw=1.5)
plt.title('SEM 1D Animation   $v_{min} = %s v_{max}$' %percent, size=16)
plt.xlabel(' x (m)')
plt.ylabel(' Displacement ')

plt.ion() # set interective mode
plt.show()

# ---------------------------------------------------------------
# Time extrapolation
# ---------------------------------------------------------------
x_t = []
for it in range(nt):
    # Source initialization
    f = np.zeros(ng)
    if it < len(src):
        f[isrc-1] = src[it-1]

    # Time extrapolation
    unew = dt**2 * Minv @ (f - K @ u) + 2 * u - uold
    uold, u = u, unew

    # Solution in space-time
    x_t.append(u)

    # --------------------------------------
    # Animation plot. Display solution
    if not it % iplot:
        for l in lines:
            l.remove()
            del l
        # --------------------------------------
        # Display lines
        lines = plt.plot(xg, u, color="black", lw = 1.5)
        plt.xlim((0,xmax))
        plt.gcf().canvas.draw()


I am working on the file "07_spectral_elements/se_hetero_1d_solution.ipynb".
The first task is:
6. Displaying the wave-field
Can you write Python code for it?


I am working on the file "07_spectral_elements/se_homo_1d.ipynb".
The first task is:
2. The Mass Matrix
Now we initialize the mass and stiffness matrices. In general, the mass matrix at the elemental level is given
\begin{equation}
M_{ji}^e \ = \ w_j  \ \rho (\xi)  \ \frac{\mathrm{d}x}{\mathrm{d}\xi} \delta_{ij} \vert_ {\xi = \xi_j}
\end{equation}
Exercise 1
Implement the mass matrix using the integration weights at GLL locations $w$, the jacobian $J$, and density $\rho$. Then, perform the global assembly of the mass matrix, compute its inverse, and display the inverse mass matrix to visually inspect how it looks like.
Can you write Python code for it?


I am working on the file "07_spectral_elements/se_homo_1d.ipynb".
The first task is:
4. Finite element solution
Finally we implement the spectral element solution using the computed mass $M$ and stiffness $K$ matrices together with a finite differences extrapolation scheme
\begin{equation}
\mathbf{u}(t + dt) = dt^2 (\mathbf{M}^T)^{-1}[\mathbf{f} - \mathbf{K}^T\mathbf{u}] + 2\mathbf{u} - \mathbf{u}(t-dt).
\end{equation}
Can you write Python code for it?

# SE Solution, Time extrapolation
# ---------------------------------------------------------------

# initialize source time function and force vector f
src  = ricker(dt,Tdom)
isrc = int(np.floor(ng/2))   # Source location

# Initialization of solution vectors
u = np.zeros(ng)
uold = u
unew = u
f = u

# Initialize animated plot
# ---------------------------------------------------------------
plt.figure(figsize=(10,6))
lines = plt.plot(xg, u, lw=1.5)
plt.title('SEM 1D Animation', size=16)
plt.xlabel(' x (m)')
plt.ylabel(' Amplitude ')

plt.ion() # set interective mode
plt.show()

# ---------------------------------------------------------------
# Time extrapolation
# ---------------------------------------------------------------
for it in range(nt):
    # Source initialization
    f= np.zeros(ng)
    if it < len(src):
        f[isrc-1] = src[it-1]

    # Time extrapolation
    unew = dt**2 * Minv @ (f - K @ u) + 2 * u - uold
    uold, u = u, unew

    # --------------------------------------
    # Animation plot. Display solution
    if not it % iplot:
        for l in lines:
            l.remove()
            del l
        # --------------------------------------
        # Display lines
        lines = plt.plot(xg, u, color="black", lw = 1.5)
        plt.gcf().canvas.draw()


I am working on the file "07_spectral_elements/se_homo_1d_solution.ipynb".
The first task is:
1. Initialization of setup
Can you write Python code for it?

# Initialization of setup
# ---------------------------------------------------------------
nt    = 10000         # number of time steps
xmax  = 10000.        # Length of domain [m]
vs    = 2500.         # S velocity [m/s]
rho   = 2000          # Density [kg/m^3]
mu    = rho * vs**2   # Shear modulus mu
N     = 3             # Order of Lagrange polynomials
ne    = 250           # Number of elements
Tdom  = .2            # Dominant period of Ricker source wavelet
iplot = 20            # Plotting each iplot snapshot

# variables for elemental matrices
Me = np.zeros(N+1, dtype =  float)
Ke = np.zeros((N+1, N+1), dtype =  float)
# ----------------------------------------------------------------

# Initialization of GLL points integration weights
[xi, w] = gll(N)    # xi, N+1 coordinates [-1 1] of GLL points
                    # w Integration weights at GLL locations
# Space domain
le = xmax/ne        # Length of elements
# Vector with GLL points
k = 0
xg = np.zeros((N*ne)+1)
xg[k] = 0
for i in range(1,ne+1):
    for j in range(0,N):
        k = k+1
        xg[k] = (i-1)*le + .5*(xi[j+1]+1)*le

# ---------------------------------------------------------------
dxmin = min(np.diff(xg))
eps = 0.1           # Courant value
dt = eps*dxmin/vs   # Global time step

# Mapping - Jacobian
J = le/2
Ji = 1/J    # Inverse Jacobian

# 1st derivative of Lagrange polynomials
l1d = lagrange1st(N)   # Array with GLL as columns for each N+1 polynomial


I am working on the file "07_spectral_elements/se_homo_1d_solution.ipynb".
The first task is:
2. The Mass Matrix
Now we initialize the mass and stiffness matrices. In general, the mass matrix at the elemental level is given
\begin{equation}
M_{ji}^e \ = \ w_j  \ \rho (\xi)  \ \frac{\mathrm{d}x}{\mathrm{d}\xi} \delta_{ij} \vert_ {\xi = \xi_j}
\end{equation}
Exercise 1
Implements the mass matrix using the integration weights at GLL locations $w$, the jacobian $J$, and density $\rho$. Then, perform the global assembly of the mass matrix, compute its inverse, and display the inverse mass matrix to visually inspect how it looks like.
Can you write Python code for it?


I am working on the file "07_spectral_elements/se_homo_1d_solution.ipynb".
The first task is:
4. Finite element solution
Finally we implement the spectral element solution using the computed mass $M$ and stiffness $K$ matrices together with a finite differences extrapolation scheme
\begin{equation}
\mathbf{u}(t + dt) = dt^2 (\mathbf{M}^T)^{-1}[\mathbf{f} - \mathbf{K}^T\mathbf{u}] + 2\mathbf{u} - \mathbf{u}(t-dt).
\end{equation}
Can you write Python code for it?


I am working on the file "07_tensorboard_ex6.ipynb".
The first task is:
6.3)
Can you write Python code for it?

filenames = ["data/life_satisfaction.csv"]
n_epochs = 500
batch_size = 5

graph = tf.Graph()
with graph.as_default():
    with tf.name_scope("reader"):
        reader = tf.TextLineReader(skip_header_lines=1)

        filename_queue = tf.train.string_input_producer(filenames, num_epochs=n_epochs)
        record_id, record = reader.read(filename_queue)

        record_defaults = [[''], [0.0], [0.0]]
        country, gdp_per_capita, life_satisfaction = tf.decode_csv(record, record_defaults=record_defaults)

        X_batch, y_batch = tf.train.batch([gdp_per_capita, life_satisfaction], batch_size=batch_size)
        X_batch_reshaped = tf.reshape(X_batch, [-1, 1])
        y_batch_reshaped = tf.reshape(y_batch, [-1, 1])

    with tf.name_scope("linear_model"):
        X = tf.placeholder_with_default(X_batch_reshaped, shape=[None, 1], name="X")
        y = tf.placeholder_with_default(y_batch_reshaped, shape=[None, 1], name="y")

        b = tf.Variable(0.0, name="b")
        w = tf.Variable(tf.zeros([1, 1]), name="w")
        y_pred = tf.add(tf.matmul(X / 10000, w), b, name="y_pred")  # X @ w + b

    with tf.name_scope("train"):
        mse = tf.reduce_mean(tf.square(y_pred - y), name="mse")
        global_step = tf.Variable(0, trainable=False, name='global_step')
        optimizer = tf.train.MomentumOptimizer(learning_rate, momentum)
        training_op = optimizer.minimize(mse, global_step=global_step)

    init = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())
    saver = tf.train.Saver()

summary_writer = tf.summary.FileWriter("tf_logs/run_number_2_solution/", graph=graph)


I am working on the file "07_tensorboard_ex6.ipynb".
The first task is:
6.5)
Can you write Python code for it?

graph1 = tf.Graph()
with graph1.as_default():
    a = tf.constant(1.0)

summary_writer = tf.summary.FileWriter(logdir(), graph=graph)

graph2 = tf.Graph()
with graph2.as_default():
    a = tf.constant(1.0, name="a")
    b = tf.Variable(2.0, name="b")
    c = a * b


I am working on the file "07_tensorboard_ex6.ipynb".
The first task is:
6.6)
Can you write Python code for it?

filenames = ["data/life_satisfaction.csv"]
n_epochs = 500
batch_size = 5

graph = tf.Graph()
with graph.as_default():
    with tf.name_scope("reader"):
        reader = tf.TextLineReader(skip_header_lines=1)

        filename_queue = tf.train.string_input_producer(filenames, num_epochs=n_epochs)
        record_id, record = reader.read(filename_queue)

        record_defaults = [[''], [0.0], [0.0]]
        country, gdp_per_capita, life_satisfaction = tf.decode_csv(record, record_defaults=record_defaults)

        X_batch, y_batch = tf.train.batch([gdp_per_capita, life_satisfaction], batch_size=batch_size)
        X_batch_reshaped = tf.reshape(X_batch, [-1, 1])
        y_batch_reshaped = tf.reshape(y_batch, [-1, 1])

    with tf.name_scope("linear_model"):
        X = tf.placeholder_with_default(X_batch_reshaped, shape=[None, 1], name="X")
        y = tf.placeholder_with_default(y_batch_reshaped, shape=[None, 1], name="y")

        b = tf.Variable(0.0, name="b")
        w = tf.Variable(tf.zeros([1, 1]), name="w")
        y_pred = tf.add(tf.matmul(X / 10000, w), b, name="y_pred")

    with tf.name_scope("train"):
        mse = tf.reduce_mean(tf.square(y_pred - y), name="mse")
        mse_summary = tf.summary.scalar('MSE', mse)                     # <= ADDED
        global_step = tf.Variable(0, trainable=False, name='global_step')
        optimizer = tf.train.MomentumOptimizer(learning_rate, momentum)
        training_op = optimizer.minimize(mse, global_step=global_step)

    init = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())
    saver = tf.train.Saver()

summary_writer = tf.summary.FileWriter(logdir(), graph)

with tf.Session(graph=graph) as sess:
    init.run()
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(coord=coord)
    try:
        while not coord.should_stop():
            _, mse_summary_val, global_step_val = sess.run([training_op, mse_summary, global_step])
            if global_step_val % 10 == 0:
               ...


I am working on the file "07_tensorboard_ex6.ipynb".
The first task is:
Create a few different graphs and instantiate a different FileWriter for each one, using a different log directory every time (with the help of the logdir() function). Refresh TensorBoard and notice that you can browse any graph you want by selecting the appropriate run.
6.6) Now we will use TensorBoard to visualize the learning curve, that is the evolution of the cost function during training.

First add a scalar summary operation in the graph, using tf.summary.scalar("MSE", mse).
Next, update the training code to evaluate this scalar summary and write the result to the events file using the FileWriter's add_summary() method (also specifying the training step). For performance reasons, you probably want to do this only every 10 training iterations or so.
Next, train the model.
Refresh TensorBoard, and visit the Scalars tab. Select the appropriate run and visualize the learning curve. Try zooming in and out, and play around with the options, in particular the smoothing option.

Try not to peek at the solution below before you have done the exercise! :)

Exercise 6 - Solution
6.1)
N/A
6.2)
Can you write Python code for it?

summary_writer = tf.summary.FileWriter("tf_logs/run_number_1_solution/", graph=graph)


I am working on the file "07_tensorboard_ex6.ipynb".
The first task is:
From previous notebooks
Can you write Python code for it?