I am working on the file "08_Text_Analysis/06 - Part_4 - Text Classification - Hacker News - DNNClassifier with TF.IDF.ipynb".
The first task is:
5.2 Serving function
Can you write Python code for it?


I am working on the file "08_Text_Analysis/06 - Part_4 - Text Classification - Hacker News - DNNClassifier with TF.IDF.ipynb".
The first task is:
5.3 TrainSpec & EvalSpec
Can you write Python code for it?

train_spec = tf.estimator.TrainSpec(
    input_fn = generate_tfrecords_input_fn(
        Params.TRANSFORMED_TRAIN_DATA_FILE_PREFIX+"*",
        mode = tf.estimator.ModeKeys.TRAIN,
        num_epochs=hparams.num_epochs,
        batch_size=hparams.batch_size
    ),
    max_steps=hparams.max_steps,
    hooks=None
)

eval_spec = tf.estimator.EvalSpec(
    input_fn = generate_tfrecords_input_fn(
        Params.TRANSFORMED_EVAL_DATA_FILE_PREFIX+"*",
        mode=tf.estimator.ModeKeys.EVAL,
        num_epochs=1,
        batch_size=hparams.batch_size
    ),
    exporters=[tf.estimator.LatestExporter(
        name="estimate", # the name of the folder in which the model will be exported to under export
        serving_input_receiver_fn=generate_serving_input_fn(),
        exports_to_keep=1,
        as_text=False)],
    steps=None,
    throttle_secs=EVAL_EVERY_SEC
)


I am working on the file "08_Text_Analysis/06 - Part_4 - Text Classification - Hacker News - DNNClassifier with TF.IDF.ipynb".
The first task is:
6. Run experiment
Can you write Python code for it?

from datetime import datetime
import shutil

if Params.TRAIN:
    if not Params.RESUME_TRAINING:
        print("Removing previous training artefacts...")
        shutil.rmtree(model_dir, ignore_errors=True)
    else:
        print("Resuming training...")

    tf.logging.set_verbosity(tf.logging.INFO)

    time_start = datetime.utcnow()
    print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
    print(".......................................")

    estimator = create_estimator(hparams, run_config)

    tf.estimator.train_and_evaluate(
        estimator=estimator,
        train_spec=train_spec,
        eval_spec=eval_spec
    )

    time_end = datetime.utcnow()
    print(".......................................")
    print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
    print("")
    time_elapsed = time_end - time_start
    print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))
else:
    print "Training was skipped!"


I am working on the file "08_Text_Analysis/06 - Part_4 - Text Classification - Hacker News - DNNClassifier with TF.IDF.ipynb".
The first task is:
Importing libraries
Can you write Python code for it?

import tensorflow as tf
from tensorflow import data

from tensorflow.contrib.learn.python.learn.utils import input_fn_utils
from tensorflow_transform.beam.tft_beam_io import transform_fn_io
from tensorflow_transform.tf_metadata import metadata_io
from tensorflow_transform.tf_metadata import dataset_schema
from tensorflow_transform.tf_metadata import dataset_metadata
from tensorflow_transform.saved import saved_transform_io

print tf.__version__


I am working on the file "08_Text_Analysis/06 - Part_4 - Text Classification - Hacker News - DNNClassifier with TF.IDF.ipynb".
The first task is:
Text Classification using TensorFlow and Google Cloud - Part 4
This bigquery-public-data:hacker_news contains all stories and comments from Hacker News from its launch in 2006.  Each story contains a story id, url, the title of the story, tthe author that made the post, when it was written, and the number of points the story received.
The objective is, given the title of the story, we want to build an ML model that can predict the source of this story.
TF DNNClassifier with TF.IDF Text Reprsentation
This notebook illustrates how to build a TF premade estimator, namely DNNClassifier, while the input text will be repesented as TF.IDF computed during the preprocessing phase in Part 1. The overall steps are as follows:

Define the metadata
Define data input function
Create feature columns (using the tfidf)
Create the premade DNNClassifier estimator
Setup experiement
Hyper-parameters & RunConfig
Serving function (for exported model)
TrainSpec & EvalSpec

Run experiement
Evalute the model
Use SavedModel for prediction

Setting Global Parameters
Can you write Python code for it?

import os

class Params:
    pass

# Set to run on GCP
Params.GCP_PROJECT_ID = 'ksalama-gcp-playground'
Params.REGION = 'europe-west1'
Params.BUCKET = 'ksalama-gcs-cloudml'

Params.PLATFORM = 'local' # local | GCP

Params.DATA_DIR = 'data/news'  if Params.PLATFORM == 'local' else 'gs://{}/data/news'.format(Params.BUCKET)

Params.TRANSFORMED_DATA_DIR = os.path.join(Params.DATA_DIR, 'transformed')
Params.TRANSFORMED_TRAIN_DATA_FILE_PREFIX = os.path.join(Params.TRANSFORMED_DATA_DIR, 'train')
Params.TRANSFORMED_EVAL_DATA_FILE_PREFIX = os.path.join(Params.TRANSFORMED_DATA_DIR, 'eval')

Params.TEMP_DIR = os.path.join(Params.DATA_DIR, 'tmp')

Params.MODELS_DIR = 'models/news' if Params.PLATFORM == 'local' else 'gs://{}/models/news'.format(Params.BUCKET)

Params.TRANSFORM_ARTEFACTS_DIR = os.path.join(Params.MODELS_DIR,'transform')

Params.TRAIN = True

Params.RESUME_TRAINING = False

Params.EAGER = False

if Params.EAGER:
    tf.enable_eager_execution()


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Analysis of Transfer-Values using PCA
Use Principal Component Analysis (PCA) from scikit-learn to reduce the array-lengths of the transfer-values from 2048 to 2 so they can be plotted.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Another method for doing dimensionality reduction is t-SNE. Unfortunately, t-SNE is very slow so we first use PCA to reduce the transfer-values from 2048 to 50 elements.
Can you write Python code for it?

pca = PCA(n_components=50)
transfer_values_50d = pca.fit_transform(transfer_values)


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Calculate Transfer-Values
Import a helper-function for caching the transfer-values of the Inception model.
Can you write Python code for it?

from inception import transfer_values_cache


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Calculate the predicted class for the test-set.
Can you write Python code for it?

def predict_cls_test():
    return predict_cls(transfer_values = transfer_values_test,
                       labels = labels_test,
                       cls_true = cls_test)


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Calculate the true class as an integer. This could also be a placeholder variable.
Can you write Python code for it?

y_true_cls = tf.argmax(y_true, dimension=1)


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Check that it is now an array with 3000 samples and 2 transfer-values per sample.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Check that it is now an array with 3000 samples and 2 values per sample.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Check that the array has 3000 samples and 2048 transfer-values for each sample.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Check the shape of the array with the transfer-values. There are 50,000 images in the training-set and for each image there are 2048 transfer-values.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Classification Accuracy
The output of the network y_pred is an array with 10 elements. The class number is the index of the largest element in the array.
Can you write Python code for it?

y_pred_cls = tf.argmax(y_pred, dimension=1)


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Close TensorFlow Session
We are now done using TensorFlow, so we close the session to release its resources. Note that there are two TensorFlow-sessions so we close both, one session is inside the model-object.
Can you write Python code for it?

# This has been commented out in case you want to modify and experiment
# with the Notebook without having to restart it.
# model.close()
# session.close()


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Create a new PCA-object and set the target array-length to 2.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Create a new t-SNE object for the final dimensionality reduction and set the target to 2-dim.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Create an array of booleans whether the predicted class equals the true class of each image.
Can you write Python code for it?

correct_prediction = tf.equal(y_pred_cls, y_true_cls)


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Create another placeholder variable for inputting the true class-label of each image. These are so-called One-Hot encoded arrays with 10 elements, one for each possible class in the data-set.
Can you write Python code for it?

y_true = tf.placeholder(tf.float32, shape=[None, num_classes], name='y_true')


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Download the Inception Model
The Inception model is downloaded from the internet. This is the default directory where you want to save the data-files. The directory will be created if it does not exist.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Download the data for the Inception model if it doesn't already exist in the directory. It is 85 MB.
See Tutorial #07 for more details.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Function for selecting a random batch of transfer-values from the training-set.
Can you write Python code for it?

def random_batch():
    # Number of images (transfer-values) in the training-set.
    num_images = len(transfer_values_train)

    # Create a random index.
    idx = np.random.choice(num_images,
                           size=train_batch_size,
                           replace=False)

    # Use the random index to select random x and y-values.
    # We use the transfer-values instead of images as x-values.
    x_batch = transfer_values_train[idx]
    y_batch = labels_train[idx]

    return x_batch, y_batch


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Get the class-numbers for the samples you selected.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Helper-Functions for Showing Results
Helper-function to plot example errors
Function for plotting examples of images from the test-set that have been mis-classified.
Can you write Python code for it?

def plot_example_errors(cls_pred, correct):
    # This function is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # correct is a boolean array whether the predicted class
    # is equal to the true class for each image in the test-set.

    # Negate the boolean array.
    incorrect = (correct == False)

    # Get the images from the test-set that have been
    # incorrectly classified.
    images = images_test[incorrect]

    # Get the predicted classes for those images.
    cls_pred = cls_pred[incorrect]

    # Get the true classes for those images.
    cls_true = cls_test[incorrect]

    n = min(9, len(images))

    # Plot the first n images.
    plot_images(images=images[0:n],
                cls_true=cls_true[0:n],
                cls_pred=cls_pred[0:n])


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Helper-function for plotting images
Function used to plot at most 9 images in a 3x3 grid, and writing the true and predicted classes below each image.
Can you write Python code for it?

def plot_images(images, cls_true, cls_pred=None, smooth=True):

    assert len(images) == len(cls_true)

    # Create figure with sub-plots.
    fig, axes = plt.subplots(3, 3)

    # Adjust vertical spacing.
    if cls_pred is None:
        hspace = 0.3
    else:
        hspace = 0.6
    fig.subplots_adjust(hspace=hspace, wspace=0.3)

    # Interpolation type.
    if smooth:
        interpolation = 'spline16'
    else:
        interpolation = 'nearest'

    for i, ax in enumerate(axes.flat):
        # There may be less than 9 images, ensure it doesn't crash.
        if i < len(images):
            # Plot image.
            ax.imshow(images[i],
                      interpolation=interpolation)

            # Name of the true class.
            cls_true_name = class_names[cls_true[i]]

            # Show true and predicted classes.
            if cls_pred is None:
                xlabel = "True: {0}".format(cls_true_name)
            else:
                # Name of the predicted class.
                cls_pred_name = class_names[cls_pred[i]]

                xlabel = "True: {0}\nPred: {1}".format(cls_true_name, cls_pred_name)

            # Show the classes as the label on the x-axis.
            ax.set_xlabel(xlabel)

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Helper-function for plotting the reduced transfer-values.
Can you write Python code for it?

def plot_scatter(values, cls):
    # Create a color-map with a different color for each class.
    import matplotlib.cm as cm
    cmap = cm.rainbow(np.linspace(0.0, 1.0, num_classes))

    # Get the color for each sample.
    colors = cmap[cls]

    # Extract the x- and y-values.
    x = values[:, 0]
    y = values[:, 1]

    # Plot it.
    plt.scatter(x, y, color=colors)
    plt.show()


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Helper-function for plotting transfer-values
Can you write Python code for it?

def plot_transfer_values(i):
    print("Input image:")

    # Plot the i'th image from the test-set.
    plt.imshow(images_test[i], interpolation='nearest')
    plt.show()

    print("Transfer-values for the image using Inception model:")

    # Transform the transfer-values into an image.
    img = transfer_values_test[i]
    img = img.reshape((32, 64))

    # Plot the image for the transfer-values.
    plt.imshow(img, interpolation='nearest', cmap='Reds')
    plt.show()

plot_transfer_values(i=16)

plot_transfer_values(i=17)


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Helper-function for showing the classification accuracy
Function for printing the classification accuracy on the test-set.
It takes a while to compute the classification for all the images in the test-set, that's why the results are re-used by calling the above functions directly from this function, so the classifications don't have to be recalculated by each function.
Can you write Python code for it?

def print_test_accuracy(show_example_errors=False,
                        show_confusion_matrix=False):

    # For all the images in the test-set,
    # calculate the predicted classes and whether they are correct.
    correct, cls_pred = predict_cls_test()

    # Classification accuracy and the number of correct classifications.
    acc, num_correct = classification_accuracy(correct)

    # Number of images being classified.
    num_images = len(correct)

    # Print the accuracy.
    msg = "Accuracy on Test-Set: {0:.1%} ({1} / {2})"
    print(msg.format(acc, num_correct, num_images))

    # Plot some examples of mis-classifications, if desired.
    if show_example_errors:
        print("Example errors:")
        plot_example_errors(cls_pred=cls_pred, correct=correct)

    # Plot the confusion matrix, if desired.
    if show_confusion_matrix:
        print("Confusion Matrix:")
        plot_confusion_matrix(cls_pred=cls_pred)


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Helper-function to get a random training-batch
There are 50,000 images (and arrays with transfer-values for the images) in the training-set. It takes a long time to calculate the gradient of the model using all these images (transfer-values). We therefore only use a small batch of images (transfer-values) in each iteration of the optimizer.
If your computer crashes or becomes very slow because you run out of RAM, then you may try and lower this number, but you may then need to perform more optimization iterations.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Helper-function to perform optimization
This function performs a number of optimization iterations so as to gradually improve the variables of the neural network. In each iteration, a new batch of data is selected from the training-set and then TensorFlow executes the optimizer using those training samples.  The progress is printed every 100 iterations.
Can you write Python code for it?

def optimize(num_iterations):
    # Start-time used for printing time-usage below.
    start_time = time.time()

    for i in range(num_iterations):
        # Get a batch of training examples.
        # x_batch now holds a batch of images (transfer-values) and
        # y_true_batch are the true labels for those images.
        x_batch, y_true_batch = random_batch()

        # Put the batch into a dict with the proper names
        # for placeholder variables in the TensorFlow graph.
        feed_dict_train = {x: x_batch,
                           y_true: y_true_batch}

        # Run the optimizer using this batch of training data.
        # TensorFlow assigns the variables in feed_dict_train
        # to the placeholder variables and then runs the optimizer.
        # We also want to retrieve the global_step counter.
        i_global, _ = session.run([global_step, optimizer],
                                  feed_dict=feed_dict_train)

        # Print status to screen every 100 iterations (and last).
        if (i_global % 100 == 0) or (i == num_iterations - 1):
            # Calculate the accuracy on the training-batch.
            batch_acc = session.run(accuracy,
                                    feed_dict=feed_dict_train)

            # Print status.
            msg = "Global Step: {0:>6}, Training Batch Accuracy: {1:>6.1%}"
            print(msg.format(i_global, batch_acc))

    # Ending time.
    end_time = time.time()

    # Difference between start and end-times.
    time_dif = end_time - start_time

    # Print the time-usage.
    print("Time usage: " + str(timedelta(seconds=int(round(time_dif)))))


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Helper-function to plot confusion matrix
Can you write Python code for it?

# Import a function from sklearn to calculate the confusion-matrix.
from sklearn.metrics import confusion_matrix

def plot_confusion_matrix(cls_pred):
    # This is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # Get the confusion matrix using sklearn.
    cm = confusion_matrix(y_true=cls_test,  # True class for test-set.
                          y_pred=cls_pred)  # Predicted class.

    # Print the confusion matrix as text.
    for i in range(num_classes):
        # Append the class-name to each line.
        class_name = "({}) {}".format(i, class_names[i])
        print(cm[i, :], class_name)

    # Print the class-numbers for easy reference.
    class_numbers = [" ({0})".format(i) for i in range(num_classes)]
    print("".join(class_numbers))


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Helper-functions for calculating classifications
This function calculates the predicted classes of images and also returns a boolean array whether the classification of each image is correct.
The calculation is done in batches because it might use too much RAM otherwise. If your computer crashes then you can try and lower the batch-size.
Can you write Python code for it?

# Split the data-set in batches of this size to limit RAM usage.
batch_size = 256

def predict_cls(transfer_values, labels, cls_true):
    # Number of images.
    num_images = len(transfer_values)

    # Allocate an array for the predicted classes which
    # will be calculated in batches and filled into this array.
    cls_pred = np.zeros(shape=num_images, dtype=np.int)

    # Now calculate the predicted classes for the batches.
    # We will just iterate through all the batches.
    # There might be a more clever and Pythonic way of doing this.

    # The starting index for the next batch is denoted i.
    i = 0

    while i < num_images:
        # The ending index for the next batch is denoted j.
        j = min(i + batch_size, num_images)

        # Create a feed-dict with the images and labels
        # between index i and j.
        feed_dict = {x: transfer_values[i:j],
                     y_true: labels[i:j]}

        # Calculate the predicted class using TensorFlow.
        cls_pred[i:j] = session.run(y_pred_cls, feed_dict=feed_dict)

        # Set the start-index for the next batch to the
        # end-index of the current batch.
        i = j

    # Create a boolean array whether each image is correctly classified.
    correct = (cls_true == cls_pred)

    return correct, cls_pred


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Helper-functions for calculating the classification accuracy
This function calculates the classification accuracy given a boolean array whether each image was correctly classified. E.g. classification_accuracy([True, True, False, False, False]) = 2/5 = 0.4. The function also returns the number of correct classifications.
Can you write Python code for it?

def classification_accuracy(correct):
    # When averaging a boolean array, False means 0 and True means 1.
    # So we are calculating: number of True / len(correct) which is
    # the same as the classification accuracy.

    # Return the classification accuracy
    # and the number of correct classifications.
    return correct.mean(), correct.sum()


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Imports
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np
import time
from datetime import timedelta
import os

# Functions and classes for loading and using the Inception model.
import inception

# We use Pretty Tensor to define the new classifier.
import prettytensor as pt


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Initialize Variables
The variables for the new network must be initialized before we start optimizing them.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
It takes a while to compute the PCA so the number of samples has been limited to 3000. You can try and use the full training-set if you like.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Load Data for CIFAR-10
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Load the Inception Model
Load the Inception model so it is ready for classifying images.
Note the deprecation warning, which might cause the program to fail in the future.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Load the class-names.
Can you write Python code for it?

class_names = cifar10.load_class_names()
class_names


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Load the test-set.
Can you write Python code for it?

images_test, cls_test, labels_test = cifar10.load_test_data()


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Load the training-set. This returns the images, the class-numbers as integers, and the class-numbers as One-Hot encoded arrays called labels.
Can you write Python code for it?

images_train, cls_train, labels_train = cifar10.load_training_data()


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Method for optimizing the new neural network.
Can you write Python code for it?

optimizer = tf.train.AdamOptimizer(learning_rate=1e-4).minimize(loss, global_step)


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Neural Network
Create the neural network for doing the classification on the CIFAR-10 data-set. This takes as input the transfer-values from the Inception model which will be fed into the placeholder variable x. The network outputs the predicted class in y_pred.
See Tutorial #03 for more details on how to use Pretty Tensor to construct neural networks.
Can you write Python code for it?

# Wrap the transfer-values as a Pretty Tensor object.
x_pretty = pt.wrap(x)

with pt.defaults_scope(activation_fn=tf.nn.relu):
    y_pred, loss = x_pretty.\
        fully_connected(size=1024, name='layer_fc1').\
        softmax_classifier(class_count=num_classes, labels=y_true)


# Wrap the transfer-values as a Pretty Tensor object.
x_pretty = pt.wrap(x)

with pt.defaults_scope(activation_fn=tf.nn.relu):
    y_pred, loss = x_pretty.\
        fully_connected(size=1024, name='layer_fc1').\
        softmax_classifier(num_classes=num_classes, labels=y_true)


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
New Classifier in TensorFlow
Now we will create another neural network in TensorFlow. This network will take as input the transfer-values from the Inception model and output the predicted classes for CIFAR-10 images.
It is assumed that you are already familiar with how to build neural networks in TensorFlow, otherwise see e.g. Tutorial #03.
Placeholder Variables
First we need the array-length for transfer-values which is stored as a variable in the object for the Inception model.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Now create a placeholder variable for inputting the transfer-values from the Inception model into the new network that we are building. The shape of this variable is [None, transfer_len] which means it takes an input array with an arbitrary number of samples as indicated by the keyword None and each sample has 2048 elements, equal to transfer_len.
Can you write Python code for it?

x = tf.placeholder(tf.float32, shape=[None, transfer_len], name='x')


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Optimization Method
Create a variable for keeping track of the number of optimization iterations performed.
Can you write Python code for it?

global_step = tf.Variable(initial_value=0,
                          name='global_step', trainable=False)


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Perform the final reduction using t-SNE. The current implemenation of t-SNE in scikit-learn cannot handle data with many samples so this might crash if you use the full training-set.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Performance after 10,000 optimization iterations
After 10,000 optimization iterations, the classification accuracy is about 90% on the test-set. Compare this to the basic Convolutional Neural Network from Tutorial #06 which had less than 80% accuracy on the test-set.
Can you write Python code for it?

optimize(num_iterations=10000)

print_test_accuracy(show_example_errors=True,
                    show_confusion_matrix=True)


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Plot a few images to see if data is correct
Can you write Python code for it?

# Get the first images from the test-set.
images = images_test[0:9]

# Get the true classes for those images.
cls_true = cls_test[0:9]

# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true, smooth=False)


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Plot the transfer-values that have been reduced to 2-dim using t-SNE, which shows better separation than the PCA-plot above.
This means the transfer-values from the Inception model appear to contain enough information to separate the CIFAR-10 images into classes, although there is still some overlap so the separation is not perfect.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Plot the transfer-values that have been reduced using PCA. There are 10 different colors for the different classes in the CIFAR-10 data-set. The colors are grouped together but with very large overlap. This may be because PCA cannot properly separate the transfer-values.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
PrettyTensor version:
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Results
Performance before any optimization
The classification accuracy on the test-set is very low because the model variables have only been initialized and not optimized at all, so it just classifies the images randomly.
Can you write Python code for it?

print_test_accuracy(show_example_errors=False,
                    show_confusion_matrix=False)


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Set the file-paths for the caches of the training-set and test-set.
Can you write Python code for it?

file_path_cache_train = os.path.join(cifar10.data_path, 'inception_cifar10_train.pkl')
file_path_cache_test = os.path.join(cifar10.data_path, 'inception_cifar10_test.pkl')

print("Processing Inception transfer-values for training-images ...")

# Scale images because Inception needs pixels to be between 0 and 255,
# while the CIFAR-10 functions return pixels between 0.0 and 1.0
images_scaled = images_train * 255.0

# If transfer-values have already been calculated then reload them,
# otherwise calculate them and save them to a cache-file.
transfer_values_train = transfer_values_cache(cache_path=file_path_cache_train,
                                              images=images_scaled,
                                              model=model)

print("Processing Inception transfer-values for test-images ...")

# Scale images because Inception needs pixels to be between 0 and 255,
# while the CIFAR-10 functions return pixels between 0.0 and 1.0
images_scaled = images_test * 255.0

# If transfer-values have already been calculated then reload them,
# otherwise calculate them and save them to a cache-file.
transfer_values_test = transfer_values_cache(cache_path=file_path_cache_test,
                                             images=images_scaled,
                                             model=model)


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Set the path for storing the data-set on your computer.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Similarly, there are 10,000 images in the test-set with 2048 transfer-values for each image.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
TensorFlow Run
Create TensorFlow Session
Once the TensorFlow graph has been created, we have to create a TensorFlow session which is used to execute the graph.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
The CIFAR-10 data-set is about 163 MB and will be downloaded automatically if it is not located in the given path.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
The classification accuracy is calculated by first type-casting the array of booleans to floats, so that False becomes 0 and True becomes 1, and then taking the average of these numbers.
Can you write Python code for it?

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
The data dimensions have already been defined in the cifar10 module, so we just need to import the ones we need.
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
This was developed using Python 3.5.2 (Anaconda) and TensorFlow version:
Can you write Python code for it?


I am working on the file "08_Transfer_Learning.ipynb".
The first task is:
Use PCA to reduce the transfer-value arrays from 2048 to 2 elements.
Can you write Python code for it?


I am working on the file "08_Writing_Reading_Files_Lecture.ipynb".
The first task is:
I used while loop to show the logic behind the reading, however for loop gives us a more elegant way.
Can you write Python code for it?

input_file = open('data/sample.txt', 'r')
for line in input_file:
    print(line)


I am working on the file "08_Writing_Reading_Files_Lecture.ipynb".
The first task is:
Reading Files
The readline method returns as a string the next line of a text file, including the end-of-line character, \n.When the end-of-file is reached, it returns an empty string:
Can you write Python code for it?

input_file = open('data/sample.txt', 'r')
empty_str = ''
line = input_file.readline()
while line != empty_str:
    print(line)
    line = input_file.readline()

input_file.close()


I am working on the file "08_Writing_Reading_Files_Lecture.ipynb".
The first task is:
Removing the Contents of a String
str.strip(w): Returns a copy of str with all leading and trailing characters that appear in w removed.
Can you write Python code for it?


I am working on the file "08_Writing_Reading_Files_Lecture.ipynb".
The first task is:
Replacing the Contents of a String
str.replace(w,t): Returns a copy of str wita ll occurrences of w replaced with t.
Can you write Python code for it?


I am working on the file "08_Writing_Reading_Files_Lecture.ipynb".
The first task is:
Searching the Contents of a String
str.find(w): Returns the index of the first occurrence of w in str, Returns -1 if not found
Can you write Python code for it?


I am working on the file "08_Writing_Reading_Files_Lecture.ipynb".
The first task is:
The write method does not add a newline character to the output string . Thus, a newline character will be output only if it is part of the string being written. But in the example above line variable comes with \n at the end.
String Processing
The information in a text file, as with all information, is most likely going to be searched, analyzed, and/or updated. Collectively, the operations performed on strings is called string processing.
We have learned some basic operations on strings such as

accessing elements: name[k]
getting the length: len(str)

String Traversal: The characters in a string can be easily traversed, without the use of an explicit index variable, using the for chr in string form of the for statement.
Can you write Python code for it?

space = ' '
num_spaces = 0
line = input_file.readline()
for k in range(0, len(line)):
    if line[k] == space:
        num_spaces = num_spaces + 1

num_spaces


I am working on the file "08_Writing_Reading_Files_Lecture.ipynb".
The first task is:
We won't get any error because we are creating a new file only error we might get is harddisk full error from system. After we are done manipulating the file we have to close the file. with close() function.
Can you write Python code for it?


I am working on the file "08_Writing_Reading_Files_Lecture.ipynb".
The first task is:
str.lower() and str.upper(): Returns lower/upper case version of str
Can you write Python code for it?

s

s.upper()

s.lower()

s # Does not change... You have to assign it to an new variable or overwrite

s = s.lower()

s


I am working on the file "08_artifical_neural_networks_ex7ex8.ipynb".
The first task is:
8.2)
Can you write Python code for it?

n_epochs = 20
batch_size = 50

with tf.Session(graph=graph) as sess:
    init.run()
    for epoch in range(n_epochs):
        for iteration in range(mnist.train.num_examples // batch_size):
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
        acc_train = accuracy.eval(feed_dict={X: X_batch, y: y_batch})
        acc_val = accuracy.eval(feed_dict={X: mnist.validation.images, y: mnist.validation.labels})
        print(epoch, "Train accuracy:", acc_train, "Validation accuracy:", acc_val)

    save_path = saver.save(sess, "./my_mnist_model")


I am working on the file "08_artifical_neural_networks_ex7ex8.ipynb".
The first task is:
8.2) Write the training code, and train the model for about 20 epochs (i.e. enough training iterations to go through the training set 20 times). Evaluate it on the test set: you should get over 95% accuracy.
Hint: you should open a session, initialize the variables, then write the main training loop. Inside it you should use minst.train.next_batch(batch_size) to get the next training batch (say with batch_size=50), then run the training_op, feeding it the training batch (don't forget to feed both X and y). Every few hundred iterations, evaluate the model's accuracy on the validation set (mnist.validation.images and mnist.validation.labels), and print the result. At the end of training, save the model.
8.3) Bonus question: load the model you just trained and saved, and use it to make predictions on the first 200 images of the test set (mnist.test). Display the images that the model got wrong, and show the class probabilities that it guessed. Notice that some of the images it gets wrong are pretty poorly written, but some are obvious to us humans. We will see that Convolutional Neural Networks can do a much better job and reach human performance.
Try not to peek at the solution below before you have done the exercise! :)

Exercise 8 - Solution
8.1)
Can you write Python code for it?

n_inputs = 28 * 28
n_hidden1 = 100
n_hidden2 = 100
n_outputs = 10

graph = tf.Graph()
with graph.as_default():
    with tf.name_scope("inputs"):
        X = tf.placeholder(tf.float32, shape=[None, n_inputs], name="X")
        y = tf.placeholder(tf.int32, shape=[None], name="y")

    with tf.name_scope("hidden1"):
        b1 = tf.Variable(tf.zeros([n_hidden1]), name="b1")
        W1 = tf.Variable(tf.random_uniform([n_inputs, n_hidden1], -1.0, 1.0), name="W1")
        hidden1 = tf.nn.relu(tf.matmul(X, W1) + b1)

    with tf.name_scope("hidden2"):
        b2 = tf.Variable(tf.zeros([n_hidden2]), name="b2")
        W2 = tf.Variable(tf.random_uniform([n_hidden1, n_hidden2], -1.0, 1.0), name="W2")
        hidden2 = tf.nn.relu(tf.matmul(hidden1, W2) + b2)

    with tf.name_scope("output"):
        b3 = tf.Variable(tf.zeros([n_outputs]), name="b3")
        W3 = tf.Variable(tf.random_uniform([n_hidden2, n_outputs], -1.0, 1.0), name="W3")
        logits = tf.matmul(hidden2, W3) + b3
        Y_proba = tf.nn.softmax(logits, name="Y_proba")

    with tf.name_scope("train"):
        xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=y)
        loss = tf.reduce_mean(xentropy)
        optimizer = tf.train.AdamOptimizer()
        training_op = optimizer.minimize(loss)

    with tf.name_scope("eval"):
        correct = tf.nn.in_top_k(logits, y, 1)
        accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))

    with tf.name_scope("init_and_save"):
        init = tf.global_variables_initializer()
        saver = tf.train.Saver()


I am working on the file "08_artifical_neural_networks_ex7ex8.ipynb".
The first task is:
8.3)
Can you write Python code for it?

graph = tf.Graph()
with tf.Session(graph=graph) as sess:
    saver = tf.train.import_meta_graph("./my_mnist_model.meta")
    saver.restore(sess, "./my_mnist_model")
    X = graph.get_tensor_by_name("inputs/X:0")
    Y_proba = graph.get_tensor_by_name("output/Y_proba:0")
    Y_proba_val = Y_proba.eval(feed_dict={X: mnist.test.images})

for example_index in range(200):
    y_proba = Y_proba_val[example_index]
    y_pred = np.argmax(y_proba)
    y_label = mnist.test.labels[example_index]
    if y_pred != y_label:
        print("Actual class:{}, Predicted class: {}, Main probabilities: {}".format(
                  y_label,
                  y_pred,
                  ", ".join(["{}:{:.1f}%".format(n, 100*p)
                             for n, p in enumerate(y_proba) if p > 0.01])))
        plt.imshow(mnist.test.images[example_index].reshape([28, 28]), cmap="binary", interpolation="nearest")
        plt.show()


I am working on the file "08_artifical_neural_networks_ex7ex8.ipynb".
The first task is:
Exercise 8

8.1) Take a close look at the following neural network model and make sure you understand every line. Next, add an extra hidden layer composed of 100 neurons.
Can you write Python code for it?

n_inputs = 28 * 28
n_hidden1 = 100
n_outputs = 10

graph = tf.Graph()
with graph.as_default():
    with tf.name_scope("inputs"):
        X = tf.placeholder(tf.float32, shape=[None, n_inputs], name="X")
        y = tf.placeholder(tf.int32, shape=[None], name="y")

    with tf.name_scope("hidden1"):
        b1 = tf.Variable(tf.zeros([n_hidden1]), name="b1")
        W1 = tf.Variable(tf.random_uniform([n_inputs, n_hidden1], -1.0, 1.0), name="W1")
        hidden1 = tf.nn.relu(tf.matmul(X, W1) + b1)

    with tf.name_scope("output"):
        b2 = tf.Variable(tf.zeros([n_outputs]), name="b2")
        W2 = tf.Variable(tf.random_uniform([n_hidden1, n_outputs], -1.0, 1.0), name="W2")
        logits = tf.matmul(hidden1, W2) + b2
        Y_proba = tf.nn.softmax(logits, name="Y_proba")

    with tf.name_scope("train"):
        xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=y)
        loss = tf.reduce_mean(xentropy)
        optimizer = tf.train.AdamOptimizer()
        training_op = optimizer.minimize(loss)

    with tf.name_scope("eval"):
        correct = tf.nn.in_top_k(logits, y, 1)
        accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))

    with tf.name_scope("init_and_save"):
        init = tf.global_variables_initializer()
        saver = tf.train.Saver()


I am working on the file "08_dx_fourier_pricing.ipynb".
The first task is:
Accordingly, the benchmarking for the European call options based on the Fourier-based M76_call_value function.
Can you write Python code for it?

euro_call_jd.present_value()
  # method call needed for initialization

%time valuation_benchmarking(euro_call_jd, dx.M76_call_value)


I am working on the file "08_dx_fourier_pricing.ipynb".
The first task is:
Benchmarking Function
All other valuation benchmarks are generated with Fourier-based pricing functions for which the handling is identical. We therefore use the following function for the benchmarks from now on:
Can you write Python code for it?

def valuation_benchmarking(valuation_object, fourier_function):
    print('%4s  | %7s | %7s | %7s | %7s | %7s' % ('T', 'strike', 'mcs', 'fou', 'dif', 'rel'))
    for maturity in pd.date_range(start=start_date, freq=freq, periods=periods):
        valuation_object.update(maturity=maturity)
        me_option.add_constant('maturity', maturity)
        for strike in np.linspace(start, end, strikes) * initial_value:
            T = (maturity - me_option.pricing_date).days / 365.
            valuation_object.update(strike=strike)
            mcs = valuation_object.present_value()
            me_option.add_constant('strike', strike)
            fou = fourier_function(me_option)
            print('%4.3f | %7.3f | %7.4f | %7.4f | %7.4f | %7.3f '
                % (T, strike, mcs, fou, mcs - fou, (mcs - fou) / fou * 100))


I am working on the file "08_dx_fourier_pricing.ipynb".
The first task is:
Equipped with the single market environments and the valuation environment, we can instantiate the simulation model objects.
Can you write Python code for it?

gbm = dx.geometric_brownian_motion('gbm', me)

jd = dx.jump_diffusion('jd', me)

sv = dx.stochastic_volatility('sv', me)

svjd = dx.stoch_vol_jump_diffusion('svjd', me)


I am working on the file "08_dx_fourier_pricing.ipynb".
The first task is:
Geometric Brownian Motion
We need to initialize the valuation object first.
Can you write Python code for it?

euro_put_gbm.present_value()
  # method call needed for initialization


I am working on the file "08_dx_fourier_pricing.ipynb".
The first task is:
Jump Diffusion
The next model is the jump diffusion as proposed by Merton (1976).
Can you write Python code for it?

euro_put_jd.present_value()
  # method call needed for initialization


I am working on the file "08_dx_fourier_pricing.ipynb".
The first task is:
Plain Vanilla Put and Call Options
Based on the just defined risk factors, we define 8 diffent options---a European put and call option per risk factor, respectively.
Can you write Python code for it?

# market environment for the options
me_option = dx.market_environment('option', dt.datetime(2015, 1, 1))
me_option.add_constant('maturity', dt.datetime(2015, 12, 31))
me_option.add_constant('strike', 100.)
me_option.add_constant('currency', 'EUR')
me_option.add_environment(me)
me_option.add_environment(val_env)

euro_put_gbm = dx.valuation_mcs_european_single('euro_put', gbm, me_option,
                                  'np.maximum(strike - maturity_value, 0)')
euro_call_gbm = dx.valuation_mcs_european_single('euro_call', gbm, me_option,
                                  'np.maximum(maturity_value - strike, 0)')

euro_put_jd = dx.valuation_mcs_european_single('euro_put', jd, me_option,
                                  'np.maximum(strike - maturity_value, 0)')
euro_call_jd = dx.valuation_mcs_european_single('euro_call', jd, me_option,
                                  'np.maximum(maturity_value - strike, 0)')

euro_put_sv = dx.valuation_mcs_european_single('euro_put', sv, me_option,
                                  'np.maximum(strike - maturity_value, 0)')
euro_call_sv = dx.valuation_mcs_european_single('euro_call', sv, me_option,
                                  'np.maximum(maturity_value - strike, 0)')

euro_put_svjd = dx.valuation_mcs_european_single('euro_put', svjd, me_option,
                                  'np.maximum(strike - maturity_value, 0)')
euro_call_svjd = dx.valuation_mcs_european_single('euro_call', svjd, me_option,
                                  'np.maximum(maturity_value - strike, 0)')


I am working on the file "08_dx_fourier_pricing.ipynb".
The first task is:
Risk Factors
The examples and benchmarks to follow rely on four different models:

geometric Brownian motion (Black-Scholes-Merton 1973)
jump diffusion (Merton 1976)
stochastic volatility (Heston 1993)
stochastic volatility jump diffusion (Bates 1996)

For details on these models and the Fourier-based option pricing approach refer to Hilpisch (2015) (cf. http://eu.wiley.com/WileyCDA/WileyTitle/productCd-1119037999.html).
We first define the single market and valuation environments.
Can you write Python code for it?

# constant short rate
r = dx.constant_short_rate('r', 0.01)

# geometric Brownian motion
me = dx.market_environment('me', dt.datetime(2015, 1, 1))
me.add_constant('initial_value', 100.)
me.add_constant('volatility', 0.2)
me.add_constant('final_date', dt.datetime(2015, 12, 31))
me.add_constant('currency', 'EUR')

# jump component
me.add_constant('lambda', 0.4)
me.add_constant('mu', -0.6)
me.add_constant('delta', 0.2)

# stochastic volatiltiy component
me.add_constant('rho', -.5)
me.add_constant('kappa', 5.0)
me.add_constant('theta', 0.02)
me.add_constant('vol_vol', 0.3)

# valuation environment
val_env = dx.market_environment('val_env', dt.datetime(2015, 1, 1))
val_env.add_constant('paths', 55000)
    # 25,000 paths
val_env.add_constant('frequency', 'D')
    # weekly frequency
val_env.add_curve('discount_curve', r)
val_env.add_constant('starting_date', dt.datetime(2015, 1, 1))
val_env.add_constant('final_date', dt.datetime(2015, 12, 31))

# add valuation environment to market environment
me.add_environment(val_env)


I am working on the file "08_dx_fourier_pricing.ipynb".
The first task is:
Second, the benchmarking for the European call options based on the Fourier-based H93_call_value function.
Can you write Python code for it?

euro_call_sv.present_value()
  # method call needed for initialization

%time valuation_benchmarking(euro_call_sv, dx.H93_call_value)


I am working on the file "08_dx_fourier_pricing.ipynb".
The first task is:
Stochastic Volatility
Stochastic volatility models like the one of Heston (1993) are popular to reproduce implied volatility smiles observed in markets. First, the benchmarking for the European put options using the Fourier-based H93_put_value function.
Can you write Python code for it?

euro_put_sv.present_value()
  # method call needed for initialization

%time valuation_benchmarking(euro_put_sv, dx.H93_put_value)


I am working on the file "08_dx_fourier_pricing.ipynb".
The first task is:
Stochastic Volatility Jump-Diffusion
Finally, we consider the combination of the stochastic volatility and jump diffusion models from before as proposed by Bates (1996). The Fourier-based pricing function for European put options is called B96_put_value.
Can you write Python code for it?

euro_put_svjd.present_value()
  # method call needed for initialization

%time valuation_benchmarking(euro_put_svjd, dx.B96_put_value)


I am working on the file "08_dx_fourier_pricing.ipynb".
The first task is:
The Fourier-based counterpart function for European call options is called B96_call_value.
Can you write Python code for it?

euro_call_svjd.present_value()
  # method call needed for initialization

%time valuation_benchmarking(euro_call_svjd, dx.B96_call_value)


I am working on the file "08_dx_fourier_pricing.ipynb".
The first task is:
The following routine benchmarks the Monte Carlo value estimates for the European put option against the output from the valuation object based on the analytical pricing formula. The results are quite good since this model is quite easy to discretize exactly and therefore generally shows good convergence of the Monte Carlo estimates.
Can you write Python code for it?

%%time
# European put
print('%4s  | %7s | %7s | %7s | %7s | %7s' % ('T', 'strike', 'mcs', 'fou', 'dif', 'rel'))
for maturity in pd.date_range(start=start_date, freq=freq, periods=periods):
    bsm_option.maturity = maturity
    euro_put_gbm.update(maturity=maturity)
    for strike in np.linspace(start, end, strikes) * initial_value:
        T = (maturity - me_option.pricing_date).days / 365.
        euro_put_gbm.update(strike=strike)
        mcs = euro_put_gbm.present_value()
        bsm_option.strike = strike
        ana = bsm_option.put_value()
        print('%4.3f | %7.3f | %7.4f | %7.4f | %7.4f | %7.2f '
                % (T, strike, mcs, ana, mcs - ana, (mcs - ana) / ana * 100))


I am working on the file "08_dx_fourier_pricing.ipynb".
The first task is:
The same now for the European call option.
Can you write Python code for it?

euro_call_gbm.present_value()
  # method call needed for initialization

%%time
# European calls
print '%4s  | %7s | %7s | %7s | %7s | %7s' % ('T', 'strike', 'mcs', 'fou', 'dif', 'rel')
for maturity in pd.date_range(start=start_date, freq=freq, periods=periods):
    euro_call_gbm.update(maturity=maturity)
    for strike in np.linspace(start, end, strikes) * initial_value:
        T = (maturity - me_option.pricing_date).days / 365.
        euro_call_gbm.update(strike=strike)
        mcs = euro_call_gbm.present_value()
        bsm_option.strike = strike
        bsm_option.maturity = maturity
        ana = bsm_option.call_value()
        print '%4.3f | %7.3f | %7.4f | %7.4f | %7.4f | %7.2f ' \
                % (T, strike, mcs, ana, mcs - ana, (mcs - ana) / ana * 100)


euro_call_gbm.present_value()
  # method call needed for initialization

%%time
# European calls
print('%4s  | %7s | %7s | %7s | %7s | %7s' % ('T', 'strike', 'mcs', 'fou', 'dif', 'rel'))
for maturity in pd.date_range(start=start_date, freq=freq, periods=periods):
    euro_call_gbm.update(maturity=maturity)
    for strike in np.linspace(start, end, strikes) * initial_value:
        T = (maturity - me_option.pricing_date).days / 365.
        euro_call_gbm.update(strike=strike)
        mcs = euro_call_gbm.present_value()
        bsm_option.strike = strike
        bsm_option.maturity = maturity
        ana = bsm_option.call_value()
        print('%4.3f | %7.3f | %7.4f | %7.4f | %7.4f | %7.2f ' \
                % (T, strike, mcs, ana, mcs - ana, (mcs - ana) / ana * 100))


I am working on the file "08_dx_fourier_pricing.ipynb".
The first task is:
There is a Fourier-based pricing function available which is called M76_put_value and which is used for the benchmarking for the European put options that follows.
Can you write Python code for it?


I am working on the file "08_dx_fourier_pricing.ipynb".
The first task is:
There is a valuation class for European put and call options in the Black-Scholes-Merton model available called BSM_european_option. It is based on the analytical pricing formula for that model and is instantiated as follows:
Can you write Python code for it?

bsm_option = dx.BSM_european_option('bsm_opt', me_option)


I am working on the file "08_dx_fourier_pricing.ipynb".
The first task is:
Valuation Benchmarking
In this sub-section, we benchmark the Monte Carlo value estimates against the Fourier-based pricing results.
Can you write Python code for it?


I am working on the file "08_dx_fourier_pricing.ipynb".
The first task is:
We first define some parameters used throughout.
Can you write Python code for it?

freq = '2m'  # used for maturity definitions
periods = 3  # number of intervals for maturity grid
strikes = 5  # number of strikes per maturity
initial_value = 100  # initial value for all risk factors
start = 0.8  # lowest strike in percent of spot
end = 1.2  # highest strike in percent of spot
start_date = '2015/3/1'  # start date for simulation/pricing


I am working on the file "08_finite_volumes/fv_advection_1d.ipynb".
The first task is:
2. Finite Volumes setup
Can you write Python code for it?

# Initialize Space
x  = np.linspace(0,xmax,nx)
dx = min(np.diff(x))

# use wave based CFL criterion
dt = eps*dx/c    # calculate tim step from stability criterion

# Simulation time
nt = int(np.floor(tmax/dt))

# Initialize shape of fields
Q   = np.zeros(nx)
dQ  = np.zeros(nx)
dQ1 = np.zeros(nx)
dQ2 = np.zeros(nx)
Qa  = np.zeros(nx)


I am working on the file "08_finite_volumes/fv_advection_1d.ipynb".
The first task is:
4. Solution for the scalar advection problem
Can you write Python code for it?


I am working on the file "08_finite_volumes/fv_hetero_1d.ipynb".
The first task is:
1. Initialization of setup
Can you write Python code for it?

# Initialization of setup
# --------------------------------------------------------------------------
nx    = 800         # number of grid points
c0    = 2500        # acoustic velocity in m/s
rho    = 2500       # density in kg/m^3
Z0    = rho*c0      # impedance
mu    = rho*c0**2   # shear modulus
rho0  = rho         # density
mu0   = mu          # shear modulus

xmax  = 10000       # in m
eps   = 0.5         # CFL
tmax  = 1.5         # simulation time in s
isnap = 10          # plotting rate
sig   = 200         # argument in the inital condition
x0    = 2500        # position of the initial condition


I am working on the file "08_finite_volumes/fv_hetero_1d.ipynb".
The first task is:
3. Finite Volumes setup
Can you write Python code for it?

# Finite Volumes setup
# --------------------------------------------------------------------------
A = np.zeros((2,2,nx))
Z = np.zeros((1,nx))
c = np.zeros((1,nx))

# Initialize velocity
c = c + c0
c[int(nx/2):nx] = c[int(nx/2):nx]*2
Z = rho*c

# Initialize A for each cell
for i in range(1,nx):
    A0 = np.array([[0, -mu], [-1/rho, 0]])
    if i > nx/2:
        A0= np.array([[0, -4*mu], [-1/rho, 0]])
    A[:,:,i] = A0

# Initialize Space
x, dx = np.linspace(0,xmax,nx,retstep=True)

# use wave based CFL criterion
dt = eps*dx/np.max(c) # calculate tim step from stability criterion

# Simulation time
nt = int(np.floor(tmax/dt))

# Initialize wave fields
Q = np.zeros((2,nx))
Qnew = np.zeros((2,nx))


I am working on the file "08_finite_volumes/fv_hetero_1d.ipynb".
The first task is:
4. Initial condition
Can you write Python code for it?


I am working on the file "08_finite_volumes/fv_hetero_1d.ipynb".
The first task is:
4. Solution for the inhomogeneous problem
Upwind finite volume scheme
We decompose the solution into right propagating $\mathbf{\Lambda}_i^{+}$ and left propagating eigenvalues $\mathbf{\Lambda}_i^{-}$ where
\begin{equation}
\mathbf{\Lambda}_i^{+}=
     \begin{pmatrix}
       -c_i & 0 \
        0 & 0
     \end{pmatrix}
\qquad\text{,}\qquad
\mathbf{\Lambda}_i^{-}=
     \begin{pmatrix}
        0 & 0 \
        0 & c_i
     \end{pmatrix}
\qquad\text{and}\qquad
\mathbf{A}_i^{\pm} = \mathbf{R}^{-1}\mathbf{\Lambda}_i^{\pm}\mathbf{R}
\end{equation}
This strategy allows us to formulate an upwind finite volume scheme for any hyperbolic system as
\begin{equation}
\mathbf{Q}{i}^{n+1}  = \mathbf{Q}{i}^{n} - \frac{dt}{dx}(\mathbf{A}i^{+}\Delta\mathbf{Q}{l} - \mathbf{A}i^{-}\Delta\mathbf{Q}{r})
\end{equation}
with corresponding flux term given by
\begin{equation}
\mathbf{F}{l} =  \mathbf{A}_i^{+}\Delta\mathbf{Q}{l}
\qquad\text{,}\qquad
\mathbf{F}{r}  =  \mathbf{A}_i^{-}\Delta\mathbf{Q}{r}
\end{equation}
Lax-Wendroff finite volume scheme
The upwind solution presents a strong diffusive behavior. In this sense, the Lax-Wendroff perform better, with the advantage that it is not needed to decompose the eigenvalues into right and left propagations. Here the matrix $\mathbf{A}_i$ can be used in its original form. The Lax-Wendroff follows
\begin{equation}
\mathbf{Q}{i}^{n+1}  = \mathbf{Q}{i}^{n} - \frac{dt}{2dx}\mathbf{A}i(\Delta\mathbf{Q}{l} + \Delta\mathbf{Q}{r}) + \frac{1}{2}\Big(\frac{dt}{dx}\Big)^2\mathbf{A}_i^2(\Delta\mathbf{Q}{l} - \Delta\mathbf{Q}_{r})
\end{equation}
Can you write Python code for it?

# Initialize animated plot
# ---------------------------------------------------------------
fig = plt.figure(figsize=(10,6))
ax1 = fig.add_subplot(2,1,1)
ax2 = fig.add_subplot(2,1,2)
ax1.axvspan(((nx-1)/2+1)*dx, nx*dx, alpha=0.2, facecolor='b')
ax2.axvspan(((nx-1)/2+1)*dx, nx*dx, alpha=0.2, facecolor='b')
ax1.set_xlim([0, xmax])
ax2.set_xlim([0, xmax])
ax1.set_ylabel('Stress')
ax2.set_ylabel('Velocity')
ax2.set_xlabel(' x ')

line1 = ax1.plot(x, Q[0,:], 'k', x, s, 'r--')
line2 = ax2.plot(x, Q[1,:], 'k', x, v, 'r--')

plt.suptitle('Heterogeneous F. volume - Lax-Wendroff method', size=16)
ax1.text(0.1*xmax, 0.8*max(sx), '$\mu$ = $\mu_{o}$')
ax1.text(0.8*xmax, 0.8*max(sx), '$\mu$ = $4\mu_{o}$')

plt.ion()    # set interective mode
plt.show()

# ---------------------------------------------------------------
# Time extrapolation
# ---------------------------------------------------------------
for j in range(nt):
    # Finite Volume Extrapolation scheme-------------------------
    for i in range(1,nx-1):
        # Lax-Wendroff method
        dQl = Q[:,i] - Q[:,i-1]
        dQr = Q[:,i+1] - Q[:,i]
        Qnew[:,i] = Q[:,i] - dt/(2*dx)*A[:,:,i] @ (dQl + dQr)\
                + 1/2*(dt/dx)**2 *A[:,:,i] @ A[:,:,i] @ (dQr - dQl)
    # Absorbing boundary conditions
    Qnew[:,0] = Qnew[:,1]
    Qnew[:,nx-1] = Qnew[:,nx-2]
    Q, Qnew = Qnew, Q

    # Finite Difference Extrapolation scheme---------------------
    # Stress derivative
    for i in range(1, nx-1):
        ds[i] = (s[i+1] - s[i])/dx
    # Velocity extrapolation
    v = v + dt*ds/rhofd
    # Velocity derivative
    for i in range(1, nx-1):
        dv[i] = (v[i] - v[i-1])/dx
    # Stress extrapolation
    s = s + dt*mufd*dv

    # --------------------------------------
    # Animation plot. Display solutions
    if not j % isnap:
        for l in line1:
            l.remove()
            del l
        for l in line2:
            l.remove()
            del l

        line1 = ax1.plot(x, Q[0,:], 'k', x, s, 'r--')
       ...


I am working on the file "08_finite_volumes/fv_homo_1d.ipynb".
The first task is:
1. Initialization of setup
Can you write Python code for it?

# Initialization of setup
# --------------------------------------------------------------------------
nx    = 800        # number of grid points
c     = 2500       # acoustic velocity in m/s
ro    = 2500       # density in kg/m^3
Z     = ro*c       # impedance
mu    = ro*c**2    # shear modulus

xmax  = 10000      # Length in m
eps   = 0.5        # CFL
tmax  = 2.0        # simulation time in s
isnap = 10         # plotting rate
sig   = 200        # argument in the inital condition
x0    = 5000       # position of the initial condition

imethod = 'upwind'  # 'Lax-Wendroff', 'upwind'

# Initialize Space
x, dx = np.linspace(0,xmax,nx,retstep=True)

# use wave based CFL criterion
dt = eps*dx/c    # calculate time step from stability criterion

# Simulation time
nt = int(np.floor(tmax/dt))

# Initialize wave fields
Q = np.zeros((2,nx))
Qnew = np.zeros((2,nx))
Qa = np.zeros((2,nx))


I am working on the file "08_finite_volumes/fv_homo_1d.ipynb".
The first task is:
2. Initial condition
Seismic disturbances are introduced through specification of a particular spatial initial condition, in this case we use a Gaussian function.
Exercise 1
Implement the spatial initial condition given by:
\begin{equation}
Q(x,t=0)  = e^{-1/\sigma^2 (x - x_{o})^2}
\end{equation}
Then, visualize the initial condition in a given plot.
Can you write Python code for it?


I am working on the file "08_finite_volumes/fv_homo_1d.ipynb".
The first task is:
3. Solution for the homogeneous problem
Upwind finite volume scheme
We decompose the solution into right propagating $\mathbf{\Lambda}^{+}$ and left propagating eigenvalues $\mathbf{\Lambda}^{-}$ where
\begin{equation}
\mathbf{\Lambda}^{+}=
     \begin{pmatrix}
       -c & 0 \
        0 & 0
     \end{pmatrix}
\qquad\text{,}\qquad
\mathbf{\Lambda}^{-}=
     \begin{pmatrix}
        0 & 0 \
        0 & c
     \end{pmatrix}
\qquad\text{and}\qquad
\mathbf{A}^{\pm} = \mathbf{R}^{-1}\mathbf{\Lambda}^{\pm}\mathbf{R}
\end{equation}
This strategy allows us to formulate an upwind finite volume scheme for any hyperbolic system as
\begin{equation}
\mathbf{Q}{i}^{n+1}  = \mathbf{Q}{i}^{n} - \frac{dt}{dx}(\mathbf{A}^{+}\Delta\mathbf{Q}{l} - \mathbf{A}^{-}\Delta\mathbf{Q}{r})
\end{equation}
with corresponding flux term given by
\begin{equation}
\mathbf{F}{l} =  \mathbf{A}^{+}\Delta\mathbf{Q}{l}
\qquad\text{,}\qquad
\mathbf{F}{r}  =  \mathbf{A}^{-}\Delta\mathbf{Q}{r}
\end{equation}
Lax-Wendroff finite volume scheme
The upwind solution presents a strong diffusive behaviour. In this sense, the Lax-Wendroff perform better, with the advantage that it is not needed to decompose the eigenvalues into right and left propagations. Here the matrix $\mathbf{A}$ can be used in its original form. The Lax-Wendroff follows
\begin{equation}
\mathbf{Q}{i}^{n+1}  = \mathbf{Q}{i}^{n} - \frac{dt}{2dx}\mathbf{A}(\mathbf{Q}{i+1}^{n} - \mathbf{Q}{i-1}^{n}) + \frac{1}{2}\Big(\frac{dt}{dx}\Big)^2\mathbf{A}^2(\mathbf{Q}{i+1}^{n} - 2\mathbf{Q}{i}^{n} + \mathbf{Q}_{i-1}^{n})
\end{equation}
Exercise 2
Initialize all relevant matrices, i.e $R$, $R^{-1}$, $\mathbf{\Lambda}^{+}$, $\mathbf{\Lambda}^{-}$, $\mathbf{A}^{+}$, $\mathbf{A}^{-}$, $\mathbf{A}$.
Can you write Python code for it?


I am working on the file "08_finite_volumes/fv_homo_1d.ipynb".
The first task is:
4. Finite Volumes solution
Can you write Python code for it?


I am working on the file "08_finite_volumes/fv_homo_1d_solution.ipynb".
The first task is:
1. Initialization of setup
Can you write Python code for it?


I am working on the file "08_finite_volumes/fv_homo_1d_solution.ipynb".
The first task is:
2. Initial condition
Seismic disturbances are introduced through specification of a particular spatial initial condition, in this case we use a Gaussian function.
Exercise 1
Implement the spatial initial condition given by:
\begin{equation}
Q(x,t=0)  = e^{-1/\sigma^2 (x - x_{o})^2}
\end{equation}
Then, visualize the initial condition in a given plot.
Can you write Python code for it?


I am working on the file "08_finite_volumes/fv_homo_1d_solution.ipynb".
The first task is:
4. Finite Volumes solution
Can you write Python code for it?


I am working on the file "08_linear_regression.ipynb".
The first task is:
Confidence in our Model
Question: Is linear regression a high bias/low variance model, or a low bias/high variance model?
Answer: High bias/low variance. Under repeated sampling, the line will stay roughly in the same place (low variance), but the average of those models won't do a great job capturing the true relationship (high bias). Note that low variance is a useful characteristic when you don't have a lot of training data!
A closely related concept is confidence intervals. Statsmodels calculates 95% confidence intervals for our model coefficients, which are interpreted as follows: If the population from which this sample was drawn was sampled 100 times, approximately 95 of those confidence intervals would contain the "true" coefficient.
Can you write Python code for it?

# print the confidence intervals for the model coefficients
lm.conf_int()


I am working on the file "08_linear_regression.ipynb".
The first task is:
Example: Advertising Data
Let's take a look at some data, ask some questions about that data, and then use linear regression to answer those questions!
Can you write Python code for it?

# read data into a DataFrame
data = pd.read_csv('http://www-bcf.usc.edu/~gareth/ISL/Advertising.csv', index_col=0)
data.head()


I am working on the file "08_linear_regression.ipynb".
The first task is:
Here is how we interpret the coding:
- rural is coded as Area_suburban=0 and Area_urban=0
- suburban is coded as Area_suburban=1 and Area_urban=0
- urban is coded as Area_suburban=0 and Area_urban=1
Why do we only need two dummy variables, not three? Because two dummies captures all of the information about the Area feature, and implicitly defines rural as the baseline level. (In general, if you have a categorical feature with k levels, you create k-1 dummy variables.)
If this is confusing, think about why we only needed one dummy variable for Size (IsLarge), not two dummy variables (IsSmall and IsLarge).
Let's include the two new dummy variables in the model:
Can you write Python code for it?

# create X and y
feature_cols = ['TV', 'Radio', 'Newspaper', 'IsLarge', 'Area_suburban', 'Area_urban']
X = data[feature_cols]
y = data.Sales

# instantiate, fit
lm = LinearRegression()
lm.fit(X, y)

# print coefficients
zip(feature_cols, lm.coef_)


I am working on the file "08_linear_regression.ipynb".
The first task is:
How do we interpret the IsLarge coefficient? For a given amount of TV/Radio/Newspaper ad spending, being a large market is associated with an average increase in Sales of 57.42 widgets (as compared to a Small market, which is called the baseline level).
What if we had reversed the 0/1 coding and created the feature 'IsSmall' instead? The coefficient would be the same, except it would be negative instead of positive. As such, your choice of category for the baseline does not matter, all that changes is your interpretation of the coefficient.
Handling Categorical Predictors with More than Two Categories
Let's create a new feature called Area, and randomly assign observations to be rural, suburban, or urban:
Can you write Python code for it?

# set a seed for reproducibility
np.random.seed(123456)

# assign roughly one third of observations to each group
nums = np.random.rand(len(data))
mask_suburban = (nums > 0.33) & (nums < 0.66)
mask_urban = nums > 0.66
data['Area'] = 'rural'
data.loc[mask_suburban, 'Area'] = 'suburban'
data.loc[mask_urban, 'Area'] = 'urban'
data.head()


I am working on the file "08_linear_regression.ipynb".
The first task is:
How do we interpret these coefficients? For a given amount of Radio and Newspaper ad spending, an increase of $1000 in TV ad spending is associated with an increase in Sales of 45.765 widgets.
A lot of the information we have been reviewing piece-by-piece is available in the model summary output:
Can you write Python code for it?

# print a summary of the fitted model
lm.summary()


I am working on the file "08_linear_regression.ipynb".
The first task is:
Interpreting Model Coefficients
How do we interpret the TV coefficient ($\beta_1$)?
- A "unit" increase in TV ad spending is associated with a 0.047537 "unit" increase in Sales.
- Or more clearly: An additional $1,000 spent on TV ads is associated with an increase in sales of 47.537 widgets.
Note that if an increase in TV ad spending was associated with a decrease in sales, $\beta_1$ would be negative.
Using the Model for Prediction
Let's say that there was a new market where the TV advertising spend was $50,000. What would we predict for the Sales in that market?
$$y = \beta_0 + \beta_1x$$
$$y = 7.032594 + 0.047537 \times 50$$
Can you write Python code for it?

# manually calculate the prediction
7.032594 + 0.047537*50


I am working on the file "08_linear_regression.ipynb".
The first task is:
Is that a "good" R-squared value? It's hard to say. The threshold for a good R-squared value depends widely on the domain. Therefore, it's most useful as a tool for comparing different models.
Multiple Linear Regression
Simple linear regression can easily be extended to include multiple features. This is called multiple linear regression:
$y = \beta_0 + \beta_1x_1 + ... + \beta_nx_n$
Each $x$ represents a different feature, and each feature has its own coefficient. In this case:
$y = \beta_0 + \beta_1 \times TV + \beta_2 \times Radio + \beta_3 \times Newspaper$
Let's use Statsmodels to estimate these coefficients:
Can you write Python code for it?

# create a fitted model with all three features
lm = smf.ols(formula='Sales ~ TV + Radio + Newspaper', data=data).fit()

# print the coefficients
lm.params


I am working on the file "08_linear_regression.ipynb".
The first task is:
Keep in mind that we only have a single sample of data, and not the entire population of data. The "true" coefficient is either within this interval or it isn't, but there's no way to actually know. We estimate the coefficient with the data we do have, and we show uncertainty about that estimate by giving a range that the coefficient is probably within.
Note that using 95% confidence intervals is just a convention. You can create 90% confidence intervals (which will be more narrow), 99% confidence intervals (which will be wider), or whatever intervals you like.
Hypothesis Testing and p-values
Closely related to confidence intervals is hypothesis testing. Generally speaking, you start with a null hypothesis and an alternative hypothesis (that is opposite the null). Then, you check whether the data supports rejecting the null hypothesis or failing to reject the null hypothesis.
(Note that "failing to reject" the null is not the same as "accepting" the null hypothesis. The alternative hypothesis may indeed be true, except that you just don't have enough data to show that.)
As it relates to model coefficients, here is the conventional hypothesis test:
- null hypothesis: There is no relationship between TV ads and Sales (and thus $\beta_1$ equals zero)
- alternative hypothesis: There is a relationship between TV ads and Sales (and thus $\beta_1$ is not equal to zero)
How do we test this hypothesis? Intuitively, we reject the null (and thus believe the alternative) if the 95% confidence interval does not include zero. Conversely, the p-value represents the probability that the coefficient is actually zero:
Can you write Python code for it?

# print the p-values for the model coefficients
lm.pvalues


I am working on the file "08_linear_regression.ipynb".
The first task is:
Note that p-values and confidence intervals are not (easily) accessible through scikit-learn.
Handling Categorical Predictors with Two Categories
Up to now, all of our predictors have been numeric. What if one of our predictors was categorical?
Let's create a new feature called Size, and randomly assign observations to be small or large:
Can you write Python code for it?

import numpy as np

# set a seed for reproducibility
np.random.seed(12345)

# create a Series of booleans in which roughly half are True
nums = np.random.rand(len(data))
mask_large = nums > 0.5

# initially set Size to small, then change roughly half to be large
data['Size'] = 'small'
data.loc[mask_large, 'Size'] = 'large'
data.head()


I am working on the file "08_linear_regression.ipynb".
The first task is:
Questions About the Advertising Data
Let's pretend you work for the company that manufactures and markets this widget. The company might ask you the following: On the basis of this data, how should we spend our advertising money in the future?
This general question might lead you to more specific questions:
1. Is there a relationship between ads and sales?
2. How strong is that relationship?
3. Which ad types contribute to sales?
4. What is the effect of each ad type of sales?
5. Given ad spending in a particular market, can sales be predicted?
We will explore these questions below!
Simple Linear Regression
Simple linear regression is an approach for predicting a quantitative response using a single feature (or "predictor" or "input variable"). It takes the following form:
$y = \beta_0 + \beta_1x$
What does each term represent?
- $y$ is the response
- $x$ is the feature
- $\beta_0$ is the intercept
- $\beta_1$ is the coefficient for x
Together, $\beta_0$ and $\beta_1$ are called the model coefficients. To create your model, you must "learn" the values of these coefficients. And once we've learned these coefficients, we can use the model to predict Sales!
Estimating ("Learning") Model Coefficients
Generally speaking, coefficients are estimated using the least squares criterion, which means we are find the line (mathematically) which minimizes the sum of squared residuals (or "sum of squared errors"):

What elements are present in the diagram?
- The black dots are the observed values of x and y.
- The blue line is our least squares line.
- The red lines are the residuals, which are the distances between the observed values and the least squares line.
How do the model coefficients relate to the least squares line?
- $\beta_0$ is the intercept (the value of $y$ when $x$=0)
- $\beta_1$ is the slope (the change in $y$ divided by change in $x$)
Here is a graphical depiction of those calculations:

Let's use Statsmodels to estimate the model...
Can you write Python code for it?

# this is the standard import if you're using "formula notation" (similar to R)
import statsmodels.formula.api as smf

# create a fitted model in one line
lm = smf.ols(formula='Sales ~ TV', data=data).fit()

# print the coefficients
lm.params


I am working on the file "08_linear_regression.ipynb".
The first task is:
R-squared will always increase as you add more features to the model, even if they are unrelated to the response. Thus, selecting the model with the highest R-squared is not a reliable approach for choosing the best linear model.
There is alternative to R-squared called adjusted R-squared that penalizes model complexity (to control for overfitting), but it generally under-penalizes complexity.
So is there a better approach to feature selection? Cross-validation. It provides a more reliable estimate of out-of-sample error, and thus is a better way to choose which of your models will best generalize to out-of-sample data. There is extensive functionality for cross-validation in scikit-learn, including automated methods for searching different sets of parameters and different models. Importantly, cross-validation can be applied to any model, whereas the methods described above only apply to linear models.
Linear Regression in scikit-learn
Let's redo some of the Statsmodels code above in scikit-learn:
Can you write Python code for it?

# create X and y
feature_cols = ['TV', 'Radio', 'Newspaper']
X = data[feature_cols]
y = data.Sales

# follow the usual sklearn pattern: import, instantiate, fit
from sklearn.linear_model import LinearRegression
lm = LinearRegression()
lm.fit(X, y)

# print intercept and coefficients
print lm.intercept_
print lm.coef_

# pair the feature names with the coefficients
zip(feature_cols, lm.coef_)

# predict for a new observation
lm.predict([100, 25, 25])

# calculate the R-squared
lm.score(X, y)


I am working on the file "08_linear_regression.ipynb".
The first task is:
There are 200 observations, and thus 200 markets in the dataset.
Can you write Python code for it?

# visualize the relationship between the features and the response using scatterplots
fig, axs = plt.subplots(1, 3, sharey=True)
data.plot(kind='scatter', x='TV', y='Sales', ax=axs[0], figsize=(16, 8))
data.plot(kind='scatter', x='Radio', y='Sales', ax=axs[1])
data.plot(kind='scatter', x='Newspaper', y='Sales', ax=axs[2])


I am working on the file "08_linear_regression.ipynb".
The first task is:
Thus, we would predict Sales of 9,409 widgets in that market.
Of course, we can also use Statsmodels to make the prediction:
Can you write Python code for it?

# you have to create a DataFrame since the Statsmodels formula interface expects it
X_new = pd.DataFrame({'TV': [50]})
X_new.head()

# use the model to make predictions on a new value
lm.predict(X_new)


I am working on the file "08_linear_regression.ipynb".
The first task is:
What are the features?
- TV: advertising dollars spent on TV for a single product in a given market (in thousands of dollars)
- Radio: advertising dollars spent on Radio
- Newspaper: advertising dollars spent on Newspaper
What is the response?
- Sales: sales of a single product in a given market (in thousands of widgets)
Can you write Python code for it?

# print the shape of the DataFrame
data.shape


I am working on the file "08_parallelism/08_parallelism.ipynb".
The first task is:
And also perform a time benchmark with timeit:
Can you write Python code for it?

%%timeit
jobs = []
for s_index in s_indexes:
    jobs.append( multiprocessing.Process( target=numba_f, args=(x, y, s_index, chunk_size, r) ) )
for job in jobs:
    job.start()
for job in jobs:
    job.join()


I am working on the file "08_parallelism/08_parallelism.ipynb".
The first task is:
As you can see the performance is not as good as with the Threading library, but anyway there is an improvement to the naive/sequential version.
Can you write Python code for it?

%%memit
jobs = []
for s_index in s_indexes:
    jobs.append( multiprocessing.Process( target=numba_f, args=(x, y, s_index, chunk_size, r) ) )
for job in jobs:
    job.start()
for job in jobs:
    job.join()


I am working on the file "08_parallelism/08_parallelism.ipynb".
The first task is:
By default, the %px magic executes commands in blocking mode; the cell only returns when the commands have completed on all engines. It is possible to run non-blocking commands with the --noblock or -a option. In this case, the cell returns immediately, and the task's status and results can be polled asynchronously from IPython's interactive session:
Can you write Python code for it?

%%px -a
import time
time.sleep(5)

print(_.elapsed, _.ready())


I am working on the file "08_parallelism/08_parallelism.ipynb".
The first task is:
Finally we create the threads, launch its execution, and wait for each one to finish its computation:
Can you write Python code for it?

# threads creation
threads = []
for s_index in s_indexes:
    threads.append( threading.Thread( target=numba_f, args=(x, y, s_index, chunk_size, r) ) )
# threads start execution
for thread in threads:
    thread.start()
# we wait for each thread to finish its execution
for thread in threads:
    thread.join()
# all threads have finished here


I am working on the file "08_parallelism/08_parallelism.ipynb".
The first task is:
Here is an example:
Can you write Python code for it?


I am working on the file "08_parallelism/08_parallelism.ipynb".
The first task is:
Next we configure the number of threads to use, the chunk size of each thread and the starting index of each thread:
Can you write Python code for it?

# number of threads
T = 4

# data size for each thread
chunk_size = N // T

# starting index for each thread
s_indexes = [i*chunk_size for i in range(T)]


I am working on the file "08_parallelism/08_parallelism.ipynb".
The first task is:
Now we can benchmark the execution of both version with timeit magic:
Can you write Python code for it?

%%timeit
threads = []
for s_index in s_indexes:
    threads.append( threading.Thread( target=numba_f, args=(x, y, s_index, chunk_size, r) ) )
for thread in threads:
    thread.start()
for thread in threads:
    thread.join()

%%memit
threads = []
for s_index in s_indexes:
    threads.append( threading.Thread( target=numba_f, args=(x, y, s_index, chunk_size, r) ) )
for thread in threads:
    thread.start()
for thread in threads:
    thread.join()

%%timeit
numba_f(x, y, 0, N, r)


I am working on the file "08_parallelism/08_parallelism.ipynb".
The first task is:
Now we create the huge unidimensional arrays to be passed to this function:
Can you write Python code for it?

# data size, 100 million items
N = int(100e6)

# data
x = np.random.random(N)
y = np.random.random(N)

# array for results
r = np.zeros(N)


I am working on the file "08_parallelism/08_parallelism.ipynb".
The first task is:
The %pxresult blocks until the task finishes:
Can you write Python code for it?


I am working on the file "08_parallelism/08_parallelism.ipynb".
The first task is:
There are several steps to distribute code across multiple cores:
1. Launching several IPython engines (there is typically one process per core).
2. Creating a Client that acts as a proxy to these engines.
3. Using the client to launch tasks on the engines and retrieve the results.
Engines are Python processes that execute code on different computing units. Once the engines have been launched, we create a Client instance that will give us access to these engines:
Can you write Python code for it?


I am working on the file "08_parallelism/08_parallelism.ipynb".
The first task is:
There are several ways to run code in parallel on the engines. First, we can use the %px
 magic command.
The code passed to the %px magic command is executed on all engines. Here, we display the OS process identifier (also called PID) of every engine. Every engine is an independent Python process.
Can you write Python code for it?

%px import os, time

%px print(os.getpid())


I am working on the file "08_parallelism/08_parallelism.ipynb".
The first task is:
There are two ways to access the engines:

With the direct interface, we have a direct access to every engine.
With the load-balanced interface, we submit jobs to a scheduler which dynamically assigns them to the engines depending on their current load.

Direct Interface
The ids attribute of the client shows us the identifiers of the engines that were automatically detected by IPython:
Can you write Python code for it?


I am working on the file "08_parallelism/08_parallelism.ipynb".
The first task is:
There's also a synchronous version of map() called map_sync() which blocks until the tasks have completed, and directly returns the results.
This function doesn't return the results. Instead, it launches the 100 tasks in parallel and returns an AsyncResult object. The AsyncResult object can be used to interactively poll the tasks status and eventually retrieve the results. This object has a metadata attribute: a list of dictionaries for all engines with useful information.
Can you write Python code for it?

print(len(ar.metadata))
ar.metadata[0]

ar.ready(), ar.progress


I am working on the file "08_parallelism/08_parallelism.ipynb".
The first task is:
We can also specify the exact list of engines to run code on. The --targets option accepts a list of engine identifiers.
Note that we used the cell magic %%px this time instead of the line magic. The cell magic allows us to execute several lines of code on the engines.
Can you write Python code for it?


I am working on the file "08_parallelism/08_parallelism.ipynb".
The first task is:
We use a parallel map() function to run these tasks in parallel. Our sample() function is called 100 times, taking n // 100 as its argument every time. We will combine the 100 results later.
Can you write Python code for it?


I am working on the file "08_parallelism/08_parallelism.ipynb".
The first task is:
We will now run this simulation in parallel. First, we divide this task into 100 smaller subtasks where the number of points is divided by 100
Can you write Python code for it?


I am working on the file "08_parallelism/08_parallelism.ipynb".
The first task is:
We will use the same example used above:
Can you write Python code for it?

# creation of processes
jobs = []
for s_index in s_indexes:
    jobs.append( multiprocessing.Process( target=numba_f, args=(x, y, s_index, chunk_size, r) ) )
# start execution of processes
for job in jobs:
    job.start()
# we wait for each process to finish its execution
for job in jobs:
    job.join()
# all processes have finished here


I am working on the file "08_simhash_prob_search/simhash_prob.ipynb".
The first task is:
Вероятностный поиск полудублей.
Считаем побитные вероятности.
Can you write Python code for it?

%%time
alpha = 0.1  # suppose that we add/remove `alpha` of the words in a document
stds = Xs.std(axis=0) * alpha
distributions = [sps.norm(0, std) for std in stds]
probabilities = []  # probabilities
for X in np.abs(Xs):
    probabilities.append([dist.cdf(-x) for x, dist in zip(X, distributions)])

probabilities = np.array(probabilities)
np.save("probabilities", probabilities)

probabilities = np.load("probabilities.npy")


I am working on the file "08_simhash_prob_search/simhash_prob.ipynb".
The first task is:
Заведём сет удовлетворяющих xor-ов между хешами.
Can you write Python code for it?

# This function will serve us later as well
def permutations(num_ones, size=SIMHASH_SIZE, start=0):
    # Returns all possible positions of `num_ones` ones in a sequence of `size` bits,
    # whereas the other bits are considered as zeros
    if num_ones == 0:
        yield []
        return
    for position in range(start, size):
        for positions in permutations(num_ones - 1, size, position + 1):
            yield [position] + positions

accepted_xors = set()

for num_ones in xrange(MAX_DISTANCE + 1):
    for positions in permutations(num_ones):
        xor = ["0"] * SIMHASH_SIZE
        for pos in positions:
            xor[pos] = "1"
        accepted_xors.add(np.uint64(int("".join(xor), 2)))

print len(accepted_xors)

def similar(hash1, hash2):
    return (hash1 ^ hash2) in accepted_xors


I am working on the file "08_simhash_prob_search/simhash_prob.ipynb".
The first task is:
Построим индексы.
Can you write Python code for it?

indices = [[list() for __ in xrange(2 ** PART_SIZE)] for _ in xrange(NUM_PARTS)]
for simhash_id in xrange(num_samples):
    simhash_parts = simhashes_parts[simhash_id]
    for part_id in xrange(NUM_PARTS):
        indices[part_id][simhash_parts[part_id]].append(simhash_id)


I am working on the file "08_simhash_prob_search/simhash_prob.ipynb".
The first task is:
Сортируем вычисленные вероятности.
Can you write Python code for it?

most_probable_flipped_bits = []
for probs in probabilities:
    most_probable_flipped_bits.append(np.flip(np.argsort(probs), 0))
most_probable_flipped_bits = np.array(most_probable_flipped_bits)


I am working on the file "08_simhash_prob_search/simhash_prob.ipynb".
The first task is:
Среднее число обращений к множеству симхешей для обнаружения 30%, 50% и 80% полудубликатов:
Can you write Python code for it?

print [np.searchsorted(proportions_found, proportion) for proportion in (0.3, 0.5, 0.8)]


I am working on the file "08_simhash_prob_search/simhash_prob.ipynb".
The first task is:
Точный поиск полудублей.
Поделим simhash-и на 4 части для индексирования.
Can you write Python code for it?

NUM_PARTS = MAX_DISTANCE + 1
PART_SIZE = SIMHASH_SIZE / NUM_PARTS

neg_part_mask = "0" * PART_SIZE
pos_part_mask = "1" * PART_SIZE
masks = [neg_part_mask * part_id + pos_part_mask + neg_part_mask * (NUM_PARTS - part_id - 1)\
         for part_id in range(NUM_PARTS)]
masks = np.array([int(mask, 2) for mask in masks], dtype=np.uint64)

def get_part(simhash, part_id):
    return int(simhash & masks[part_id]) >> (PART_SIZE * (NUM_PARTS - part_id - 1))

simhashes_parts = np.zeros((len(simhashes), NUM_PARTS), dtype=np.int32)
for simhash_id, simhash in enumerate(simhashes):
    for part_id in xrange(NUM_PARTS):
        simhashes_parts[simhash_id][part_id] = get_part(simhash, part_id)


I am working on the file "08_transfer-learning/Transfer_Learning.ipynb".
The first task is:
Below I'm running images through the VGG network in batches.
Can you write Python code for it?

# Set the batch size higher if you can fit in in your GPU memory
batch_size = 100
codes_list = []
labels = []
batch = []

codes = None
vgg = vgg16.Vgg16()
input_ = tf.placeholder(tf.float32, [None, 224, 224, 3])
with tf.name_scope("content_vgg"):
    vgg.build(input_)

with tf.Session() as sess:
    for each in classes:
        print("Starting {} images".format(each))
        class_path = data_dir + each
        files = os.listdir(class_path)
        for ii, file in enumerate(files, 1):
            # Add images to the current batch
            # utils.load_image crops the input images for us, from the center
            img = utils.load_image(os.path.join(class_path, file))
            batch.append(img.reshape((1, 224, 224, 3)))
            labels.append(each)

            # Running the batch through the network to get the codes
            if ii % batch_size == 0 or ii == len(files):

                # Image batch to pass to VGG network
                images = np.concatenate(batch)

                feed_dict = {input_: images}
                codes_batch =  sess.run(vgg.relu6, feed_dict=feed_dict)

                # Here I'm building an array of the codes
                if codes is None:
                    codes = codes_batch
                else:
                    codes = np.concatenate((codes, codes_batch))

                # Reset to start building the next batch
                batch = []
                print('{} images processed'.format(ii))

# write codes to file
with open('codes', 'w') as f:
    codes.tofile(f)

# write labels to file
import csv
with open('labels', 'w') as f:
    writer = csv.writer(f, delimiter='\n')
    writer.writerow(labels)


I am working on the file "08_transfer-learning/Transfer_Learning.ipynb".
The first task is:
Building the Classifier
Now that we have codes for all the images, we can build a simple classifier on top of them. The codes behave just like normal input into a simple neural network. Below I'm going to have you do most of the work.
Can you write Python code for it?

# read codes and labels from file
import csv

with open('labels') as f:
    reader = csv.reader(f, delimiter='\n')
    labels = np.array([each for each in reader if len(each) > 0]).squeeze()
with open('codes') as f:
    codes = np.fromfile(f, dtype=np.float32)
    codes = codes.reshape((len(labels), -1))


I am working on the file "08_transfer-learning/Transfer_Learning.ipynb".
The first task is:
Classifier layers
Once you have the convolutional codes, you just need to build a classfier from some fully connected layers. You use the codes as the inputs and the image labels as targets. Otherwise the classifier is a typical neural network.

Exercise: With the codes and labels loaded, build the classifier. Consider the codes as your inputs, each of them are 4096D vectors. You'll want to use a hidden layer and an output layer as your classifier. Remember that the output layer needs to have one unit for each class and a softmax activation function. Use the cross entropy to calculate the cost.
Can you write Python code for it?

from tensorflow import layers

inputs_ = tf.placeholder(tf.float32, shape=[None, codes.shape[1]])
labels_ = tf.placeholder(tf.int64, shape=[None, labels_vecs.shape[1]])

fc = tf.layers.dense(inputs_, 2000, activation=tf.nn.relu)
logits = tf.layers.dense(fc, labels_vecs.shape[1], activation=None)

cost = tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits(labels=labels_,
                                                             logits=logits))

optimizer = tf.train.AdamOptimizer(learning_rate=0.0001).minimize(cost)

# Operations for validation/test accuracy
predicted = tf.nn.softmax(logits)
correct_pred = tf.equal(tf.argmax(predicted, 1), tf.argmax(labels_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))


I am working on the file "08_transfer-learning/Transfer_Learning.ipynb".
The first task is:
ConvNet Codes
Below, we'll run through all the images in our dataset and get codes for each of them. That is, we'll run the images through the VGGNet convolutional layers and record the values of the first fully connected layer. We can then write these to a file for later when we build our own classifier.
Here we're using the vgg16 module from tensorflow_vgg. The network takes images of size $224 \times 224 \times 3$ as input. Then it has 5 sets of convolutional layers. The network implemented here has this structure (copied from the source code):
```
self.conv1_1 = self.conv_layer(bgr, "conv1_1")
self.conv1_2 = self.conv_layer(self.conv1_1, "conv1_2")
self.pool1 = self.max_pool(self.conv1_2, 'pool1')
self.conv2_1 = self.conv_layer(self.pool1, "conv2_1")
self.conv2_2 = self.conv_layer(self.conv2_1, "conv2_2")
self.pool2 = self.max_pool(self.conv2_2, 'pool2')
self.conv3_1 = self.conv_layer(self.pool2, "conv3_1")
self.conv3_2 = self.conv_layer(self.conv3_1, "conv3_2")
self.conv3_3 = self.conv_layer(self.conv3_2, "conv3_3")
self.pool3 = self.max_pool(self.conv3_3, 'pool3')
self.conv4_1 = self.conv_layer(self.pool3, "conv4_1")
self.conv4_2 = self.conv_layer(self.conv4_1, "conv4_2")
self.conv4_3 = self.conv_layer(self.conv4_2, "conv4_3")
self.pool4 = self.max_pool(self.conv4_3, 'pool4')
self.conv5_1 = self.conv_layer(self.pool4, "conv5_1")
self.conv5_2 = self.conv_layer(self.conv5_1, "conv5_2")
self.conv5_3 = self.conv_layer(self.conv5_2, "conv5_3")
self.pool5 = self.max_pool(self.conv5_3, 'pool5')
self.fc6 = self.fc_layer(self.pool5, "fc6")
self.relu6 = tf.nn.relu(self.fc6)
```
So what we want are the values of the first fully connected layer, after being ReLUd (self.relu6).
Can you write Python code for it?

import os

import numpy as np
import tensorflow as tf

from tensorflow_vgg import vgg16
from tensorflow_vgg import utils

data_dir = 'flower_photos/'
contents = os.listdir(data_dir)
classes = [each for each in contents if os.path.isdir(data_dir + each)]


I am working on the file "08_transfer-learning/Transfer_Learning.ipynb".
The first task is:
Data prep
As usual, now we need to one-hot encode our labels and create validation/test sets. First up, creating our labels!
From scikit-learn, use LabelBinarizer to create one-hot encoded vectors from the labels.
Can you write Python code for it?

from sklearn.preprocessing import LabelBinarizer
lb = LabelBinarizer()
lb.fit(labels)
labels_vecs = lb.transform(labels)
labels_vecs


I am working on the file "08_transfer-learning/Transfer_Learning.ipynb".
The first task is:
Find photos that were mistakenly calassified
Can you write Python code for it?

data_dir = 'flower_photos/'
contents = os.listdir(data_dir)
classes = [each for each in contents if os.path.isdir(data_dir + each)]

with tf.Session() as sess:
    saver = tf.train.Saver()
    with tf.Session() as sess2:
        saver.restore(sess2, tf.train.latest_checkpoint('checkpoints'))

        for each in classes:
            print("Starting {} images".format(each))
            class_path = data_dir + each
            files = os.listdir(class_path)

            for file in files:
                batch = []
                labels = []

                img = utils.load_image(os.path.join(class_path, file))
                batch.append(img.reshape((1, 224, 224, 3)))
                labels.append(lb.transform([each])[0])
                images = np.concatenate(batch)

                feed_dict = {input_: images}
                code = sess.run(vgg.relu6, feed_dict=feed_dict)

                feed = {inputs_: code, labels_: labels}
                correct, prediction = sess2.run([correct_pred, predicted], feed_dict=feed)

                if not correct[0]:
                    #test_img = imread(os.path.join(class_path, file))
                    #plt.imshow(test_img)
                    #plt.barh(np.arange(5), prediction)
                    #_ = plt.yticks(np.arange(5), lb.classes_)
                    print(os.path.join(class_path, file))


I am working on the file "08_transfer-learning/Transfer_Learning.ipynb".
The first task is:
Training
Here, we'll train the network.
Can you write Python code for it?

epochs = 10
batches = 100
saver = tf.train.Saver()
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for e in range(epochs):
        b = 0
        for x, y in get_batches(train_x, train_y, batches):
            feed = {inputs_: x,
                    labels_: y}
            batch_cost, _ = sess.run([cost, optimizer], feed_dict=feed)
            print("Epoch: {}/{} ".format(e+1, epochs),
                  "Batch: {}/{} ".format(b+1, batches),
                  "Training loss: {:.4f}".format(batch_cost))
            b += 1
    saver.save(sess, "checkpoints/flowers.ckpt")


I am working on the file "09 - Ordinary Differential Equation (ODE) Examples.ipynb".
The first task is:
Derivatives and Integrals:
Can you write Python code for it?

# Use sine function to demonstrate derivatives and integrals
X_Pi = np.linspace(0,np.pi,100)
X_2Pi = np.linspace(0,2*np.pi,200)

plt.plot(X_2Pi, np.sin(X_2Pi), label="0 - 2 Pi")
plt.plot(X_Pi, np.sin(X_Pi), linestyle="dashed", color='red',linewidth=3, label="0 - Pi" )
plt.legend(loc='best')
plt.xlabel("x")
plt.ylabel("y");


I am working on the file "09 - Ordinary Differential Equation (ODE) Examples.ipynb".
The first task is:
Equivalently, $ z = [\theta, \omega] $  can be used as an array without referencing $ [\theta, \omega] $ explicitly, saving a couple of lines:
Can you write Python code for it?

def dy2_dx(z, x):
    return (z[1], -3*z[1] -2*z[0])

dy_dxs = odeint(dy2_dx, (0,1), X) # Outputs [y, y']

plt.plot(X, dy_dxs[:, 0], label="y")
plt.plot(X, dy_dxs[:, 1], label="y'")
plt.legend()
plt.show()


I am working on the file "09 - Ordinary Differential Equation (ODE) Examples.ipynb".
The first task is:
First Order Ordinary Differential Equations:
These are differential equations that contain at least one independent variable and its derivatives. Ubiquitous in physics problems, a common example is a body that has both acceleration and velocity.
Can you write Python code for it?