I am working on the file "Exercises/Term1/TensorFlow-L2/lab.ipynb".
The first task is:
<img src="image/mean_variance.png" style="height: 75%;width: 75%; position: relative; right: 5%">
Problem 1
The first problem involves normalizing the features for your training and test data.  I want you to implement Min-Max scaling in the normalize() function to a range of a=0.1 and b=0.9.  Since the notMNIST image data is in greyscale, you'll have to use a max of 255 and min of 0.
Min-Max Scaling:
$
X'=a+{\frac {\left(X-X_{\min }\right)\left(b-a\right)}{X_{\max }-X_{\min }}}
$
If you're having trouble solving problem 1, you can view the solution here.
Can you write Python code for it?

# Problem 1 - Implement Min-Max scaling for greyscale image data
def normalize_greyscale(image_data):
    """
    Normalize the image data with Min-Max scaling to a range of [0.1, 0.9]
    :param image_data: The image data to be normalized
    :return: Normalized image data
    """
    a = 0.1
    b = 0.9

    Xmin = 0
    Xmax = 255

    norm_img = a + (image_data - Xmin)*(b-a)/(Xmax - Xmin)
    return norm_img

### DON'T MODIFY ANYTHING BELOW ###
# Test Cases
np.testing.assert_array_almost_equal(
    normalize_greyscale(np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 255])),
    [0.1, 0.103137254902, 0.106274509804, 0.109411764706, 0.112549019608, 0.11568627451, 0.118823529412, 0.121960784314,
     0.125098039216, 0.128235294118, 0.13137254902, 0.9],
    decimal=3)
np.testing.assert_array_almost_equal(
    normalize_greyscale(np.array([0, 1, 10, 20, 30, 40, 233, 244, 254,255])),
    [0.1, 0.103137254902, 0.13137254902, 0.162745098039, 0.194117647059, 0.225490196078, 0.830980392157, 0.865490196078,
     0.896862745098, 0.9])

train_features = normalize_greyscale(train_features)
test_features = normalize_greyscale(test_features)
is_features_normal = True

print('Tests Passed!')

# Turn labels into numbers and apply One-Hot Encoding
encoder = LabelBinarizer()
encoder.fit(train_labels)
train_labels = encoder.transform(train_labels)
test_labels = encoder.transform(test_labels)

# Change to float32, so it can be multiplied against the features in TensorFlow, which are float32
train_labels = train_labels.astype(np.float32)
test_labels = test_labels.astype(np.float32)
is_labels_encod = True

print('Labels One-Hot Encoded')

assert is_features_normal, 'You skipped the step to normalize the features'
assert is_labels_encod, 'You skipped the step to One-Hot Encode the labels'

# Get randomized datasets for training and validation
train_features, valid_features, train_labels, valid_labels = train_test_split(
    train_features,
    train_labels,
    test_size=0.05,
    random_state=832289)

print('Training...


I am working on the file "Exercises/Term1/TensorFlow-L2/lab.ipynb".
The first task is:
Checkpoint
All your progress is now saved to the pickle file.  If you need to leave and comeback to this lab, you no longer have to start from the beginning.  Just run the code block below and it will load all the data and modules required to proceed.
Can you write Python code for it?

%matplotlib inline

# Load the modules
import pickle
import math

import numpy as np
import tensorflow as tf
from tqdm import tqdm
import matplotlib.pyplot as plt

# Reload the data
pickle_file = 'notMNIST.pickle'
with open(pickle_file, 'rb') as f:
  pickle_data = pickle.load(f)
  train_features = pickle_data['train_dataset']
  train_labels = pickle_data['train_labels']
  valid_features = pickle_data['valid_dataset']
  valid_labels = pickle_data['valid_labels']
  test_features = pickle_data['test_dataset']
  test_labels = pickle_data['test_labels']
  del pickle_data  # Free up memory

print('Data and modules loaded.')


I am working on the file "Exercises/Term1/TensorFlow-L2/lab.ipynb".
The first task is:
Test
Set the epochs, batch_size, and learning_rate with the best learning parameters you discovered in problem 3.  You're going to test your model against your hold out dataset/testing data.  This will give you a good indicator of how well the model will do in the real world.  You should have a test accuracy of atleast 80%.
Can you write Python code for it?

# ToDo: Set the epochs, batch_size, and learning_rate with the best parameters from problem 3
epochs = 5
batch_size = 500
learning_rate = 0.1

### DON'T MODIFY ANYTHING BELOW ###
# The accuracy measured against the test set
test_accuracy = 0.0

with tf.Session() as session:

    session.run(init)
    batch_count = int(math.ceil(len(train_features)/batch_size))

    for epoch_i in range(epochs):

        # Progress bar
        batches_pbar = tqdm(range(batch_count), desc='Epoch {:>2}/{}'.format(epoch_i+1, epochs), unit='batches')

        # The training cycle
        for batch_i in batches_pbar:
            # Get a batch of training features and labels
            batch_start = batch_i*batch_size
            batch_features = train_features[batch_start:batch_start + batch_size]
            batch_labels = train_labels[batch_start:batch_start + batch_size]

            # Run optimizer
            _ = session.run(optimizer, feed_dict={features: batch_features, labels: batch_labels})

        # Check accuracy against Test data
        test_accuracy = session.run(accuracy, feed_dict=test_feed_dict)

assert test_accuracy >= 0.80, 'Test accuracy at {}, should be equal to or greater than 0.80'.format(test_accuracy)
print('Nice Job! Test Accuracy is {}'.format(test_accuracy))


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
Cost-function to be optimized
To make the model better at classifying the input images, we must somehow change the variables for weights and biases. To do this we first need to know how well the model currently performs by comparing the predicted output of the model y_pred to the desired output y_true.
The cross-entropy is a performance measure used in classification. The cross-entropy is a continuous function that is always positive and if the predicted output of the model exactly matches the desired output then the cross-entropy equals zero. The goal of optimization is therefore to minimize the cross-entropy so it gets as close to zero as possible by changing the weights and biases of the model.
TensorFlow has a built-in function for calculating the cross-entropy. Note that it uses the values of the logits because it also calculates the softmax internally.
Can you write Python code for it?

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits=logits,
                                                        labels=y_true)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
Data dimensions
The data dimensions are used in several places in the source-code below. In computer programming it is generally best to use variables and constants rather than having to hard-code specific numbers every time that number is used. This means the numbers only have to be changed in one single place. Ideally these would be inferred from the data that has been read, but here we just write the numbers.
Can you write Python code for it?

# We know that MNIST images are 28 pixels in each dimension.
img_size = 28

# Images are stored in one-dimensional arrays of this length.
img_size_flat = img_size * img_size

# Tuple with height and width of images used to reshape arrays.
img_shape = (img_size, img_size)

# Number of classes, one class for each of 10 digits.
num_classes = 10


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
Finally we have the placeholder variable for the true class of each image in the placeholder variable x. These are integers and the dimensionality of this placeholder variable is set to [None] which means the placeholder variable is a one-dimensional vector of arbitrary length.
Can you write Python code for it?

y_true_cls = tf.placeholder(tf.int64, [None])


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
Function for performing a number of optimization iterations so as to gradually improve the weights and biases of the model. In each iteration, a new batch of data is selected from the training-set and then TensorFlow executes the optimizer using those training samples.
Can you write Python code for it?

def optimize(num_iterations):
    for i in range(num_iterations):
        # Get a batch of training examples.
        # x_batch now holds a batch of images and
        # y_true_batch are the true labels for those images.
        x_batch, y_true_batch = data.train.next_batch(batch_size)

        # Put the batch into a dict with the proper names
        # for placeholder variables in the TensorFlow graph.
        # Note that the placeholder for y_true_cls is not set
        # because it is not used during training.
        feed_dict_train = {x: x_batch,
                           y_true: y_true_batch}

        # Run the optimizer using this batch of training data.
        # TensorFlow assigns the variables in feed_dict_train
        # to the placeholder variables and then runs the optimizer.
        session.run(optimizer, feed_dict=feed_dict_train)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
Function for plotting examples of images from the test-set that have been mis-classified.
Can you write Python code for it?

def plot_example_errors():
    # Use TensorFlow to get a list of boolean values
    # whether each test-image has been correctly classified,
    # and a list for the predicted class of each image.
    correct, cls_pred = session.run([correct_prediction, y_pred_cls],
                                    feed_dict=feed_dict_test)

    # Negate the boolean array.
    incorrect = (correct == False)

    # Get the images from the test-set that have been
    # incorrectly classified.
    images = data.test.images[incorrect]

    # Get the predicted classes for those images.
    cls_pred = cls_pred[incorrect]

    # Get the true classes for those images.
    cls_true = data.test.cls[incorrect]

    # Plot the first 9 images.
    plot_images(images=images[0:9],
                cls_true=cls_true[0:9],
                cls_pred=cls_pred[0:9])


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
Function for printing and plotting the confusion matrix using scikit-learn.
Can you write Python code for it?

def print_confusion_matrix():
    # Get the true classifications for the test-set.
    cls_true = data.test.cls

    # Get the predicted classifications for the test-set.
    cls_pred = session.run(y_pred_cls, feed_dict=feed_dict_test)

    # Get the confusion matrix using sklearn.
    cm = confusion_matrix(y_true=cls_true,
                          y_pred=cls_pred)

    # Print the confusion matrix as text.
    print(cm)

    # Plot the confusion matrix as an image.
    plt.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)

    # Make various adjustments to the plot.
    plt.tight_layout()
    plt.colorbar()
    tick_marks = np.arange(num_classes)
    plt.xticks(tick_marks, range(num_classes))
    plt.yticks(tick_marks, range(num_classes))
    plt.xlabel('Predicted')
    plt.ylabel('True')


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
Function for printing the classification accuracy on the test-set.
Can you write Python code for it?

def print_accuracy():
    # Use TensorFlow to compute the accuracy.
    acc = session.run(accuracy, feed_dict=feed_dict_test)

    # Print the accuracy.
    print("Accuracy on test-set: {0:.1%}".format(acc))


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
Helper-function to plot the model weights
Function for plotting the weights of the model. 10 images are plotted, one for each digit that the model is trained to recognize.
Can you write Python code for it?

def plot_weights():
    # Get the values for the weights from the TensorFlow variable.
    w = session.run(weights)

    # Get the lowest and highest values for the weights.
    # This is used to correct the colour intensity across
    # the images so they can be compared with each other.
    w_min = np.min(w)
    w_max = np.max(w)

    # Create figure with 3x4 sub-plots,
    # where the last 2 sub-plots are unused.
    fig, axes = plt.subplots(3, 4)
    fig.subplots_adjust(hspace=0.3, wspace=0.3)

    for i, ax in enumerate(axes.flat):
        # Only use the weights for the first 10 sub-plots.
        if i<10:
            # Get the weights for the i'th digit and reshape it.
            # Note that w.shape == (img_size_flat, 10)
            image = w[:, i].reshape(img_shape)

            # Set the label for the sub-plot.
            ax.set_xlabel("Weights: {0}".format(i))

            # Plot the image.
            ax.imshow(image, vmin=w_min, vmax=w_max, cmap='seismic')

        # Remove ticks from each sub-plot.
        ax.set_xticks([])
        ax.set_yticks([])


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
Helper-functions to show performance
Dict with the test-set data to be used as input to the TensorFlow graph. Note that we must use the correct names for the placeholder variables in the TensorFlow graph.
Can you write Python code for it?

feed_dict_test = {x: data.test.images,
                  y_true: data.test.labels,
                  y_true_cls: data.test.cls}


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
Load Data
The MNIST data-set is about 12 MB and will be downloaded automatically if it is not located in the given path.
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
data = input_data.read_data_sets("data/MNIST/", one_hot=True)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
Model
This simple mathematical model multiplies the images in the placeholder variable x with the weights and then adds the biases.
The result is a matrix of shape [num_images, num_classes] because x has shape [num_images, img_size_flat] and weights has shape [img_size_flat, num_classes], so the multiplication of those two matrices is a matrix with shape [num_images, num_classes] and then the biases vector is added to each row of that matrix.
Note that the name logits is typical TensorFlow terminology, but other people may call the variable something else.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
Now logits is a matrix with num_images rows and num_classes columns, where the element of the $i$'th row and $j$'th column is an estimate of how likely the $i$'th input image is to be of the $j$'th class.
However, these estimates are a bit rough and difficult to interpret because the numbers may be very small or large, so we want to normalize them so that each row of the logits matrix sums to one, and each element is limited between zero and one. This is calculated using the so-called softmax function and the result is stored in y_pred.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
One-Hot Encoding
The data-set has been loaded as so-called One-Hot encoding. This means the labels have been converted from a single number to a vector whose length equals the number of possible classes. All elements of the vector are zero except for the $i$'th element which is one and means the class is $i$. For example, the One-Hot encoded labels for the first 5 images in the test-set are:
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
Performance after 1 optimization iteration
Already after a single optimization iteration, the model has increased its accuracy on the test-set to 40.7% up from 9.8%. This means that it mis-classifies the images about 6 out of 10 times, as demonstrated on a few examples below.
Can you write Python code for it?

optimize(num_iterations=1)

print_accuracy()

plot_example_errors()


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
Performance after 1000 optimization iterations
After 1000 optimization iterations, the model only mis-classifies about one in ten images. As demonstrated below, some of the mis-classifications are justified because the images are very hard to determine with certainty even for humans, while others are quite obvious and should have been classified correctly by a good model. But this simple model cannot reach much better performance and more complex models are therefore needed.
Can you write Python code for it?

# We have already performed 10 iterations.
optimize(num_iterations=9000)

print_accuracy()

plot_example_errors()


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
Performance before any optimization
The accuracy on the test-set is 9.8%. This is because the model has only been initialized and not optimized at all, so it always predicts that the image shows a zero digit, as demonstrated in the plot below, and it turns out that 9.8% of the images in the test-set happens to be zero digits.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
Performance measures
We need a few more performance measures to display the progress to the user.
This is a vector of booleans whether the predicted class equals the true class of each image.
Can you write Python code for it?

correct_prediction = tf.equal(y_pred_cls, y_true_cls)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
Plot a few images to see if data is correct
Can you write Python code for it?

# Get the first images from the test-set.
images = data.test.images[0:9]

# Get the true classes for those images.
cls_true = data.test.cls[0:9]

# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
The MNIST data-set has now been loaded and consists of 70.000 images and associated labels (i.e. classifications of the images). The data-set is split into 3 mutually exclusive sub-sets. We will only use the training and test-sets in this tutorial.
Can you write Python code for it?

print("Size of:")
print("- Training-set:\t\t{}".format(len(data.train.labels)))
print("- Test-set:\t\t{}".format(len(data.test.labels)))
print("- Validation-set:\t{}".format(len(data.validation.labels)))


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
The model has now been trained for 1000 optimization iterations, with each iteration using 100 images from the training-set. Because of the great variety of the images, the weights have now become difficult to interpret and we may doubt whether the model truly understands how digits are composed from lines, or whether the model has just memorized many different variations of pixels.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
The predicted class can be calculated from the y_pred matrix by taking the index of the largest element in each row.
Can you write Python code for it?

y_pred_cls = tf.argmax(y_pred, dimension=1)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
The second variable that must be optimized is called biases and is defined as a 1-dimensional tensor (or vector) of length num_classes.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
This calculates the classification accuracy by first type-casting the vector of booleans to floats, so that False becomes 0 and True becomes 1, and then calculating the average of these numbers.
Can you write Python code for it?

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
This was developed using Python 3.5.2 (Anaconda) and TensorFlow version:
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
We also need the classes as single numbers for various comparisons and performance measures, so we convert the One-Hot encoded vectors to a single number by taking the index of the highest element. Note that the word 'class' is a keyword used in Python so we need to use the name 'cls' instead.
Can you write Python code for it?

data.test.cls = np.array([label.argmax() for label in data.test.labels])


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
We are now done using TensorFlow, so we close the session to release its resources.
Can you write Python code for it?

# This has been commented out in case you want to modify and experiment
# with the Notebook without having to restart it.
# session.close()


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
We can also print and plot the so-called confusion matrix which lets us see more details about the mis-classifications. For example, it shows that images actually depicting a 5 have sometimes been mis-classified as all other possible digits, but mostly either 3, 6 or 8.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/01_Simple_Linear_Model.ipynb".
The first task is:
We can now see the class for the first five images in the test-set. Compare these to the One-Hot encoded vectors above. For example, the class for the first image is 7, which corresponds to a One-Hot encoded vector where all elements are zero except for the element with index 7.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Applying each of these convolutional filters to the first input image gives the following output images, which are then used as input to the second convolutional layer. Note that these images are down-sampled to 14 x 14 pixels which is half the resolution of the original input image.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Check that the output of the fully-connected layer is a tensor with shape (?, 128) where the ? means there is an arbitrary number of images and fc_size == 128.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Check that the tensors now have shape (?, 1764) which means there's an arbitrary number of images which have been flattened to vectors of length 1764 each. Note that 1764 = 7 x 7 x 36.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Configuration of Neural Network
The configuration of the Convolutional Neural Network is defined here for convenience, so you can easily find and change these numbers and re-run the Notebook.
Can you write Python code for it?

# Convolutional Layer 1.
filter_size1 = 5          # Convolution filters are 5 x 5 pixels.
num_filters1 = 16         # There are 16 of these filters.

# Convolutional Layer 2.
filter_size2 = 5          # Convolution filters are 5 x 5 pixels.
num_filters2 = 36         # There are 36 of these filters.

# Fully-connected layer.
fc_size = 128             # Number of neurons in fully-connected layer.


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Convolutional Layer 1
Create the first convolutional layer. It takes x_image as input and creates num_filters1 different filters, each having width and height equal to filter_size1. Finally we wish to down-sample the image so it is half the size by using 2x2 max-pooling.
Can you write Python code for it?

layer_conv1, weights_conv1 = \
    new_conv_layer(input=x_image,
                   num_input_channels=num_channels,
                   filter_size=filter_size1,
                   num_filters=num_filters1,
                   use_pooling=True)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Convolutional Layer 2
Create the second convolutional layer, which takes as input the output from the first convolutional layer. The number of input channels corresponds to the number of filters in the first convolutional layer.
Can you write Python code for it?

layer_conv2, weights_conv2 = \
    new_conv_layer(input=layer_conv1,
                   num_input_channels=num_filters1,
                   filter_size=filter_size2,
                   num_filters=num_filters2,
                   use_pooling=True)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Cost-function to be optimized
To make the model better at classifying the input images, we must somehow change the variables for all the network layers. To do this we first need to know how well the model currently performs by comparing the predicted output of the model y_pred to the desired output y_true.
The cross-entropy is a performance measure used in classification. The cross-entropy is a continuous function that is always positive and if the predicted output of the model exactly matches the desired output then the cross-entropy equals zero. The goal of optimization is therefore to minimize the cross-entropy so it gets as close to zero as possible by changing the variables of the network layers.
TensorFlow has a built-in function for calculating the cross-entropy. Note that the function calculates the softmax internally so we must use the output of layer_fc2 directly rather than y_pred which has already had the softmax applied.
Can you write Python code for it?

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits=layer_fc2,
                                                        labels=y_true)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Flatten Layer
The convolutional layers output 4-dim tensors. We now wish to use these as input in a fully-connected network, which requires for the tensors to be reshaped or flattened to 2-dim tensors.
Can you write Python code for it?

layer_flat, num_features = flatten_layer(layer_conv2)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
From these images, it looks like the second convolutional layer might detect lines and patterns in the input images, which are less sensitive to local variations in the original input images.
These images are then flattened and input to the fully-connected layer, but that is not shown here.
Close TensorFlow Session
We are now done using TensorFlow, so we close the session to release its resources.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Fully-Connected Layer 1
Add a fully-connected layer to the network. The input is the flattened layer from the previous convolution. The number of neurons or nodes in the fully-connected layer is fc_size. ReLU is used so we can learn non-linear relations.
Can you write Python code for it?

layer_fc1 = new_fc_layer(input=layer_flat,
                         num_inputs=num_features,
                         num_outputs=fc_size,
                         use_relu=True)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Fully-Connected Layer 2
Add another fully-connected layer that outputs vectors of length 10 for determining which of the 10 classes the input image belongs to. Note that ReLU is not used in this layer.
Can you write Python code for it?

layer_fc2 = new_fc_layer(input=layer_fc1,
                         num_inputs=fc_size,
                         num_outputs=num_classes,
                         use_relu=False)

layer_fc2


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Function for performing a number of optimization iterations so as to gradually improve the variables of the network layers. In each iteration, a new batch of data is selected from the training-set and then TensorFlow executes the optimizer using those training samples.  The progress is printed every 100 iterations.
Can you write Python code for it?

# Counter for total number of iterations performed so far.
total_iterations = 0

def optimize(num_iterations):
    # Ensure we update the global variable rather than a local copy.
    global total_iterations

    # Start-time used for printing time-usage below.
    start_time = time.time()

    for i in range(total_iterations,
                   total_iterations + num_iterations):

        total_batch = int(mnist.train.num_examples/batch_size)
        # Get a batch of training examples.
        # x_batch now holds a batch of images and
        # y_true_batch are the true labels for those images.
        x_batch, y_true_batch = data.train.next_batch(train_batch_size)

        # Put the batch into a dict with the proper names
        # for placeholder variables in the TensorFlow graph.
        feed_dict_train = {x: x_batch,
                           y_true: y_true_batch}

        # Run the optimizer using this batch of training data.
        # TensorFlow assigns the variables in feed_dict_train
        # to the placeholder variables and then runs the optimizer.
        session.run(optimizer, feed_dict=feed_dict_train)

        # Print status every 100 iterations.
        if i % 100 == 0:
            # Calculate the accuracy on the training-set.
            acc = session.run(accuracy, feed_dict=feed_dict_train)

            # Message for printing.
            msg = "Optimization Iteration: {0:>6}, Training Accuracy: {1:>6.1%}"

            # Print it.
            print(msg.format(i + 1, acc))

    # Update the total number of iterations performed.
    total_iterations += num_iterations

    # Ending time.
    end_time = time.time()

    # Difference between start and end-times.
    time_dif = end_time - start_time

    # Print the time-usage.
    print("Time usage: " + str(timedelta(seconds=int(round(time_dif)))))


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Helper-function for creating a new Convolutional Layer
This function creates a new convolutional layer in the computational graph for TensorFlow. Nothing is actually calculated here, we are just adding the mathematical formulas to the TensorFlow graph.
It is assumed that the input is a 4-dim tensor with the following dimensions:

Image number.
Y-axis of each image.
X-axis of each image.
Channels of each image.

Note that the input channels may either be colour-channels, or it may be filter-channels if the input is produced from a previous convolutional layer.
The output is another 4-dim tensor with the following dimensions:

Image number, same as input.
Y-axis of each image. If 2x2 pooling is used, then the height and width of the input images is divided by 2.
X-axis of each image. Ditto.
Channels produced by the convolutional filters.
Can you write Python code for it?

def new_conv_layer(input,              # The previous layer.
                   num_input_channels, # Num. channels in prev. layer.
                   filter_size,        # Width and height of each filter.
                   num_filters,        # Number of filters.
                   use_pooling=True):  # Use 2x2 max-pooling.

    # Shape of the filter-weights for the convolution.
    # This format is determined by the TensorFlow API.
    shape = [filter_size, filter_size, num_input_channels, num_filters]

    # Create new weights aka. filters with the given shape.
    weights = new_weights(shape=shape)

    # Create new biases, one for each filter.
    biases = new_biases(length=num_filters)

    # Create the TensorFlow operation for convolution.
    # Note the strides are set to 1 in all dimensions.
    # The first and last stride must always be 1,
    # because the first is for the image-number and
    # the last is for the input-channel.
    # But e.g. strides=[1, 2, 2, 1] would mean that the filter
    # is moved 2 pixels across the x- and y-axis of the image.
    # The padding is set to 'SAME' which means the input image
    # is padded with zeroes so the size of the output is the same.
    layer = tf.nn.conv2d(input=input,
                         filter=weights,
                         strides=[1, 1, 1, 1],
                         padding='SAME')

    # Add the biases to the results of the convolution.
    # A bias-value is added to each filter-channel.
    layer += biases

    # Use pooling to down-sample the image resolution?
    if use_pooling:
        # This is 2x2 max-pooling, which means that we
        # consider 2x2 windows and select the largest value
        # in each window. Then we move 2 pixels to the next window.
        layer = tf.nn.max_pool(value=layer,
                               ksize=[1, 2, 2, 1],
                               strides=[1, 2, 2, 1],
                               padding='SAME')

    # Rectified Linear Unit (ReLU).
    # It calculates max(x, 0) for each input...


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Helper-function for creating a new Fully-Connected Layer
This function creates a new fully-connected layer in the computational graph for TensorFlow. Nothing is actually calculated here, we are just adding the mathematical formulas to the TensorFlow graph.
It is assumed that the input is a 2-dim tensor of shape [num_images, num_inputs]. The output is a 2-dim tensor of shape [num_images, num_outputs].
Can you write Python code for it?

def new_fc_layer(input,          # The previous layer.
                 num_inputs,     # Num. inputs from prev. layer.
                 num_outputs,    # Num. outputs.
                 use_relu=True): # Use Rectified Linear Unit (ReLU)?

    # Create new weights and biases.
    weights = new_weights(shape=[num_inputs, num_outputs])
    biases = new_biases(length=num_outputs)

    # Calculate the layer as the matrix multiplication of
    # the input and weights, and then add the bias-values.
    layer = tf.matmul(input, weights) + biases

    # Use ReLU?
    if use_relu:
        layer = tf.nn.relu(layer)

    return layer


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Helper-function for flattening a layer
A convolutional layer produces an output tensor with 4 dimensions. We will add fully-connected layers after the convolution layers, so we need to reduce the 4-dim tensor to 2-dim which can be used as input to the fully-connected layer.
Can you write Python code for it?

def flatten_layer(layer):
    # Get the shape of the input layer.
    layer_shape = layer.get_shape()

    # The shape of the input layer is assumed to be:
    # layer_shape == [num_images, img_height, img_width, num_channels]

    # The number of features is: img_height * img_width * num_channels
    # We can use a function from TensorFlow to calculate this.
    num_features = layer_shape[1:4].num_elements()

    # Reshape the layer to [num_images, num_features].
    # Note that we just set the size of the second dimension
    # to num_features and the size of the first dimension to -1
    # which means the size in that dimension is calculated
    # so the total size of the tensor is unchanged from the reshaping.
    layer_flat = tf.reshape(layer, [-1, num_features])

    # The shape of the flattened layer is now:
    # [num_images, img_height * img_width * num_channels]

    # Return both the flattened layer and the number of features.
    return layer_flat, num_features


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Helper-function for plotting images
Function used to plot 9 images in a 3x3 grid, and writing the true and predicted classes below each image.
Can you write Python code for it?

def plot_images(images, cls_true, cls_pred=None):
    assert len(images) == len(cls_true) == 9

    # Create figure with 3x3 sub-plots.
    fig, axes = plt.subplots(3, 3)
    fig.subplots_adjust(hspace=0.3, wspace=0.3)

    for i, ax in enumerate(axes.flat):
        # Plot image.
        ax.imshow(images[i].reshape(img_shape), cmap='binary')

        # Show true and predicted classes.
        if cls_pred is None:
            xlabel = "True: {0}".format(cls_true[i])
        else:
            xlabel = "True: {0}, Pred: {1}".format(cls_true[i], cls_pred[i])

        # Show the classes as the label on the x-axis.
        ax.set_xlabel(xlabel)

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Helper-function for plotting the output of a convolutional layer
Can you write Python code for it?

def plot_conv_layer(layer, image):
    # Assume layer is a TensorFlow op that outputs a 4-dim tensor
    # which is the output of a convolutional layer,
    # e.g. layer_conv1 or layer_conv2.

    # Create a feed-dict containing just one image.
    # Note that we don't need to feed y_true because it is
    # not used in this calculation.
    feed_dict = {x: [image]}

    # Calculate and retrieve the output values of the layer
    # when inputting that image.
    values = session.run(layer, feed_dict=feed_dict)

    # Number of filters used in the conv. layer.
    num_filters = values.shape[3]

    # Number of grids to plot.
    # Rounded-up, square-root of the number of filters.
    num_grids = math.ceil(math.sqrt(num_filters))

    # Create figure with a grid of sub-plots.
    fig, axes = plt.subplots(num_grids, num_grids)

    # Plot the output images of all the filters.
    for i, ax in enumerate(axes.flat):
        # Only plot the images for valid filters.
        if i<num_filters:
            # Get the output image of using the i'th filter.
            # See new_conv_layer() for details on the format
            # of this 4-dim tensor.
            img = values[0, :, :, i]

            # Plot image.
            ax.imshow(img, interpolation='nearest', cmap='binary')

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Helper-function for showing the performance
Function for printing the classification accuracy on the test-set.
It takes a while to compute the classification for all the images in the test-set, that's why the results are re-used by calling the above functions directly from this function, so the classifications don't have to be recalculated by each function.
Note that this function can use a lot of computer memory, which is why the test-set is split into smaller batches. If you have little RAM in your computer and it crashes, then you can try and lower the batch-size.
Can you write Python code for it?

# Split the test-set into smaller batches of this size.
test_batch_size = 256

def print_test_accuracy(show_example_errors=False,
                        show_confusion_matrix=False):

    # Number of images in the test-set.
    num_test = len(data.test.images)

    # Allocate an array for the predicted classes which
    # will be calculated in batches and filled into this array.
    cls_pred = np.zeros(shape=num_test, dtype=np.int)

    # Now calculate the predicted classes for the batches.
    # We will just iterate through all the batches.
    # There might be a more clever and Pythonic way of doing this.

    # The starting index for the next batch is denoted i.
    i = 0

    while i < num_test:
        # The ending index for the next batch is denoted j.
        j = min(i + test_batch_size, num_test)

        # Get the images from the test-set between index i and j.
        images = data.test.images[i:j, :]

        # Get the associated labels.
        labels = data.test.labels[i:j, :]

        # Create a feed-dict with these images and labels.
        feed_dict = {x: images,
                     y_true: labels}

        # Calculate the predicted class using TensorFlow.
        cls_pred[i:j] = session.run(y_pred_cls, feed_dict=feed_dict)

        # Set the start-index for the next batch to the
        # end-index of the current batch.
        i = j

    # Convenience variable for the true class-numbers of the test-set.
    cls_true = data.test.cls

    # Create a boolean array whether each image is correctly classified.
    correct = (cls_true == cls_pred)

    # Calculate the number of correctly classified images.
    # When summing a boolean array, False means 0 and True means 1.
    correct_sum = correct.sum()

    # Classification accuracy is the number of correctly classified
    # images divided by the total number of images in the test-set.
    acc = float(correct_sum) / num_test

    # Print the accuracy.
    msg = "Accuracy on Test-Set: {0:.1%} ({1} / {2})"
    print(msg.format(acc, correct_sum,...


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Helper-function to perform optimization iterations
There are 55,000 images in the training-set. It takes a long time to calculate the gradient of the model using all these images. We therefore only use a small batch of images in each iteration of the optimizer.
If your computer crashes or becomes very slow because you run out of RAM, then you may try and lower this number, but you may then need to perform more optimization iterations.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Helper-function to plot confusion matrix
Can you write Python code for it?

def plot_confusion_matrix(cls_pred):
    # This is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # Get the true classifications for the test-set.
    cls_true = data.test.cls

    # Get the confusion matrix using sklearn.
    cm = confusion_matrix(y_true=cls_true,
                          y_pred=cls_pred)

    # Print the confusion matrix as text.
    print(cm)

    # Plot the confusion matrix as an image.
    plt.matshow(cm)

    # Make various adjustments to the plot.
    plt.colorbar()
    tick_marks = np.arange(num_classes)
    plt.xticks(tick_marks, range(num_classes))
    plt.yticks(tick_marks, range(num_classes))
    plt.xlabel('Predicted')
    plt.ylabel('True')

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Initialize variables
The variables for weights and biases must be initialized before we start optimizing them.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Input Images
Helper-function for plotting an image.
Can you write Python code for it?

def plot_image(image):
    plt.imshow(image.reshape(img_shape),
               interpolation='nearest',
               cmap='binary')

    plt.show()


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
It is difficult to see from these images what the purpose of the convolutional filters might be. It appears that they have merely created several variations of the input image, as if light was shining from different angles and casting shadows in the image.
Convolution Layer 2
Now plot the filter-weights for the second convolutional layer.
There are 16 output channels from the first conv-layer, which means there are 16 input channels to the second conv-layer. The second conv-layer has a set of filter-weights for each of its input channels. We start by plotting the filter-weigths for the first channel.
Note again that positive weights are red and negative weights are blue.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Load Data
The MNIST data-set is about 12 MB and will be downloaded automatically if it is not located in the given path.
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
data = input_data.read_data_sets('data/MNIST/', one_hot=True)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Performance Measures
We need a few more performance measures to display the progress to the user.
This is a vector of booleans whether the predicted class equals the true class of each image.
Can you write Python code for it?

correct_prediction = tf.equal(y_pred_cls, y_true_cls)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Performance after 1 optimization iteration
The classification accuracy does not improve much from just 1 optimization iteration, because the learning-rate for the optimizer is set very low.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Performance after 100 optimization iterations
After 100 optimization iterations, the model has significantly improved its classification accuracy.
Can you write Python code for it?

optimize(num_iterations=99) # We already performed 1 iteration above.

print_test_accuracy(show_example_errors=True)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Performance after 1000 optimization iterations
After 1000 optimization iterations, the model has greatly increased its accuracy on the test-set to more than 90%.
Can you write Python code for it?

optimize(num_iterations=900) # We performed 100 iterations above.

print_test_accuracy(show_example_errors=True)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Performance before any optimization
The accuracy on the test-set is very low because the model variables have only been initialized and not optimized at all, so it just classifies the images randomly.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Placeholder variables
Placeholder variables serve as the input to the TensorFlow computational graph that we may change each time we execute the graph. We call this feeding the placeholder variables and it is demonstrated further below.
First we define the placeholder variable for the input images. This allows us to change the images that are input to the TensorFlow graph. This is a so-called tensor, which just means that it is a multi-dimensional vector or matrix. The data-type is set to float32 and the shape is set to [None, img_size_flat], where None means that the tensor may hold an arbitrary number of images with each image being a vector of length img_size_flat.
Can you write Python code for it?

x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='x')


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Plot a few images to see if data is correct
Can you write Python code for it?

# Get the first images from the test-set.
images = data.test.images[0:9]

# Get the true classes for those images.
cls_true = data.test.cls[0:9]

# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Plot an image from the test-set which will be used as an example below.
Can you write Python code for it?

image1 = data.test.images[0]
plot_image(image1)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
Plot another example image from the test-set.
Can you write Python code for it?

image2 = data.test.images[13]
plot_image(image2)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
TensorFlow Graph
The entire purpose of TensorFlow is to have a so-called computational graph that can be executed much more efficiently than if the same calculations were to be performed directly in Python. TensorFlow can be more efficient than NumPy because TensorFlow knows the entire computation graph that must be executed, while NumPy only knows the computation of a single mathematical operation at a time.
TensorFlow can also automatically calculate the gradients that are needed to optimize the variables of the graph so as to make the model perform better. This is because the graph is a combination of simple mathematical expressions so the gradient of the entire graph can be calculated using the chain-rule for derivatives.
TensorFlow can also take advantage of multi-core CPUs as well as GPUs - and Google has even built special chips just for TensorFlow which are called TPUs (Tensor Processing Units) and are even faster than GPUs.
A TensorFlow graph consists of the following parts which will be detailed below:

Placeholder variables used for inputting data to the graph.
Variables that are going to be optimized so as to make the convolutional network perform better.
The mathematical formulas for the convolutional network.
A cost measure that can be used to guide the optimization of the variables.
An optimization method which updates the variables.

In addition, the TensorFlow graph may also contain various debugging statements e.g. for logging data to be displayed using TensorBoard, which is not covered in this tutorial.
Helper-functions for creating new variables
Functions for creating new TensorFlow variables in the given shape and initializing them with random values. Note that the initialization is not actually done at this point, it is merely being defined in the TensorFlow graph.
Can you write Python code for it?

def new_weights(shape):
    return tf.Variable(tf.truncated_normal(shape, stddev=0.05))

def new_biases(length):
    return tf.Variable(tf.constant(0.05, shape=[length]))


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
The MNIST data-set has now been loaded and consists of 70,000 images and associated labels (i.e. classifications of the images). The data-set is split into 3 mutually exclusive sub-sets. We will only use the training and test-sets in this tutorial.
Can you write Python code for it?

print("Size of:")
print("- Training-set:\t\t{}".format(len(data.train.labels)))
print("- Test-set:\t\t{}".format(len(data.test.labels)))
print("- Validation-set:\t{}".format(len(data.validation.labels)))


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
The class-labels are One-Hot encoded, which means that each label is a vector with 10 elements, all of which are zero except for one element. The index of this one element is the class-number, that is, the digit shown in the associated image. We also need the class-numbers as integers for the test-set, so we calculate it now.
Can you write Python code for it?

data.test.cls = np.argmax(data.test.labels, axis=1)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
The convolutional layers expect x to be encoded as a 4-dim tensor so we have to reshape it so its shape is instead [num_images, img_height, img_width, num_channels]. Note that img_height == img_width == img_size and num_images can be inferred automatically by using -1 for the size of the first dimension. So the reshape operation is:
Can you write Python code for it?

x_image = tf.reshape(x, [-1, img_size, img_size, num_channels])


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
The following images are the results of applying the convolutional filters to the second image.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
This calculates the classification accuracy by first type-casting the vector of booleans to floats, so that False becomes 0 and True becomes 1, and then calculating the average of these numbers.
Can you write Python code for it?

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
This was developed using Python 3.5.2 (Anaconda) and TensorFlow version:
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/02_Convolutional_Neural_Network.ipynb".
The first task is:
We could also have a placeholder variable for the class-number, but we will instead calculate it using argmax. Note that this is a TensorFlow operator so nothing is calculated at this point.
Can you write Python code for it?

y_true_cls = tf.argmax(y_true, dimension=1)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
Close TensorFlow Session
We are now done using TensorFlow, so we close the session to release its resources.
Can you write Python code for it?

# This has been commented out in case you want to modify and experiment
# with the Notebook without having to restart it.
# session.close()


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
Convolution Layer 1
Now plot the filter-weights for the first convolutional layer.
Note that positive weights are red and negative weights are blue.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
Convolution Layer 2
Now plot the filter-weights for the second convolutional layer.
There are 16 output channels from the first conv-layer, which means there are 16 input channels to the second conv-layer. The second conv-layer has a set of filter-weights for each of its input channels. We start by plotting the filter-weigths for the first channel.
Note again that positive weights are red and negative weights are blue.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
Function for performing a number of optimization iterations so as to gradually improve the variables of the network layers. In each iteration, a new batch of data is selected from the training-set and then TensorFlow executes the optimizer using those training samples.  The progress is printed every 100 iterations.
Can you write Python code for it?

# Counter for total number of iterations performed so far.
total_iterations = 0

def optimize(num_iterations):
    # Ensure we update the global variable rather than a local copy.
    global total_iterations

    # Start-time used for printing time-usage below.
    start_time = time.time()

    for i in range(total_iterations,
                   total_iterations + num_iterations):

        # Get a batch of training examples.
        # x_batch now holds a batch of images and
        # y_true_batch are the true labels for those images.
        x_batch, y_true_batch = data.train.next_batch(train_batch_size)

        # Put the batch into a dict with the proper names
        # for placeholder variables in the TensorFlow graph.
        feed_dict_train = {x: x_batch,
                           y_true: y_true_batch}

        # Run the optimizer using this batch of training data.
        # TensorFlow assigns the variables in feed_dict_train
        # to the placeholder variables and then runs the optimizer.
        session.run(optimizer, feed_dict=feed_dict_train)

        # Print status every 100 iterations.
        if i % 100 == 0:
            # Calculate the accuracy on the training-set.
            acc = session.run(accuracy, feed_dict=feed_dict_train)

            # Message for printing.
            msg = "Optimization Iteration: {0:>6}, Training Accuracy: {1:>6.1%}"

            # Print it.
            print(msg.format(i + 1, acc))

    # Update the total number of iterations performed.
    total_iterations += num_iterations

    # Ending time.
    end_time = time.time()

    # Difference between start and end-times.
    time_dif = end_time - start_time

    # Print the time-usage.
    print("Time usage: " + str(timedelta(seconds=int(round(time_dif)))))


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
Helper-function for plotting images
Function used to plot 9 images in a 3x3 grid, and writing the true and predicted classes below each image.
Can you write Python code for it?

def plot_images(images, cls_true, cls_pred=None):
    assert len(images) == len(cls_true) == 9

    # Create figure with 3x3 sub-plots.
    fig, axes = plt.subplots(3, 3)
    fig.subplots_adjust(hspace=0.3, wspace=0.3)

    for i, ax in enumerate(axes.flat):
        # Plot image.
        ax.imshow(images[i].reshape(img_shape), cmap='binary')

        # Show true and predicted classes.
        if cls_pred is None:
            xlabel = "True: {0}".format(cls_true[i])
        else:
            xlabel = "True: {0}, Pred: {1}".format(cls_true[i], cls_pred[i])

        # Show the classes as the label on the x-axis.
        ax.set_xlabel(xlabel)

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
Initialize variables
The variables for weights and biases must be initialized before we start optimizing them.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
Load Data
The MNIST data-set is about 12 MB and will be downloaded automatically if it is not located in the given path.
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
data = input_data.read_data_sets('data/MNIST/', one_hot=True)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
Now that we have wrapped the input image in a PrettyTensor object, we can add the convolutional and fully-connected layers in just a few lines of source-code.
Note that pt.defaults_scope(activation_fn=tf.nn.relu) makes activation_fn=tf.nn.relu an argument for each of the layers constructed inside the with-block, so that Rectified Linear Units (ReLU) are used for each of these layers. The defaults_scope makes it easy to change arguments for all of the layers.
Can you write Python code for it?

with pt.defaults_scope(activation_fn=tf.nn.relu):
    y_pred, loss = x_pretty.\
        conv2d(kernel=5, depth=16, name='layer_conv1').\
        max_pool(kernel=2, stride=2).\
        conv2d(kernel=5, depth=36, name='layer_conv2').\
        max_pool(kernel=2, stride=2).\
        flatten().\
        fully_connected(size=128, name='layer_fc1').\
        softmax_classifier(class_count=10, labels=y_true)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
Optimization Method
PrettyTensor gave us the predicted class-label (y_pred) as well as a loss-measure that must be minimized, so as to improve the ability of the Neural Network to classify the input images.
It is unclear from the documentation for PrettyTensor whether the loss-measure is cross-entropy or something else. But we now use the AdamOptimizer to minimize the loss.
Note that optimization is not performed at this point. In fact, nothing is calculated at all, we just add the optimizer-object to the TensorFlow graph for later execution.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
Performance Measures
We need a few more performance measures to display the progress to the user.
First we calculate the predicted class number from the output of the Neural Network y_pred, which is a vector with 10 elements. The class number is the index of the largest element.
Can you write Python code for it?

y_pred_cls = tf.argmax(y_pred, dimension=1)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
Performance after 10,000 optimization iterations
After 10,000 optimization iterations, the model has a classification accuracy on the test-set of about 99%.
Can you write Python code for it?

optimize(num_iterations=9000) # We performed 1000 iterations above.

print_test_accuracy(show_example_errors=True,
                    show_confusion_matrix=True)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
Performance after 1000 optimization iterations
After 1000 optimization iterations, the model has greatly increased its accuracy on the test-set to more than 90%.
Can you write Python code for it?

optimize(num_iterations=900) # We performed 100 iterations above.

print_test_accuracy(show_example_errors=True)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
PrettyTensor Implementation
This section shows how to make the exact same implementation of a Convolutional Neural Network using PrettyTensor.
The basic idea is to wrap the input tensor x_image in a PrettyTensor object which has helper-functions for adding new computational layers so as to create an entire Neural Network. This is a bit similar to the helper-functions we implemented above, but it is even simpler because PrettyTensor also keeps track of each layer's input and output dimensionalities, etc.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
TensorFlow Implementation
This section shows the original source-code from Tutorial #02 which implements the Convolutional Neural Network directly in TensorFlow. The code is not actually used in this Notebook and is only meant for easy comparison to the PrettyTensor implementation below.
The thing to note here is how many lines of code there are and the low-level details of how TensorFlow stores its data and performs the computation. It is easy to make mistakes even for fairly small Neural Networks.
Helper-functions
In the direct TensorFlow implementation, we first make some helper-functions which will be used several times in the graph construction.
These two functions create new variables in the TensorFlow graph that will be initialized with random values.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
TensorFlow Run
Create TensorFlow session
Once the TensorFlow graph has been created, we have to create a TensorFlow session which is used to execute the graph.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
The MNIST data-set has now been loaded and consists of 70,000 images and associated labels (i.e. classifications of the images). The data-set is split into 3 mutually exclusive sub-sets. We will only use the training and test-sets in this tutorial.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
The classification accuracy is calculated by first type-casting the vector of booleans to floats, so that False becomes 0 and True becomes 1, and then taking the average of these numbers.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
The following helper-function creates a fully-connected layer.
Can you write Python code for it?

def new_fc_layer(input,          # The previous layer.
                 num_inputs,     # Num. inputs from prev. layer.
                 num_outputs,    # Num. outputs.
                 use_relu=True): # Use Rectified Linear Unit (ReLU)?

    # Create new weights and biases.
    weights = new_weights(shape=[num_inputs, num_outputs])
    biases = new_biases(length=num_outputs)

    # Calculate the layer as the matrix multiplication of
    # the input and weights, and then add the bias-values.
    layer = tf.matmul(input, weights) + biases

    # Use ReLU?
    if use_relu:
        layer = tf.nn.relu(layer)

    return layer


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
The following helper-function creates a new convolutional network. The input and output are 4-dimensional tensors (aka. 4-rank tensors). Note the low-level details of the TensorFlow API, such as the shape of the weights-variable. It is easy to make a mistake somewhere which may result in strange error-messages that are difficult to debug.
Can you write Python code for it?

def new_conv_layer(input,              # The previous layer.
                   num_input_channels, # Num. channels in prev. layer.
                   filter_size,        # Width and height of filters.
                   num_filters,        # Number of filters.
                   use_pooling=True):  # Use 2x2 max-pooling.

    # Shape of the filter-weights for the convolution.
    # This format is determined by the TensorFlow API.
    shape = [filter_size, filter_size, num_input_channels, num_filters]

    # Create new weights aka. filters with the given shape.
    weights = new_weights(shape=shape)

    # Create new biases, one for each filter.
    biases = new_biases(length=num_filters)

    # Create the TensorFlow operation for convolution.
    # Note the strides are set to 1 in all dimensions.
    # The first and last stride must always be 1,
    # because the first is for the image-number and
    # the last is for the input-channel.
    # But e.g. strides=[1, 2, 2, 1] would mean that the filter
    # is moved 2 pixels across the x- and y-axis of the image.
    # The padding is set to 'SAME' which means the input image
    # is padded with zeroes so the size of the output is the same.
    layer = tf.nn.conv2d(input=input,
                         filter=weights,
                         strides=[1, 1, 1, 1],
                         padding='SAME')

    # Add the biases to the results of the convolution.
    # A bias-value is added to each filter-channel.
    layer += biases

    # Use pooling to down-sample the image resolution?
    if use_pooling:
        # This is 2x2 max-pooling, which means that we
        # consider 2x2 windows and select the largest value
        # in each window. Then we move 2 pixels to the next window.
        layer = tf.nn.max_pool(value=layer,
                               ksize=[1, 2, 2, 1],
                               strides=[1, 2, 2, 1],
                               padding='SAME')

    # Rectified Linear Unit (ReLU).
    # It calculates max(x, 0) for each input...


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
The following helper-function flattens a 4-dim tensor to 2-dim so we can add fully-connected layers after the convolutional layers.
Can you write Python code for it?

def flatten_layer(layer):
    # Get the shape of the input layer.
    layer_shape = layer.get_shape()

    # The shape of the input layer is assumed to be:
    # layer_shape == [num_images, img_height, img_width, num_channels]

    # The number of features is: img_height * img_width * num_channels
    # We can use a function from TensorFlow to calculate this.
    num_features = layer_shape[1:4].num_elements()

    # Reshape the layer to [num_images, num_features].
    # Note that we just set the size of the second dimension
    # to num_features and the size of the first dimension to -1
    # which means the size in that dimension is calculated
    # so the total size of the tensor is unchanged from the reshaping.
    layer_flat = tf.reshape(layer, [-1, num_features])

    # The shape of the flattened layer is now:
    # [num_images, img_height * img_width * num_channels]

    # Return both the flattened layer and the number of features.
    return layer_flat, num_features


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
The input image is processed in the first convolutional layer using the filter-weights. This results in 16 new images, one for each filter in the convolutional layer. The images are also down-sampled so the image resolution is decreased from 28x28 to 14x14.
These 16 smaller images are then processed in the second convolutional layer. We need filter-weights for each of these 16 channels, and we need filter-weights for each output channel of this layer. There are 36 output channels so there are a total of 16 x 36 = 576 filters in the second convolutional layer. The resulting images are down-sampled again to 7x7 pixels.
The output of the second convolutional layer is 36 images of 7x7 pixels each. These are then flattened to a single vector of length 7 x 7 x 36 = 1764, which is used as the input to a fully-connected layer with 128 neurons (or elements). This feeds into another fully-connected layer with 10 neurons, one for each of the classes, which is used to determine the class of the image, that is, which number is depicted in the image.
The convolutional filters are initially chosen at random, so the classification is done randomly. The error between the predicted and true class of the input image is measured as the so-called cross-entropy. The optimizer then automatically propagates this error back through the Convolutional Network using the chain-rule of differentiation and updates the filter-weights so as to improve the classification error. This is done iteratively thousands of times until the classification error is sufficiently low.
These particular filter-weights and intermediate images are the results of one optimization run and may look different if you re-run this Notebook.
Note that the computation in TensorFlow is actually done on a batch of images instead of a single image, which makes the computation more efficient. This means the flowchart actually has one more data-dimension when implemented in TensorFlow.
Imports
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np
from sklearn.metrics import confusion_matrix
import time
from datetime import timedelta
import math

# We also need PrettyTensor.
import prettytensor as pt


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
There are 16 input channels to the second convolutional layer, so we can make another 15 plots of filter-weights like this. We just make one more with the filter-weights for the second channel.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
Using this helper-function we can retrieve the variables. These are TensorFlow objects. In order to get the contents of the variables, you must do something like: contents = session.run(weights_conv1) as demonstrated further below.
Can you write Python code for it?

weights_conv1 = get_weights_variable(layer_name='layer_conv1')
weights_conv2 = get_weights_variable(layer_name='layer_conv2')


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/03_PrettyTensor.ipynb".
The first task is:
We could also have a placeholder variable for the class-number, but we will instead calculate it using argmax. Note that this is a TensorFlow operator so nothing is calculated at this point.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
Calculate the predicted class for the test-set.
Can you write Python code for it?

def predict_cls_test():
    return predict_cls(images = data.test.images,
                       labels = data.test.labels,
                       cls_true = data.test.cls)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
Create the directory if it does not exist.
Can you write Python code for it?

if not os.path.exists(save_dir):
    os.makedirs(save_dir)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
Data Dimensions
The data dimensions are used in several places in the source-code below. They are defined once so we can use these variables instead of numbers throughout the source-code below.
Can you write Python code for it?

# We know that MNIST images are 28 pixels in each dimension.
img_size = 28

# Images are stored in one-dimensional arrays of this length.
img_size_flat = img_size * img_size

# Tuple with height and width of images used to reshape arrays.
img_shape = (img_size, img_size)

# Number of colour channels for the images: 1 channel for gray-scale.
num_channels = 1

# Number of classes, one class for each of 10 digits.
num_classes = 10


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
Execute the function now to initialize the variables.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
Helper-function for plotting images
Function used to plot 9 images in a 3x3 grid, and writing the true and predicted classes below each image.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
Helper-function for showing the performance
Function for printing the classification accuracy on the test-set.
It takes a while to compute the classification for all the images in the test-set, that's why the results are re-used by calling the above functions directly from this function, so the classifications don't have to be recalculated by each function.
Can you write Python code for it?

def print_test_accuracy(show_example_errors=False,
                        show_confusion_matrix=False):

    # For all the images in the test-set,
    # calculate the predicted classes and whether they are correct.
    correct, cls_pred = predict_cls_test()

    # Classification accuracy and the number of correct classifications.
    acc, num_correct = cls_accuracy(correct)

    # Number of images being classified.
    num_images = len(correct)

    # Print the accuracy.
    msg = "Accuracy on Test-Set: {0:.1%} ({1} / {2})"
    print(msg.format(acc, num_correct, num_images))

    # Plot some examples of mis-classifications, if desired.
    if show_example_errors:
        print("Example errors:")
        plot_example_errors(cls_pred=cls_pred, correct=correct)

    # Plot the confusion matrix, if desired.
    if show_confusion_matrix:
        print("Confusion Matrix:")
        plot_confusion_matrix(cls_pred=cls_pred)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
Helper-function to perform optimization iterations
There are 55,000 images in the training-set. It takes a long time to calculate the gradient of the model using all these images. We therefore only use a small batch of images in each iteration of the optimizer.
If your computer crashes or becomes very slow because you run out of RAM, then you may try and lower this number, but you may then need to perform more optimization iterations.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
Helper-functions for the classification accuracy
This function calculates the classification accuracy given a boolean array whether each image was correctly classified. E.g. cls_accuracy([True, True, False, False, False]) = 2/5 = 0.4
Can you write Python code for it?

def cls_accuracy(correct):
    # Calculate the number of correctly classified images.
    # When summing a boolean array, False means 0 and True means 1.
    correct_sum = correct.sum()

    # Classification accuracy is the number of correctly classified
    # images divided by the total number of images in the test-set.
    acc = float(correct_sum) / len(correct)

    return acc, correct_sum


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
Imports
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np
from sklearn.metrics import confusion_matrix
import time
from datetime import timedelta
import math
import os

# Use PrettyTensor to simplify Neural Network construction.
import prettytensor as pt


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
Initialize variables
The variables for weights and biases must be initialized before we start optimizing them. We make a simple wrapper-function for this, because we will call it again below.
Can you write Python code for it?

def init_variables():
    session.run(tf.initialize_all_variables())


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
Now that we have wrapped the input image in a Pretty Tensor object, we can add the convolutional and fully-connected layers in just a few lines of source-code.
Note that pt.defaults_scope(activation_fn=tf.nn.relu) makes activation_fn=tf.nn.relu an argument for each of the layers constructed inside the with-block, so that Rectified Linear Units (ReLU) are used for each of these layers. The defaults_scope makes it easy to change arguments for all of the layers.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
Performance before any optimization
The accuracy on the test-set is very low because the model variables have only been initialized and not optimized at all, so it just classifies the images randomly.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
Restore Best Variables
Re-load all the variables that were saved to file during optimization.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
TensorFlow Graph
The entire purpose of TensorFlow is to have a so-called computational graph that can be executed much more efficiently than if the same calculations were to be performed directly in Python. TensorFlow can be more efficient than NumPy because TensorFlow knows the entire computation graph that must be executed, while NumPy only knows the computation of a single mathematical operation at a time.
TensorFlow can also automatically calculate the gradients that are needed to optimize the variables of the graph so as to make the model perform better. This is because the graph is a combination of simple mathematical expressions so the gradient of the entire graph can be calculated using the chain-rule for derivatives.
TensorFlow can also take advantage of multi-core CPUs as well as GPUs - and Google has even built special chips just for TensorFlow which are called TPUs (Tensor Processing Units) and are even faster than GPUs.
A TensorFlow graph consists of the following parts which will be detailed below:

Placeholder variables used for inputting data to the graph.
Variables that are going to be optimized so as to make the convolutional network perform better.
The mathematical formulas for the convolutional network.
A loss measure that can be used to guide the optimization of the variables.
An optimization method which updates the variables.

In addition, the TensorFlow graph may also contain various debugging statements e.g. for logging data to be displayed using TensorBoard, which is not covered in this tutorial.
Placeholder variables
Placeholder variables serve as the input to the TensorFlow computational graph that we may change each time we execute the graph. We call this feeding the placeholder variables and it is demonstrated further below.
First we define the placeholder variable for the input images. This allows us to change the images that are input to the TensorFlow graph. This is a so-called tensor, which just means that it is a multi-dimensional array. The data-type is set to float32 and...
Can you write Python code for it?

x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='x')


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
The class-labels are One-Hot encoded, which means that each label is a vector with 10 elements, all of which are zero except for one element. The index of this one element is the class-number, that is, the digit shown in the associated image. We also need the class-numbers as integers for the test- and validation-sets, so we calculate them now.
Can you write Python code for it?

data.test.cls = np.argmax(data.test.labels, axis=1)
data.validation.cls = np.argmax(data.validation.labels, axis=1)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
The classification accuracy is calculated by first type-casting the vector of booleans to floats, so that False becomes 0 and True becomes 1, and then taking the average of these numbers.
Can you write Python code for it?

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
The classification accuracy is high again when using the variables that were previously saved.
Note that the classification accuracy may be slightly higher or lower than that reported above, because the variables in the file were chosen to maximize the classification accuracy on the validation-set, but the optimization actually continued for another 1000 iterations after saving those variables, so we are reporting the results for two slightly different sets of variables. Sometimes this leads to slightly better or worse performance on the test-set.
Can you write Python code for it?

print_test_accuracy(show_example_errors=True,
                    show_confusion_matrix=True)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
The convolutional layers expect x to be encoded as a 4-dim tensor so we have to reshape it so its shape is instead [num_images, img_height, img_width, num_channels]. Note that img_height == img_width == img_size and num_images can be inferred automatically by using -1 for the size of the first dimension. So the reshape operation is:
Can you write Python code for it?

x_image = tf.reshape(x, [-1, img_size, img_size, num_channels])


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
Then we create a vector of booleans telling us whether the predicted class equals the true class of each image.
Can you write Python code for it?

correct_prediction = tf.equal(y_pred_cls, y_true_cls)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
This was developed using Python 3.5.2 (Anaconda) and TensorFlow version:
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/04_Save_Restore.ipynb".
The first task is:
We could also have a placeholder variable for the class-number, but we will instead calculate it using argmax. Note that this is a TensorFlow operator so nothing is calculated at this point.
Can you write Python code for it?

y_true_cls = tf.argmax(y_true, dimension=1)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Best neural network
Now we find the single neural network that performed best on the test-set.
First list the classification accuracies on the test-set for all the neural networks in the ensemble.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Boolean array whether each image in the test-set was correctly classified by the best single network and incorrectly classified by the ensemble.
Can you write Python code for it?

best_net_better = np.logical_and(best_net_correct,
                                 ensemble_incorrect)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Boolean array whether each image in the test-set was correctly classified by the ensemble and incorrectly classified by the best neural network.
Can you write Python code for it?

ensemble_better = np.logical_and(best_net_incorrect,
                                 ensemble_correct)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Boolean array whether each of the images in the test-set was correctly classified by the ensemble of neural networks.
Can you write Python code for it?

ensemble_correct = (ensemble_cls_pred == data.test.cls)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Boolean array whether the best neural network classified each image in the test-set correctly.
Can you write Python code for it?

best_net_correct = (best_net_cls_pred == data.test.cls)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Calculate a boolean array whether the images in the test-set are classified correctly.
Can you write Python code for it?

def test_correct():
    return correct_prediction(images = data.test.images,
                              labels = data.test.labels,
                              cls_true = data.test.cls)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Calculate the classification accuracy on the test-set.
Can you write Python code for it?

def test_accuracy():
    # Get the array of booleans whether the classifications are correct
    # for the test-set.
    correct = test_correct()

    # Calculate the classification accuracy and return it.
    return classification_accuracy(correct)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Check that the shape of the combined arrays is correct.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Close TensorFlow Session
We are now done using TensorFlow, so we close the session to release its resources.
Can you write Python code for it?

# This has been commented out in case you want to modify and experiment
# with the Notebook without having to restart it.
# session.close()


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Comparison of ensemble vs. the best single network
The number of images in the test-set that were correctly classified by the ensemble.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Create the directory if it does not exist.
Can you write Python code for it?

if not os.path.exists(save_dir):
    os.makedirs(save_dir)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Data Dimensions
The data dimensions are used in several places in the source-code below. They are defined once so we can use these variables instead of numbers throughout the source-code below.
Can you write Python code for it?

# We know that MNIST images are 28 pixels in each dimension.
img_size = 28

# Images are stored in one-dimensional arrays of this length.
img_size_flat = img_size * img_size

# Tuple with height and width of images used to reshape arrays.
img_shape = (img_size, img_size)

# Number of colour channels for the images: 1 channel for gray-scale.
num_channels = 1

# Number of classes, one class for each of 10 digits.
num_classes = 10


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Ensemble predictions
There are different ways to calculate the predicted labels for the ensemble. One way is to calculate the predicted class-number for each neural network, and then select the class-number with most votes. But this requires a large number of neural networks relative to the number of classes.
The method used here is instead to take the average of the predicted labels for all the networks in the ensemble. This is simple to calculate and does not require a large number of networks in the ensemble.
Can you write Python code for it?

ensemble_pred_labels = np.mean(pred_labels, axis=0)
ensemble_pred_labels.shape


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Examples: Ensemble is better than the best network
Plot examples of images that were correctly classified by the ensemble and incorrectly classified by the best single network.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Function for printing the predicted labels for the best single network.
Can you write Python code for it?

def print_labels_best_net(idx, **kwargs):
    print_labels(labels=best_net_pred_labels, idx=idx, **kwargs)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Function for selecting a random training-batch of the given size.
Can you write Python code for it?

def random_batch(x_train, y_train):
    # Total number of images in the training-set.
    num_images = len(x_train)

    # Create a random index into the training-set.
    idx = np.random.choice(num_images,
                           size=train_batch_size,
                           replace=False)

    # Use the random index to select random images and labels.
    x_batch = x_train[idx, :]  # Images.
    y_batch = y_train[idx, :]  # Labels.

    # Return the batch.
    return x_batch, y_batch


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Helper-function for creating random training-sets
We will train 5 neural networks on different training-sets that are selected at random. First we combine the original training- and validation-sets into one big set. This is done for both the images and the labels.
Can you write Python code for it?

combined_images = np.concatenate([data.train.images, data.validation.images], axis=0)
combined_labels = np.concatenate([data.train.labels, data.validation.labels], axis=0)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Helper-function for splitting the combined data-set into a random training- and validation-set.
Can you write Python code for it?

def random_training_set():
    # Create a randomized index into the full / combined training-set.
    idx = np.random.permutation(combined_size)

    # Split the random index into training- and validation-sets.
    idx_train = idx[0:train_size]
    idx_validation = idx[train_size:]

    # Select the images and labels for the new training-set.
    x_train = combined_images[idx_train, :]
    y_train = combined_labels[idx_train, :]

    # Select the images and labels for the new validation-set.
    x_validation = combined_images[idx_validation, :]
    y_validation = combined_labels[idx_validation, :]

    # Return the new training- and validation-sets.
    return x_train, y_train, x_validation, y_validation


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Helper-function to create a random training batch.
There are thousands of images in the training-set. It takes a long time to calculate the gradient of the model using all these images. We therefore only use a small batch of images in each iteration of the optimizer.
If your computer crashes or becomes very slow because you run out of RAM, then you may try and lower this number, but you may then need to perform more optimization iterations.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Helper-functions for calculating and predicting classifications
This function calculates the predicted labels of images, that is, for each image it calculates a vector of length 10 indicating which of the 10 classes the image is.
The calculation is done in batches because it might use too much RAM otherwise. If your computer crashes then you can try and lower the batch-size.
Can you write Python code for it?

# Split the data-set in batches of this size to limit RAM usage.
batch_size = 256

def predict_labels(images):
    # Number of images.
    num_images = len(images)

    # Allocate an array for the predicted labels which
    # will be calculated in batches and filled into this array.
    pred_labels = np.zeros(shape=(num_images, num_classes),
                           dtype=np.float)

    # Now calculate the predicted labels for the batches.
    # We will just iterate through all the batches.
    # There might be a more clever and Pythonic way of doing this.

    # The starting index for the next batch is denoted i.
    i = 0

    while i < num_images:
        # The ending index for the next batch is denoted j.
        j = min(i + batch_size, num_images)

        # Create a feed-dict with the images between index i and j.
        feed_dict = {x: images[i:j, :]}

        # Calculate the predicted labels using TensorFlow.
        pred_labels[i:j] = session.run(y_pred, feed_dict=feed_dict)

        # Set the start-index for the next batch to the
        # end-index of the current batch.
        i = j

    return pred_labels


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Helper-functions for plotting and printing comparisons
Function for plotting images from the test-set and their true and predicted class-numbers.
Can you write Python code for it?

def plot_images_comparison(idx):
    plot_images(images=data.test.images[idx, :],
                cls_true=data.test.cls[idx],
                ensemble_cls_pred=ensemble_cls_pred[idx],
                best_cls_pred=best_net_cls_pred[idx])


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Imports
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np
from sklearn.metrics import confusion_matrix
import time
from datetime import timedelta
import math
import os

# Use PrettyTensor to simplify Neural Network construction.
import prettytensor as pt


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Load Data
The MNIST data-set is about 12 MB and will be downloaded automatically if it is not located in the given path.
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
data = input_data.read_data_sets('data/MNIST/', one_hot=True)


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Negate the boolean array so we can use it to lookup incorrectly classified images.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Number of images in the test-set where the best single network was better than the ensemble.
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Number of images in the test-set where the ensemble was better than the best single network:
Can you write Python code for it?


I am working on the file "Exercises/Term1/TensorFlow-Tutorials/05_Ensemble_Learning.ipynb".
The first task is:
Number of optimization iterations for each neural network.
Can you write Python code for it?