I am working on the file "02-Classification/TensorFlow introduction.ipynb".
The first task is:
CNN classifier for the MNIST dataset
We will revisit the neural network built to recognise handwritten digits, this time using Tensorflow.
We can see how to build, compile and fit a convolutional neural network (CNN) model to the MNIST dataset of images of handwritten digits.
The MNIST dataset
The MNIST dataset consists of a training set of 60,000 handwritten digits with corresponding labels, and a test set of 10,000 images. The images have been normalised and centred. The dataset is frequently used in machine learning research, and has become a standard benchmark for image classification models.

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. "Gradient-based learning applied to document recognition." Proceedings of the IEEE, 86(11):2278-2324, November 1998.

And is available inside Tensorflow!
The goal is to construct a neural network that classifies images of handwritten digits into one of 10 classes.
The Keras API
In TensorFlow 2 a big
drive is being towards ease of use. At the highest level,
the easiest way to start building deep learning models in TensorFlow
is to use the Keras API, and that's what we're going
to focus on.
The Keras Projects was
authored by Francois Chollet and it's a high-level neural
networks API developed with the specific goal of
making it easy to build and infer from
deep learning models. It was developed
as an API without the backend and actually supports multiple
backend implementations. In TensorFlow 2, it's been
adopted as the high level API. Despite being high
level and easy to use, it's still very flexible; probably 99 percent of whatever you want to do, you can achieve it
using the Keras API.
You can visit the Keras
homepage, where can read more about the Keras Project and its
development principles.
When you install TensorFlow 2, you get all of this packaged
up as part of the library in the namespace,
TensorFlow.Keras.
Load the data
Can you write Python code for it?

# load the MNIST data directly from Tensorflow Keras

mnist_data = tf.keras.datasets.mnist
(train_images, train_labels), (test_images, test_labels) = mnist_data.load_data()

len(train_images) # train and test datasets are numpy arrays

len(test_images)


I am working on the file "02-Classification/TensorFlow introduction.ipynb".
The first task is:
Evaluate the model
Now we are going to see how well our network has learned by evaluating its performance on a
held out test set. We are also going to be
putting our network into action by getting its
predictions on unseen input data.
Running model.evaluate
will iterate over the test set and calculate the loss and the metrics
on that test set. So for example, if we had defined cross-entropy
loss function when we compiled the model, as well as an accuracy metric, then this loss function and this metric will be evaluated on the test set and returned
by the model.evaluate call. If we had included more metrics then they would all
just be returned by model.evaluate. Hopefully, if our model
has trained well, then these loss or
metric values won't be too far off what we achieved
on the training set.
Can you write Python code for it?

test_loss, test_accuracy = CNNModel.evaluate(scaled_test_images, test_labels, verbose=2)

print(f"Test loss: {test_loss}")
print(f"Test accuracy: {test_accuracy}")


I am working on the file "02-Classification/TensorFlow introduction.ipynb".
The first task is:
First CNN model
We start simple, like our previous example. Our neural network has 3 layers: an input layer, a hidden layer and an output layer.
You'll find that most of the neural networks
that you work with, can be built using
the basic Keras classes such as the sequential class.
It's a really easy
and intuitive way to construct deep
learning models.
The way we build the
model is really simple: we just create an instance
of the sequential class and then we pass in a
list of Keras layers.
Layers are the buidling blocks of neural networks in Keras. https://keras.io/api/layers/
The Sequential class could be used to build simple neural
networks, feed-forward networks which are also called
Multilayer Perceptrons or MLPs.
But here we'll see how to build
convolutional neural networks by including convolutional and
pooling layers in our model.
Convolutional Neural Network
In deep learning, a convolutional neural network (CNN, or ConvNet) is a class of deep neural networks, most commonly applied to analysing visual imagery
The name “convolutional neural network” indicates that the network employs a mathematical operation called convolution. CNNs use convolution in place of general matrix multiplication in at least one of their layers.
The role of the ConvNet is to reduce the images into a form which is easier to process, without losing features which are critical for getting a good prediction.
The objective of the Convolution Operation is to extract the high-level features such as edges, from the input image.
Input: the convolutional layer
In a CNN, the input is a tensor with shape (number of images) x (image height) x (image width) x (input channels). After passing through a convolutional layer, the image becomes abstracted to a feature map.
The pooling layer
Convolutional networks may include local or global pooling layers to streamline the underlying computation. Pooling layers reduce the dimensions of the data by combining the outputs of neuron clusters at one layer into a single neuron in the next...
Can you write Python code for it?

from tensorflow.keras.models import Sequential # Sequential model
from tensorflow.keras.layers import Dense, Flatten, Conv2D, MaxPooling2D # layers

# Build the Sequential convolutional neural network model

CNNModel = Sequential([
    Conv2D(filters = 8, kernel_size=(3,3), padding= 'SAME', activation='relu',
           name = 'Convolution', input_shape=scaled_train_images[0].shape),
    MaxPooling2D((2,2), name = 'Pooling'),
    Flatten(name = 'InputFlattening'),
    Dense(25, activation='relu', name = 'hiddenLayer'),
    Dense(10, activation='softmax', name='outputLayer')])


I am working on the file "02-Classification/TensorFlow introduction.ipynb".
The first task is:
Fit the model to the training data
Once the neural
network has been built as sequential models and the optimizer loss function and
metrics have been set using the compile method, we have got everything we need to
start training the model on data.
We just need to call model.fit and pass in the training inputs and the labels. They are
 Numpy arrays, where all of the dataset
inputs have been stacked together
into a single array.
Passing these arrays
into model.fit, we will then train the model for one pass through the
training data or one epoch. You'll probably want
to train a model for more than just
one epoch, though, and here we pass in the optional keyword
argument epochs equals 5. So the training will make five complete passes
through the dataset.
Another option to
mention is the batch size. By default, the
batch size is set to 32 whenever you pass in the dataset inputs and outputs as single Numpy arrays
as we are doing here. If you want to change that to
use a different batch size, you can just use the
batch_size argument.
Finally, an important
point is that calling model.fit actually
returns something: a TensorFlow history object. This object contains a record of the progress of
the network during training in terms of the loss and the metrics that we defined
when we compiled the model.
Can you write Python code for it?

#  Train the model for 5 epochs on the scaled_train_images and train_labels.
#  Save the training history, as returned by model.fit.

train_history = CNNModel.fit(scaled_train_images, train_labels, epochs=5, batch_size=256)


I am working on the file "02-Classification/TensorFlow introduction.ipynb".
The first task is:
Images are 28 x 28 pixels.
Preprocess the data
First, let's preprocess the data by scaling the training and test images so their values lie in the range from 0 to 1.
This is all we need, the one hot encoding of the target variable is already done in the dataset.
Can you write Python code for it?

import numpy as np

# takes in the training and test images as loaded in the cell above, and scales them
# so that they have minimum and maximum values equal to 0 and 1 respectively.

scaled_train_images = train_images / np.amax(train_images)
scaled_test_images  = test_images / np.amax(test_images)

# Add a dummy channel dimension (needed for convolutional layer)

scaled_train_images = scaled_train_images[..., np.newaxis]
scaled_test_images = scaled_test_images[..., np.newaxis]

scaled_train_images[0].shape


I am working on the file "02-Classification/TensorFlow introduction.ipynb".
The first task is:
In this model, I'm importing four layer types. You can see that I'm making the first layer a convolution together with a pooling and the flatten layer.
This layer creates a convolution kernel that is convolved with the layer input to produce a tensor of outputs. It has two required arguments: the first is the the dimensionality of the output space (i.e. the number of output filters in the convolution) and we define 8 filters in this layer.
And the second is the shape
of the convolutional kernel: an integer or tuple/list of 2 integers, specifying the height and width of the 2D convolution window.
Why is it that you need an input shape of 28
times 28 times 1, and not just 28 times 28? Well, you can try to see what happens when you remove that one. It in deed seems that
an error pops up. This is due to the
input requirements of the conv2d layer, since that one is necessary and indicates the number of
channels in our input.
One thing you might have
noticed is that we explicitly tell the model what shape the input data
will be at the build stage.
I'm giving explicitly
the input shape to the first layer in my model so that the weights and biases are created when I define
the model instance.
Now in this case, the weights and biases will be created and initialized
straight away.
Alternatively, you can not saying what size or shape
the input is going to be and leave the creation of weights and biases until
the training stage when you feed the data into the model.
So, our kernel is
convolving over the input image with a window of 3 by 3
pixels using 8 filters. We're also passing the output
through a relu activation function.
Finally, the SAME padding just makes sure that the spatial dimensions don't change.
The second layer is a MaxPooling layer. This layer has one required argument,
which is the pooling window size. And that's also 3 by 3 pixels.
The
input shape is two-dimensional and that's why I'm flattening (unrolling) each data
input to unroll it into a long one-dimensional
vector ready to be...
Can you write Python code for it?

# Print the model summary

CNNModel.summary()


I am working on the file "02-Classification/TensorFlow introduction.ipynb".
The first task is:
Note how the accuracy improved and the loss decreased after each epoch.
We can see that better in a couple of charts:
Plot the learning curves
We will now plot two graphs:
* Epoch vs accuracy
* Epoch vs loss
We will load the model history into a pandas DataFrame and use the plot method to output the required graphs.
Can you write Python code for it?

import pandas as pd

# load the model history into a pandas DataFrame

historyFrame = pd.DataFrame(train_history.history)
historyFrame.head

# Accuracy vs Epochs plot

acc_plot = historyFrame.plot(y="accuracy", title="Accuracy vs Epochs", legend=False)
acc_plot.set(xlabel="Epochs", ylabel="Accuracy");

# Loss vs Epochs plot

acc_plot = historyFrame.plot(y="loss", title = "Loss vs Epochs",legend=False)
acc_plot.set(xlabel="Epochs", ylabel="Loss");


I am working on the file "02-Classification/TensorFlow introduction.ipynb".
The first task is:
The array of output
predictions is a two-dimensional
array, the first dimension will be four, which matches the number of
examples we're getting predictions for and
the second dimension will be ten for the
number of classes.
This is because the network is a multi-class
classification model where our final layer has ten neurons and
softmax activation. And
the loss-function is a
categorical cross entropy.
Each row of this array is the network output
for that data input. The output layer is a
softmax layer, remember, so each layer is a set of
output probabilities emitted by the softmax function and you can see how they add up
to one in each case. Let's see which class the
model is predicting for each example.
Can you write Python code for it?

fig, axes = plt.subplots(4, 2, figsize=(16, 12))
fig.subplots_adjust(hspace=0.4, wspace=-0.2)

for i, (prediction, image, label) in enumerate(zip(predictions, random_test_images, random_test_labels)):
    axes[i, 0].imshow(np.squeeze(image))
    axes[i, 0].get_xaxis().set_visible(False)
    axes[i, 0].get_yaxis().set_visible(False)
    axes[i, 0].text(10., -1.5, f'Digit {label}')
    axes[i, 1].bar(np.arange(len(prediction)), prediction)
    axes[i, 1].set_xticks(np.arange(len(prediction)))
    axes[i, 1].set_title(f"Categorical distribution. Model prediction: {np.argmax(prediction)}")

plt.show()


I am working on the file "02-Classification/TensorFlow introduction.ipynb".
The first task is:
This is what is defining the loss function
and the optimizer for the network. And as you can see,
we're defining the loss function to be the sparse categorical crossentropy which makes
sense for this network and task.  All the labels have
a sparse representation, so just a single
integer for each label, and train_labels is a one-dimensional array with length equal to the
number of samples.
And the optimizer is Adam. These options are being passed
into the compile method of our sequential model object. We can also optionally define a set of
metrics that we want to keep track of as the model is training. These metrics will be calculated for
each epoch of training along with the evaluation of
the loss function on the training data. We'll see this in action later when
we start training some models.
Here we're just passing in a list
with one metric that is the accuracy. Of course, there are lots of options for
each one of these keyword arguments. For example, we could instead choose
the sgd or
stochastic gradient descent optimizer or rmsprop or adadelta. For the loss function, you could choose
the mean squared error although that might be more appropriate for a regression task, or the binary cross entropy.
In the list of metrics, you could add more, e.g. the mean absolute error.
One thing that you'll be noticing is that
Keras is giving us a really nice usable interface for setting up these models with a lot of readable strings
that we can pass in to many of the options. It's definitely worth knowing though that
each of these strings is a reference to another object or function and we can always use that object or
function directly. For example, instead of passing in the string Adam,
I can directly pass in an object that comes from
the tf.keras.optimizers module. The reason why you might want to do
this is because it gives you greater flexibility as many of these objects
themselves have options that you might want to have control over.
Can you write Python code for it?

# Print the resulting model attributes

print(CNNModel.loss)
print(CNNModel.optimizer)
print(CNNModel.metrics)
print(CNNModel.optimizer.lr)


I am working on the file "02-Classification/TensorFlow regularisation .ipynb".
The first task is:
Batch normalisation can be applied to all layers, not only before the last one, and this usually improves the performance:
Can you write Python code for it?

modelBN2 = Sequential([
        Dense(128, kernel_regularizer = regularizers.l2(wd), activation="relu", input_shape=(train_data.shape[1],)),
        BatchNormalization(),
        Dropout(rate),
        Dense(128, kernel_regularizer = regularizers.l2(wd), activation="relu"),
        BatchNormalization(),
        Dropout(rate),
        Dense(128, kernel_regularizer = regularizers.l2(wd), activation="relu"),
        BatchNormalization(),
        Dropout(rate),
        Dense(128, kernel_regularizer = regularizers.l2(wd), activation="relu"),
        BatchNormalization(),
        Dropout(rate),
        Dense(128, kernel_regularizer = regularizers.l2(wd), activation="relu"),
        BatchNormalization(),
        Dropout(rate),
        Dense(128, kernel_regularizer = regularizers.l2(wd), activation="relu"),
        BatchNormalization(
            momentum=0.95,
            epsilon=0.005,
            beta_initializer=RandomNormal(mean=0.0, stddev=0.05),
            gamma_initializer=Constant(value=0.9)),
        Dense(1)
])

# Compile the model

modelBN2.compile(optimizer='adam',
              loss='mse',
              metrics=['mae'])

# Train the model

trainingHistoryBN2 = modelBN2.fit(train_data, train_targets, epochs=100, validation_split=0.15, batch_size=64,
                                verbose=False)

# Evaluate the model on the test set

lossBN2, maeBN2 = modelBN2.evaluate(test_data, test_targets, verbose=False)
print(f"Loss is {lossBN2}\nMAE is {maeBN2}")


I am working on the file "02-Classification/TensorFlow regularisation .ipynb".
The first task is:
Evaluate the model on the test set
Can you write Python code for it?

lossU, maeU = firstModel.evaluate(test_data, test_targets, verbose=False)
print(f"Loss is {lossU}\nMAE is {maeU}")


I am working on the file "02-Classification/TensorFlow regularisation .ipynb".
The first task is:
First model: Train a feedforward neural network model
In Deep Learning there are no hard & fast rules to set the number of layers, the number of hidden units per layer and not even the kind of connections between layers.
It is a lot about try and improve.
We start with a "simple" sequential model with 6 layers plus one output layer...
Can you write Python code for it?

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

# Build the model

firstModel = Sequential([
        Dense(128, activation = "relu", input_shape=(train_data.shape[1],)),
        Dense(128, activation = "relu"),
        Dense(128, activation = "relu"),
        Dense(128, activation = "relu"),
        Dense(128, activation = "relu"),
        Dense(128, activation = "relu"),
        Dense(1)
    ])

# Print the model summary

firstModel.summary()


I am working on the file "02-Classification/TensorFlow regularisation .ipynb".
The first task is:
In the previous post we have seen that the model.fit method returns the
history object and it records the training set loss and metrics over the course
of the model training; now this history object also records the performance
on the validation set.
If we were to take a
look at this dictionary, would see that it
now not only has keys for the training set
loss and metric values, but also for the validation
loss of metric values.
Can you write Python code for it?


I am working on the file "02-Classification/TensorFlow regularisation .ipynb".
The first task is:
Introduction to callbacks
But even now we're monitoring the performance of our model
on the validation set, we're still having to fix
the number of epochs that the model is training
for ahead of time. What we would like
to do is to have the ability to not only monitor the performance
of the network, but also perform certain actions depending on those
performance measures.
That's where callbacks come in. Callbacks are an
important type of object TensorFlow and Keras that
are exactly designed to be able to monitor the loss in metrics at certain points in
the training run and perform some
action.
In TensorFlow, all callbacks are stored in the tensorflow.keras.callbacks
module. Inside that module, there's
a base class called callback which all other
callbacks inherit from. You can also subclass
the callback based class yourself to create
your own callbacks.
For example, one
of the methods of the callback base class
is called on_train_begin.
So we can override this method in our new callback to perform some action at the beginning
of the training run. Similarly, there
are more methods of the base class that are called at other points in the training, for example after each Epoch.
Example training callback
Can you write Python code for it?

# Write a custom callback

from tensorflow.keras.callbacks import Callback

class TrainingCallback(Callback):

    def on_train_begin(self, logs=None):
        print("Starting Training ...")

            # Print the loss and mean absolute error after each epoch
    def on_epoch_end(self, epoch, logs=None):
        print('Epoch {}: Average loss is {:7.2f}, mean absolute error is {:7.2f}.'.format(epoch, logs['loss'], logs['mae']))

    def on_train_end(self, logs=None):
        print("Finished training!")

# Train the model, with some of the data reserved for validation

regularisedModel.fit(train_data, train_targets, epochs = 3, batch_size=128, verbose=False, callbacks=[TrainingCallback()])


I am working on the file "02-Classification/TensorFlow regularisation .ipynb".
The first task is:
Let's now compile and fit our model with batch normalisation, and track the progress on training and validation sets.
Can you write Python code for it?

# Compile the model

modelBN.compile(optimizer='adam',
              loss='mse',
              metrics=['mae'])

# Train the model

trainingHistoryBN = modelBN.fit(train_data, train_targets, epochs=100, validation_split=0.15, batch_size=64,
                                verbose=False)

# Evaluate the model on the test set

lossBN, maeBN = modelBN.evaluate(test_data, test_targets, verbose=False)
print(f"Loss is {lossBN}\nMAE is {maeBN}")

# Plot the training and validation loss

plotMetricsByEpoch(trainingHistoryBN.history)


I am working on the file "02-Classification/TensorFlow regularisation .ipynb".
The first task is:
Let's say then that we want
to track the performance of our model on a separate
held-out validation set. One way we can do
this is to pass in the validation split
keyword argument. What this does is to
automatically split the inputs in the targets into separate training
and validation sets.
The 0.15 you see here means that 15 percent of the data will
be held back for validation. The model's performance
is then recorded on both the training
and validation sets.
It is also possible to use the validation
data keyword argument in the fit function if you have already a separate validation set (or you can build it tyourself using the sklearn split functon, as done above for the test dataset).
Can you write Python code for it?

# Train the model, with some of the data reserved for validation (15%)

firstTrainingHistory = firstModel.fit(train_data, train_targets, epochs = 100, validation_split = 0.15,
                    batch_size = 64, verbose = False)


I am working on the file "02-Classification/TensorFlow regularisation .ipynb".
The first task is:
Load and pre-process the data
The diabetes dataset is available directly from SKlearn:
Can you write Python code for it?

# Load the diabetes dataset

from sklearn.datasets import load_diabetes

diabetes_datasets = load_diabetes()
print(diabetes_datasets["DESCR"])

# Save the input and target variables

print(diabetes_datasets.keys())

data = diabetes_datasets["data"]
targets = diabetes_datasets["target"]

# The ten features are already normalised.
# Normalise the target data (this will make clearer training curves)

targets = (targets - targets.mean(axis = 0)) / targets.std()


I am working on the file "02-Classification/TensorFlow regularisation .ipynb".
The first task is:
Loss = 0.45
Loss improved quite a lot and we can see it also in the plots, now the results on the validation dataset are much more similar to the ones on the train dataset:
Can you write Python code for it?

# Plot the training and validation loss

plotMetricsByEpoch(trainingHistoryR2.history)


I am working on the file "02-Classification/TensorFlow regularisation .ipynb".
The first task is:
One interesting outcome from the curves is that the metrics on the validation dataset (the orange one) is not improving at all after the first epochs. It even diverges.
A clear sign of overfitting. Let's regularise!
Model regularisation
In order to improve the performance of the model, we use different regularization techniques. There are several techniques, but we will discuss these main techniques:

L1/L2 Regularization
Dropout
Batch Normalization
Early Stopping

Adding regularisation with weight decay
We start to look at using L2 weight regularization, which is also known
as weight decay in a context of neural networks.
Dense layers as well as
convolutional layers have an optional kernel_regularizer keyword argument to add in weights decay
or L1 regularization. We set the kernel
regularizer argument equal to
tf.keras.regularizes.L2 object. This object is created with
one required arguments, which is the coefficient
that multiplies the sum of squared
weights in this layer.
Remember that a dense layer
has weights and biases, and the weight matrix is
sometimes called the kernel. Here we set the weight
decay coefficients to be equal to 0.00001. Once we've built the model together with the weight decay, we can compile and fit
the model as normal.
The weight decay penalty
term is automatically added to the loss-function
when we compile the model.
This sum of squared
weights term has the effect of penalizing
large values of the weights, which encourages
the model to find the simpler function
that fits the data. This means the model is less likely to overfit to
the training set.
We can use L1 regularization
in a very similar way(L1 regularization has the effect of pacifying the network weights. Or in other words, it results in some of those weights
being set to zero). In fact, there's no reason
why you couldn't use both L1 and L2 regularization, there is a tf.keras.regularizers.l1l2 object, and pass that to the kernel
regularizer argument.
It's quite typical to
apply regularization to the weight matrix or the...
Can you write Python code for it?

from tensorflow.keras import regularizers

# we define a model that is the same as the previous one, only regularised
# wd is the weight decay
wd = 1e-5

regularisedModel = Sequential([
        Dense(128, kernel_regularizer = regularizers.l2(wd), activation="relu", input_shape=(train_data.shape[1],)),
        Dense(128, kernel_regularizer = regularizers.l2(wd), activation="relu"),
        Dense(128, kernel_regularizer = regularizers.l2(wd), activation="relu"),
        Dense(128, kernel_regularizer = regularizers.l2(wd), activation="relu"),
        Dense(128, kernel_regularizer = regularizers.l2(wd), activation="relu"),
        Dense(128, kernel_regularizer = regularizers.l2(wd), activation="relu"),
        Dense(1)])

# Compile the model

regularisedModel.compile(optmizer="adam", loss="mse", metrics=["mae"])

# Train the model, with some of the data reserved for validation

trainingHistoryR1 = regularisedModel.fit(train_data, train_targets, epochs = 100,
                   validation_split = 0.15, batch_size = 64, verbose=False)

plotMetricsByEpoch(trainingHistoryR1.history)

# Evaluate the model on the test set

lossR1, maeR1 = regularisedModel.evaluate(test_data, test_targets, verbose=False)
print(f"Loss is {lossR1}\nMAE is {maeR1}")


I am working on the file "02-Classification/TensorFlow regularisation .ipynb".
The first task is:
Plot the learning curves
Can you write Python code for it?

# Plot the training and validation loss

plotMetricsByEpoch(reg_history.history)


# Plot the training and validation loss

plotMetricsByEpoch(trainingHistoryBN2.history)


# Plot the training and validation loss

plotMetricsByEpoch(new_history.history)


I am working on the file "02-Classification/TensorFlow regularisation .ipynb".
The first task is:
The loss improved to 0.55 but the results on the validation dataset shows it's still overfitting.
Another common way to avoid regularization is by using the Dropout technique. The main idea behind using dropout is that we randomly turn off some neurons in our layer based on some probability.
In order to implement DropOut, all we have to do is to add a Dropout layer from tf.keras.layers and set a dropout rate in it.
The Dropouts layer has
a required arguments, which is the Dropouts rate. Here the rate has been set to 0.3 and that means that
each weight connection between these two dense layers is set to zero with probability 0.3
Each of the weights are
randomly dropped out independently from
one another and Dropout has also
applied independently across each element in the
batch at training time. Again, we can compile and
fit our model as normal, with Dropouts layers
included in the model.
When we're using Dropouts, we typically have
two different modes for how we run the network. At training time, we randomly Dropout
weights and we can think
of this as Training mode. However, when we're evaluating our model or making
predictions from it, we stopped randomly
dropping out the weights and we can think of
this as testing mode. These two modes are automatically handled behind the
scenes by the model.fit, .evaluate, and .predict methods.
Can you write Python code for it?

from tensorflow.keras.layers import Dropout

# we define a function to build a regularised model that we can call later to tune the parameters
# (weight D ... and dropout rate)

def getRegularisedModel(wd, rate):
    model = Sequential([
        Dense(128, kernel_regularizer = regularizers.l2(wd), activation="relu", input_shape=(train_data.shape[1],)),
        Dropout(rate),
        Dense(128, kernel_regularizer = regularizers.l2(wd), activation="relu"),
        Dropout(rate),
        Dense(128, kernel_regularizer = regularizers.l2(wd), activation="relu"),
        Dropout(rate),
        Dense(128, kernel_regularizer = regularizers.l2(wd), activation="relu"),
        Dropout(rate),
        Dense(128, kernel_regularizer = regularizers.l2(wd), activation="relu"),
        Dropout(rate),
        Dense(128, kernel_regularizer = regularizers.l2(wd), activation="relu"),
        Dropout(rate),
        Dense(1)
    ])
    return model

regularisedModelD = getRegularisedModel(1e-5, 0.3)

# Compile the model

regularisedModelD.compile(optmizer="adam", loss="mse", metrics=["mae"])

# Train the model, with some of the data reserved for validation

trainingHistoryR2 = regularisedModelD.fit(train_data, train_targets, epochs = 100,
                   validation_split = 0.15, batch_size = 64, verbose=False)

# Evaluate the model on the test set

lossR2, maeR2 = regularisedModelD.evaluate(test_data, test_targets, verbose=False)
print(f"Loss is {lossR2}\nMAE is {maeR2}")


I am working on the file "02-Classification/TensorFlow regularisation .ipynb".
The first task is:
The process of setting the hyper-parameters requires expertise and extensive trial and error. There are no simple and easy ways to set hyper-parameters — specifically, learning rate, batch size, momentum, and weight decay.
Before discussing the ways to find the optimal hyper-parameters, let us first understand learning rate, batch size, momentum and weight decay. These hyper-parameters act as knobs which can be tweaked during the training of the model. For our model to provide best result, we need to find the optimal value of these hyper-parameters.
If the learning rate (LR) is too small, overfitting can occur. Large learning rates help to regularize the training but if the learning rate is too large, the training will diverge.
Batch size must be examined in conjunction with the execution time of the training. The batch size is limited by your hardware’s memory.
It is often better to use a larger batch size so a larger learning rate can be used.
Weight decay is one form of regularization and it plays an important role in training so its value needs to be set properly.
If you have no idea of a reasonable value for weight decay, test 1/10³ , 1/10⁴ , 1/10⁵ and 0.
Test with short runs of momentum values 0.99, 0.97, 0.95, and 0.9 to get the best value for momentum.
Application - learning rate scheduler
Let's now look at a more sophisticated custom callback.
We are going to define a callback to change the learning rate of the optimiser of a model during training. We will do this by specifying the epochs and new learning rates where we would like it to be changed.
First we define the auxillary function that returns the learning rate for each epoch based on our schedule.
Can you write Python code for it?

# Define the learning rate schedule. The tuples below are (start_epoch, new_learning_rate)

lr_schedule = [
    (0, 0.01), (4, 0.007), (7, 0.005), (11, 0.003), (15, 0.001)
]

def get_new_epoch_lr(epoch, lr):
    # Checks to see if the input epoch is listed in the learning rate schedule
    # and if so, returns index in lr_schedule
    epoch_in_sched = [i for i in range(len(lr_schedule)) if lr_schedule[i][0]==int(epoch)]
    if len(epoch_in_sched)>0:
        # If it is, return the learning rate corresponding to the epoch
        return lr_schedule[epoch_in_sched[0]][1]
    else:
        # Otherwise, return the existing learning rate
        return lr

# Define the custom callback

class LRScheduler(tf.keras.callbacks.Callback):

    def __init__(self, new_lr):
        super(LRScheduler, self).__init__()
        # Add the new learning rate function to our callback
        self.new_lr = new_lr

    def on_epoch_begin(self, epoch, logs=None):
        # Make sure that the optimizer we have chosen has a learning rate, and raise an error if not
        if not hasattr(self.model.optimizer, 'lr'):
              raise ValueError('Error: Optimizer does not have a learning rate.')

        # Get the current learning rate
        curr_rate = float(tf.keras.backend.get_value(self.model.optimizer.lr))

        # Call the auxillary function to get the scheduled learning rate for the current epoch
        scheduled_rate = self.new_lr(epoch, curr_rate)

        # Set the learning rate to the scheduled learning rate
        tf.keras.backend.set_value(self.model.optimizer.lr, scheduled_rate)
        print('Learning rate for epoch {} is {:7.3f}'.format(epoch, scheduled_rate))


I am working on the file "02-Classification/TensorFlow regularisation .ipynb".
The first task is:
There are quite a few
built-in callbacks in TensorFlow and you can take a look at the list of built-in callbacks, if you go to the API
documentation on TensorFlow.org.
These actions above are simple and just serve to demonstrate
how to use callbacks, but we'll soon be
using callbacks for several important aspects
of our model chain.
Early stopping / patience
Early stopping is
a technique that monitors the performance
of the network for every epoch on a
held out validation set during the training run, and terminates the training based on the
validation performance.
The Keras module contains
a built-in callback designed for this purpose called the early
stopping cutback.
So what the early stopping
callback is doing, is monitoring the performance of the network on the
validation set and stopping the training depending on how that performance
progresses.
A natural question is, which measure of performance
is the callback monitoring? The early stopping
callback constructor takes a keyword argument
called monitor, which can be used to set which
performance metric to use.
The default setting for the early
stopping callback is the validation loss, which equals to pass in val_loss.
You can
instead choose to use the validation accuracy as the performance measure to decide when to terminate the training.
Another keyword
arguments that we can set in the early
stopping callback is the patience argument. By default, patience
is set to zero. That means that as soon as
the performance measure gets worse from one
epoch to the next, then the training is terminated. Now, this might not be
ideal since of course, the model's performance is noisy, and it might go up or down
from one epoch to the next. What we really care about is that the general trend
should be improving. That's why will often set the patience to some
number of epochs.
In this case, the training
will terminate only if there is no improvement in the monitor performance measure, the n epochs, in a row.
The early stopping callback also has a min_delta argument....
Can you write Python code for it?

from tensorflow.keras.callbacks import EarlyStopping

# Re-train the regularised model

regularisedModelES = getRegularisedModel(1e-8, 0.2)
regularisedModelES.compile(optimizer="adam", loss="mse",  metrics=["mae"])
reg_history = regularisedModelES.fit(train_data, train_targets, epochs=100,
                                       validation_split=0.15, batch_size=64, verbose=False,
                                       callbacks= [EarlyStopping(patience=3)])

# Evaluate the model on the test set

lossR2, maeR2 = regularisedModelES.evaluate(test_data, test_targets, verbose=False)
print(f"Loss is {lossR2}\nMAE is {maeR2}")


I am working on the file "02-Classification/TensorFlow regularisation .ipynb".
The first task is:
There are some parameters and hyperparameters associated with batch normalisation:

The hyperparameter momentum is the weighting given to the previous running mean when re-computing it with an extra minibatch. By default, it is set to 0.99.

The hyperparameter $\epsilon$ is used for numeric stability when performing the normalisation over the minibatch. By default it is set to 0.001.

The parameters $\beta$ and $\gamma$ are used to implement an affine transformation after normalisation. By default, $\beta$ is an all-zeros vector, and $\gamma$ is an all-ones vector.

Customising parameters
These can all be changed (along with various other properties) by adding optional arguments to tf.keras.layers.BatchNormalization().
We can also specify the axis for batch normalisation. By default, it is set as -1.
Let's see an example.
Can you write Python code for it?

# Add a customised batch normalisation layer

modelBN.add(BatchNormalization(
    momentum=0.95,
    epsilon=0.005,
    axis = -1,
    beta_initializer = tf.keras.initializers.RandomNormal(mean=0.0, stddev=0.05),
    gamma_initializer= tf.keras.initializers.Constant(value=0.9)
))

# Add the output layer

modelBN.add(Dense(1))

# Print the model summary

modelBN.summary()


I am working on the file "02-Classification/TensorFlow regularisation .ipynb".
The first task is:
We can apply the callback to the regularised model and see how the learning rate parameter gets adapted.
Can you write Python code for it?

regularisedModelLR = getRegularisedModel(1e-8, 0.2)

# Print the model summary

regularisedModelLR.summary()

# Compile the model

regularisedModelLR.compile(optimizer = "adam", loss="mse", metrics=["mae"])

# Fit the model with our learning rate scheduler callback

new_history = regularisedModelLR.fit(train_data, train_targets, epochs=20,
                            validation_split=0.15, batch_size=64, callbacks=[LRScheduler(get_new_epoch_lr)], verbose=False)


I am working on the file "02-Classification/TensorFlow regularisation .ipynb".
The first task is:
You can see that the learning rate was constantly adapted and diminished.
Can you write Python code for it?

# Evaluate the model on the test set

lossULT, maeULT = regularisedModelLR.evaluate(test_data, test_targets, verbose=False)
print(f"Loss is {lossULT}\nMAE is {maeULT}")


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
As you can see, it is a convolutional neural network with 6 layers in total and the final one classifies the input into one of ten classes.
Can you write Python code for it?

# Test accuracy of the untrained model, around 10% (random)

get_test_accuracy(model1, xTest, yTest)


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
As you can see, this time is not overwriting the file with the weights but is creating new files every 5000 training points. And notice now that
the file name has changed accordingly the format we have set in the model checkpoint file path.
Work with model saving criteria
Now we're going to create a checkpoint that only saves the model weights that had the
highest testing or validation accuracy.
Basically we see if we can just find the point at which it starts overfitting and
use just those weights. So let's go ahead and
create a new instance of this model.
Can you write Python code for it?

# Create a new instance of untrained model

model3 = get_new_model()


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
Create new model from scratch
So what about loading an entire model? Well, this time, we don't need
the original code that was used to build the model, so
the loading code looks a lot simpler: just  import and call
the load_model function from the tensorflow.keras.models module. And to load the model, I just pass the path to the model folder, and it will return the complete model
instance, architecture and saved weights.
We could now do everything that we've done
before with this loaded model instance. We could print out the model summary. We could resume training,
using the fit method. We could run evaluate to
evaluate the loss in metrics on a held-out test set. Or get model predictions from unseen
examples using predict.
Can you write Python code for it?

# Delete model

del model4

from tensorflow.keras.models import load_model

# Reload model from scratch

model4 = load_model(cp_path)

get_test_accuracy(model4, xTest, yTest)


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
Image 1: lemon
Can you write Python code for it?

# Display image

lemon_img

# Display top 5 predictions

get_top_predictions(lemon_img)


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
Image 2: viaduct
Can you write Python code for it?

# Display image

viaduct_img

# Display top 5 predictions

get_top_predictions(viaduct_img)


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
Image 3: water tower
Can you write Python code for it?

# Display image

water_tower_img

# Display top 5 predictions

get_top_predictions(water_tower_img)


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
Lastly we need to train the model, by calling the usual model.fit, parsing in the training
inputs and outputs and setting it to train for few epochs. Just as we saw previously, we're using the
callbacks keyword argument, parsing in a list which contains our ModelCheckpoint callback object. This callback will save the model weights
after every epoch.
Now, because we're using the same file name
to save them model, the saved weights will get
overwritten every epoch.
Can you write Python code for it?

# Fit model, with simple checkpoint which saves (and overwrites) model weights every epoch

history1 = model1.fit(xTrain, yTrain, epochs=3, batch_size=64, callbacks=[checkpoint], verbose=1)

# Have a look at what the checkpoint creates

! ls -lh modelCheckpoints


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
Model saving criteria
Create more customised checkpoint
More possibilities to save the model are available, for example the weights can be saved at a different frequency or
according to some criteria.
The ModelCheckpoint
callback comes with a few extra options that
help to have a little bit more customization.
For example, we are goiing to create a checkpoint that saves the model weights every
5,000 training points. Now, instead of writing save_frequency is
epoch which it was before, we put a number, which is the number of training points after it saves.
And TensorFlow has this neat trick
where you can format the checkpoint file path using variables, in this case we add the epoch and batch.
The other parameters are the same: it's save_weights_only=True so we're still saving only the weights and verbose=1 so we have some
progress printed statements during the training.
The new checkpoint is called  cp_5000 and is added as a callback in the model.fit
Can you write Python code for it?

# Create Tensorflow checkpoint object with epoch and batch details

cp_5000_path = 'model_checkpoints_5000/checkpoint_{epoch:02d}_{batch:04d}'
cp_5000 = ModelCheckpoint(cp_5000_path, save_weights_only=True, save_freq=5000, verbose=1)

# Create and fit model with checkpoint

model2 = get_new_model()
history2 = model2.fit(xTrain, yTrain,
                    epochs=3, batch_size=10,
                    validation_data=(xTest, yTest),
                    callbacks=[cp_5000])


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
Now that training is finished, you see that it saved every epoch and in total produced 3 files. The first one has the actual weights resulting from all the epochs (it was overwritten every epoch).
Can you write Python code for it?

# Evaluate the performance of the trained model

get_test_accuracy(model1, xTest, yTest)


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
Now, because we're saving the entire model
and not just the weights, the file path is used to create a subdirectory
within the current working directory. Inside that, you'll find two more
subdirectories, assets and variables. The assets folder is where
files can be stored that are used by the TensorFlow graph. In our simple example,
this folder would be empty. The variables folder contains
the saved weights of the model. These file names should look familiar
from when we were saving weights only. The last file here, saved_model.pb, is the file that stores
the TensorFlow graph itself. You can think of this as
storing the model architecture. So it contains all the information from
when we built and compiled the model, including the optimizer state, in case we want to resume
training from a saved model.
It's also possible to only save the model
architecture without the weights
Can you write Python code for it?

# Have a look at what the checkpoint creates

!ls -lh My_Checkpoints

# Enter variables directory

!ls -lh My_Checkpoints/variables


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
Our training accuracy goes up towards 1 (and will likely continue doing it after the 20 epochs) because the model is essentially
memorizing the training data set. But the val_accuracy doesn't really
improve after it reached around  0.58 during the 15th epoch;  it kind of plateaus off there  and
that is the highest val_accuracy.
Can you write Python code for it?

# Inspect the checkpoint directory

! ls -lh model_checkpoints_best


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
Saving and loading weights
Let's start by seeing how to save the weights of a model.
We will train a model and save the resulting weights, just those parameters.
Firstly we introduce two useful functions: to test the accuracy and to create a new model (a simple CNN)
Can you write Python code for it?

# Introduce function to test model accuracy

def get_test_accuracy(model, x_test, y_test):
    test_loss, test_acc = model.evaluate(x=x_test, y=y_test, verbose=0)
    print(f'accuracy: {test_acc}')

# Introduce function that creates a new instance of a simple Convolutional Neural Network

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Flatten, Conv2D, MaxPooling2D

def get_new_model():
    model = Sequential([
        Conv2D(filters=16, input_shape=(32, 32, 3), kernel_size=(3, 3),
               activation='relu'),
        Conv2D(filters=8, kernel_size=(3, 3), activation='relu'),
        MaxPooling2D(pool_size=(4, 4)),
        Flatten(),
        Dense(units=32, activation='relu'),
        Dense(units=10, activation='softmax', name = 'output-10-classes')
    ])
    model.compile(optimizer='adam',
                  loss='sparse_categorical_crossentropy',
                  metrics=['accuracy'])
    return model


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
So you see here, it's every epoch. When the val_accuracy improves,
it saves. If in one epoch the accuracy doesn't improve,
it doesn't do anything. Note that it's using the same
file name again every time, so it's going to overwrite this file. So at the end I will  be left with
only checkpoints that have the highest val_accuracy. Let's go ahead and
plot the training and testing curves.
Can you write Python code for it?

# Plot training and testing curves

import pandas as pd

df = pd.DataFrame(history3.history)
df.plot(y=['accuracy', 'val_accuracy']);


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
To load this model, you
import the ResNet-50 class from the
tensorflow.keras.applications.resnet50 module. Then create an instance of the
ResNet50 object and when this line
of code executes, the model architecture in
weights will be downloaded  into a
hidden folder in your home directory inside a
subdirectory called models. You can also see I've passed in the keyword arguments,
weights equals ImageNet. That means that the model will be loaded with weights that have been learned from training
on the ImageNet dataset. Otherwise, if you
set this to none, then the weights will be randomly initialized for a fresh training.
Another interesting
option you can use is the include top keyword argument. By default, this is equal
to true in which case the complete classifier model is downloaded and instantiated. However, if you set
include top to be false, the fully connected layer at the top of the
network isn't loaded. So what you end up with is a headless model that you can use for things like transfer
learning applications.
Can you write Python code for it?


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
Use the .h5 format to save model
Just as a note, you can also save the model using the keras format, just by adding the .h5 extension to the file name.
The Keras format is more synthetic: just one file is saved
modelToBeSaved.save('my_model.h5')
modelRetrieved = load_model('my_model.h5')
Clear directory
Can you write Python code for it?


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
We create a checkpoint file path called model_checkpoints_best with a directory
called model_checkpoints_best. And then we create an instance of ModelCheckpoint, in the smae way as above but with filepath=checkpoint_best_path.
Again, it will save just the weights,
so save_weights_only=True and we go back to do save_freq='epoch', so saving every epoch.
The critical argument here is the monitor, that it
is what tensorflow uses to decide whether to save the weights or not. There fore we have monitor='val_accuracy' and
 save_best_only=True. So it only saves the weights with
the highest value of the monitor. In this case, it's the val_accuracy.
Can you write Python code for it?

# Create Tensorflow checkpoint object which monitors the validation accuracy

checkpoint_best_path = 'model_checkpoints_best/checkpoint'
cp_best = ModelCheckpoint(checkpoint_best_path,
                          save_weights_only=True,
                          verbose=1,
                          save_best_only=True,
                          monitor='val_accuracy',
                          save_freq='epoch')


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
What happened here is that - after 5000
training points - it prints "saving model to model_checkpoints_5000/checkpoint_01_0499"
The 01 refers to the epochs,
it's the first epoch. And the 499 is essentially the 500th batch. And that's why I chose
 batch_size as 10 in the model.fit: so 5000 training points
corresponds to 500 batches.
Basically, every epoch it has the
model_checkpoints saved. The checkpoint_01_499 was
after the 500th batch and checkpoint_01_999 was after the 1000th batch,
essentially the end of the epoch. And does the same thing for
epochs 2 and 3.
Can you write Python code for it?

# Have a look at what the checkpoint creates

!ls -lh model_checkpoints_5000


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
it is a pretty big model! It has dozens of layers and 25 milion parameters
Import and preprocess 3 sample images
I'm using the load image function to load in an image I've got stored in the
current working directory. I'm also passing the
target size keyword and setting that to 224 by 224. If you take a look at the
documentation you'll see that this is the required input size for the ResNet-50 classifier model, and the load image function can handle any necessary
resizing for us.
Can you write Python code for it?

# Import 3 sample ImageNet images

from tensorflow.keras.preprocessing.image import load_img

lemon_img = load_img('../datasets/lemon.jpg', target_size=(224, 224))
viaduct_img = load_img('../datasets//viaduct.jpg', target_size=(224, 224))
water_tower_img = load_img('../datasets//water_tower.jpg', target_size=(224, 224))


I am working on the file "02-Classification/Tensorflow-save-load.ipynb".
The first task is:
let's see what is actually
in this directory right now. You see just three files: the checkpoint, checkpoint.data
and checkpoint.index. And we expect that this is going to
be the highest val_accuracy.
Clear directory
Can you write Python code for it?

! rm -r model_checkpoints_5000 model_checkpoints_best


I am working on the file "02-Classification/digits.ipynb".
The first task is:
2 - Data preprocessing
Dataset pre-processing:

Figure out the dimensions and shapes of the problem
Split the dataset into training a test subsets
"Standardise" the data
Can you write Python code for it?


I am working on the file "02-Classification/digits.ipynb".
The first task is:
3 - Deep Neural Network for Image Classification
Now we will build and apply a deep neural network to the problem.
Building the parts of our algorithm ##
The main steps for building a Neural Network are as usual:
1. Define the model structure (such as number and size of layers) and the hyperparameters
1. Initialize the model's weights
1. Loop for the number of iterations:
    - Calculate current loss (forward propagation)
    - Calculate current gradient (backward propagation)
    - Update parameters (gradient descent)
1. Use the trained weights to predict the labels
Defining the neural network structure
Our neural network has 3 layers – an input layer, a hidden layer and an output layer.
Recall that our inputs are pixel values of digit images. Since the images are of size 20×20, this gives us 400 input layer units (excluding the extra bias unit which always outputs +1).
There are 25 units in the second layer and 10 output units (corresponding to the 10 digit classes).
Can you write Python code for it?

### CONSTANTS DEFINING THE MODEL ####
    # we define a neural network with total 3 layers, x, y and 1 hidden:
n_h = 25
nn_layers = [input_layer_size, n_h, n_y]  # length is 3 (layers)


I am working on the file "02-Classification/digits.ipynb".
The first task is:
4. Results analysis
Now we can check the performance of the trained network by predicting the results of the test set and comparing them with the actual labels.
Note that the predict() function has been adapted to cope with the multi-class labels.
Can you write Python code for it?

def predict(X, yOHE, parameters):
    """
    This function is used to predict the results of a  L-layer neural network.
    It also checks them against the true labels and print the accuracy
    Arguments:
    X -- data set of examples you would like to label
    yOHE -- the true labels, as multi-class vectors
    parameters -- parameters of the trained model

    Returns:
    p -- predictions (the label) for the given dataset X
    """

    m = X.shape[1]
    nLabels = yOHE.shape[1]
    n = len(parameters) // 2 # number of layers in the neural network
    p = np.zeros((1, m)) # the predicted output, initialised to zero
    y = np.zeros((1, m)) # the actual output

    # Forward propagation
    probas, caches = L_model_forward(X, parameters)

    # probas is a matrix of shape [nLabels, m] (one-hot-encoded)
    assert (probas.shape[1] == m)

    for i in range(0, m):
            # convert probs to label predictions:
            # just take the label with max prob
        p[0,i] = np.argmax(probas[:,i])

        # convert expected results into label: takes the value with one
        y[0,i] = np.argmax(yOHE[:,i])

    # print results
    print("Accuracy: "  + str(np.sum((p == y)/m)))

    return p

print ("On the training set:")
predictions_train = predict(train_set_x, train_set_y, fit_params)
print ("On the test set:")
predictions_test = predict(X_test.T, y_test.T, fit_params)


I am working on the file "02-Classification/digits.ipynb".
The first task is:
5 - Initializing parameters
There are two types of parameters to initialize in a neural network:
- the weight matrices $W^{[i]}$
- the bias vectors $b^{[i]}$
The weight matrix is initialised with random values while the bias vector as a vector of zeros.
In general, initializing all the weights to zero results in the network failing to break symmetry. This means that every neuron in each layer will learn the same thing and the network is no more powerful than a linear classifier such as logistic regression.
To break symmetry, we initialise the weights randomly. Following random initialisation, each neuron can then proceed to learn a different function of its inputs.
Of course, different initializations lead to different results and poor initialisation can slow down the optimisation algorithm.
One good practice is not to initialise to values that are too large, instead what bring good results are the so-called Xavier or the He (for ReLU activation) initialisations.
Finally, we try here the "He Initialization"; this is named for the first author of He et al., 2015. (If you have heard of "Xavier initialization", this is similar except Xavier initialization uses a scaling factor for the weights $W^{[l]}$ of sqrt(1./layers_dims[l-1]) where He initialization would use sqrt(2./layers_dims[l-1]).)
This function is similar to the previous initialize_parameters_random(...). The only difference is that instead of multiplying np.random.randn(..,..) by 10, you will multiply it by $\sqrt{\frac{2}{\text{dimension of the previous layer}}}$, which is what He initialization recommends for layers with a ReLU activation.
Can you write Python code for it?

#  FUNCTION: initialize_parameters

def initialise_parameters_he(layer_dims):
    """
    Arguments:
    layer_dims -- python array (list) containing the dimensions of each layer in our network

    Returns:
    parameters -- python dictionary containing the parameters "W1", "b1", ..., "WL", "bL":
                    Wl -- weight matrix of shape (layer_dims[l], layer_dims[l-1])
                    bl -- bias vector of shape (layer_dims[l], 1)
    """

    parameters = {}
    L = len(layer_dims)            # number of layers in the network

    for l in range(1, L):
        parameters['W' + str(l)] = np.random.randn(layer_dims[l], layer_dims[l-1])*np.sqrt(2./layer_dims[l-1])
        parameters['b' + str(l)] = np.zeros((layer_dims[l], 1))

            # unit tests
        assert(parameters['W' + str(l)].shape == (layer_dims[l], layer_dims[l-1]))
        assert(parameters['b' + str(l)].shape == (layer_dims[l], 1))

    return parameters


I am working on the file "02-Classification/digits.ipynb".
The first task is:
6 - L2 Regularization
The standard way to avoid overfitting is called L2 regularization.
It consists of appropriately modifying your cost function, from:
$J = -\frac{1}{m} \sum\limits_{i = 1}^{m} \large{(}\small  y^{(i)}\log\left(a^{L}\right) + (1-y^{(i)})\log\left(1- a^{L}\right) \large{)}$
To:
$J_{regularized} = \small \underbrace{-\frac{1}{m} \sum\limits_{i = 1}^{m} \large{(}\small y^{(i)}\log\left(a^{L}\right) + (1-y^{(i)})\log\left(1- a^{L}\right) \large{)} }\text{cross-entropy cost} + \underbrace{\frac{1}{m} \frac{\lambda}{2} \sum\limits_l\sum\limits_k\sum\limits_j W{k,j}^{[l]2} }_\text{L2 regularization cost}$
Can you write Python code for it?

# GRADED FUNCTION: compute_cost_with_regularization

def compute_cost_with_regularisation(A3, Y, parameters, lambdaHyper):
    """
    Implement the cost function with L2 regularization. See formula (2) above.

    Arguments:
    A3 -- post-activation, output of forward propagation, of shape (output size, number of examples)
    Y -- "true" labels vector, of shape (output size, number of examples)
    parameters -- python dictionary containing parameters of the model
    lambdaHyper -- the lambda regularisation hyper-parameter.

    Returns:
    cost - value of the regularized loss function (formula (2))
    """
        # This gives you the cross-entropy part of the cost
    cross_entropy_cost = compute_cost(A3, Y)

    sum_regularization_cost = 0
    m = Y.shape[1]
    L = len(parameters) // 2    # number of layers (2 because we have W and b)

    for i in range(1, L+1):
        W_i = parameters['W' + str(i)]
        sum_regularization_cost += np.sum(np.square(W_i))

    regularization_cost = (1/m)*(lambdaHyper/2)*(sum_regularization_cost)

    cost = cross_entropy_cost + regularization_cost

    return cost

def compute_cost_with_regularisation_test_case():

    np.random.seed(1)
    Y_assess = np.array([[1, 1, 0, 1, 0]])
    W1 = np.random.randn(2, 3)
    b1 = np.random.randn(2, 1)
    W2 = np.random.randn(3, 2)
    b2 = np.random.randn(3, 1)
    W3 = np.random.randn(1, 3)
    b3 = np.random.randn(1, 1)

    parameters = {"W1": W1, "b1": b1, "W2": W2, "b2": b2, "W3": W3, "b3": b3}
    a3 = np.array([[ 0.40682402,  0.01629284,  0.16722898,  0.10118111,  0.40682402]])

    cost = compute_cost_with_regularisation(a3, Y_assess, parameters, lambdaHyper = 0.1)

    np.testing.assert_approx_equal(cost, 1.78649, significant=5)

    return "OK"

compute_cost_with_regularisation_test_case()


I am working on the file "02-Classification/digits.ipynb".
The first task is:
7 - Dropout
Finally, dropout is a widely used regularization technique that is specific to deep learning.
It randomly shuts down some neurons in each iteration.
When you shut some neurons down, you actually modify your model. The idea behind drop-out is that at each iteration, you train a different model that uses only a subset of your neurons. With dropout, your neurons thus become less sensitive to the activation of one other specific neuron, because that other neuron might be shut down at any time.
Can you write Python code for it?

def forward_propagation_with_dropout(X, parameters, keep_prob = 0.5):
    """
    Implements the forward propagation: LINEAR -> RELU + DROPOUT -> LINEAR -> RELU + DROPOUT -> LINEAR -> SIGMOID.

    Arguments:
    X -- input dataset, of shape (2, number of examples)
    parameters -- python dictionary containing the parameters of a 3-layers network.
    keep_prob - probability of keeping a neuron active during drop-out, scalar

    Returns:
    A2 -- last activation value, output of the forward propagation, of shape (1,1)
    cache -- tuple, information stored for computing the backward propagation
    """
    L = len(parameters) // 2                  # number of layers in the neural network

    # retrieve parameters
    W1 = parameters["W1"]
    b1 = parameters["b1"]
    W2 = parameters["W2"]
    b2 = parameters["b2"]

    # LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID
    Z1 = np.dot(W1, X) + b1
    A1,cache_temp = relu(Z1)

    D1 = np.random.rand(A1.shape[0], A1.shape[1])     # Step 1: initialize matrix D1 = np.random.rand(..., ...)
    D1 = D1 < keep_prob     # Step 2: convert entries of D1 to 0 or 1 (using keep_prob as the threshold)
    A1 = A1*D1                                 # Step 3: shut down some neurons of A1
    A1 = A1 / keep_prob                         # Step 4: scale the value of neurons that haven't been shut down

    Z2 = np.dot(W2, A1) + b2
    A2, cache_temp = sigmoid(Z2)

    caches = (Z1, D1, A1, W1, b1, Z2, A2, W2, b2)

    return A2, caches

def backward_propagation_with_dropout(X, Y, cache, keep_prob):
    """
    Implements the backward propagation of our baseline model to which we added dropout.

    Arguments:
    X -- input dataset, of shape (2, number of examples)
    Y -- "true" labels vector, of shape (output size, number of examples)
    cache -- cache output from forward_propagation_with_dropout()
    keep_prob - probability of keeping a neuron active during drop-out, scalar

    Returns:
    gradients -- A dictionary...


I am working on the file "02-Classification/digits.ipynb".
The first task is:
Build the 3-layer neural network
We will re-use all the helper functions defined previously to build the neural network, such as the linear forward and the backward propagation.
Please refer to the Python file nn_helpers.py for the details.
Can you write Python code for it?

from nn_helpers import *

# automatically reload the imported module in case of changes
%load_ext autoreload
%autoreload 2


I am working on the file "02-Classification/digits.ipynb".
The first task is:
Let's check the new accuracy values:
Can you write Python code for it?

print ("On the training set:")
predictions_train = predict(train_set_x, train_set_y, fit_params_reg)
print ("On the test set:")
predictions_test = predict(X_test.T, y_test.T, fit_params_reg)


I am working on the file "02-Classification/digits.ipynb".
The first task is:
Now we can put together all the functions to build a 3-layer neural network with this structure:
Can you write Python code for it?

nn_layers

np.random.seed(1)

train_set_x = X_train.T
train_set_x.shape

# y is the original output array, with labels
# train_set_y is that set, one-hot-encoded
train_set_y = y_train.T
train_set_y.shape

#  FUNCTION: L_layer_model

def simpleNN_model(X, Y, layers_dims, learning_rate = 0.0075, num_iterations = 3000, print_cost=False):
    """
    Implements a L-layer neural network: [LINEAR->RELU]*(L-1)->LINEAR->SIGMOID.

    Arguments:
    X -- data, numpy array of shape (number of examples, num_px * num_px)
    Y -- true "label" vector (containing 0 or 1), of shape (10, number of examples)
    layers_dims -- list containing the input size and each layer size, of length (number of layers + 1).
    learning_rate -- learning rate of the gradient descent update rule
    num_iterations -- number of iterations of the optimisation loop
    print_cost -- if True, it prints the cost every 200 steps

    Returns:
    parameters -- parameters learnt by the model. They can then be used to predict.
    """

    costs = []      # keep track of cost
    iterations2cost = 200 # Print the cost every these iterations

    # Parameters initialization.
    parameters = initialise_parameters(layers_dims)

    # Loop (gradient descent)
    for i in range(0, num_iterations):

        # Forward propagation: [LINEAR -> RELU]*(L-1) -> LINEAR -> SIGMOID.
        AL, caches = L_model_forward(X, parameters)

        # Compute cost.
        cost = compute_cost(AL, Y)

        # Backward propagation.
        grads = L_model_backward(AL, Y, caches)

        # Update parameters.
        parameters = update_parameters(parameters, grads, learning_rate)

        # Print the cost every iterations2cost training example
        if print_cost and i % iterations2cost == 0:
            print ("Cost after iteration %i: %f" %(i, cost))
        if print_cost and i % iterations2cost == 0:
            costs.append(cost)

    if print_cost:
        # plot the cost
        fig, ax =...


I am working on the file "02-Classification/digits.ipynb".
The first task is:
Of course, because we changed the cost, we have to change backward propagation as well!
All the gradients have to be computed with respect to this new cost: add the regularization term's gradient.
Can you write Python code for it?

def backward_propagation_with_regularisation(X, Y, Yhat, caches, lambdaHyper):
    """
    Implements the backward propagation of our baseline model to which we added an L2 regularization.

    Arguments:
    X -- input dataset, of shape (input size, number of examples)
    Yhat -- "true" labels vector, of shape (output size, number of examples)
    caches -- cache output from forward_propagation()
    lambdaHyper -- regularization hyperparameter, scalar

    Returns:
    gradients -- A dictionary with the gradients with respect to each parameter, activation and pre-activation variables
    """

    m = X.shape[1]

    L = len(caches) # the number of layers
    gradients = {}

    last_layer_cache = caches[L-1]
    ((A, W, b), Z) = last_layer_cache

    assert (Yhat.shape == Y.shape)

    dZ = Yhat - Y
    gradients["dZ" + str(L)] = dZ

    for i in reversed(range(L-1)):
        current_layer_cache = caches[i]
        ((A_prev, W_prev, b_prev), Z_prev) = current_layer_cache

        dW_entropy = 1./m * np.dot(dZ, A.T)
        dW_reg = (lambdaHyper/m)*W
        dW = dW_entropy + dW_reg

        db = 1./m * np.sum(dZ, axis=1, keepdims = True)

        dA_prev = np.dot(W.T, dZ)
        dZ_prev = np.multiply(dA_prev, np.int64(A > 0))

        gradients["dW" + str(i + 2)] = dW
        gradients["db" + str(i + 2)] = db

        gradients["dA" + str(i + 1)] = dA_prev
        gradients["dZ" + str(i + 1)] = dZ_prev

        ((A, W, b), Z) = ((A_prev, W_prev, b_prev), Z_prev)
        dZ = dZ_prev

        # finally add the gradients for the first layer
    dW_prev = 1./m * np.dot(dZ_prev, X.T) + (lambdaHyper/m)*W_prev
    db_prev = 1./m * np.sum(dZ_prev, axis=1, keepdims = True)

    gradients["dW1"] = dW_prev
    gradients["db1"] = db_prev

    return gradients


I am working on the file "02-Classification/digits.ipynb".
The first task is:
Split into train and test sets
Split into 20% of test and 80% of train sets
Can you write Python code for it?

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, OHE_y, test_size=0.2, random_state=7)

input_layer_size  = X.shape[1]
num_px = np.sqrt(input_layer_size) # 400 = 20x20 Input Images of Digits
n_y = y_train.shape[1]
m_train = X_train.shape[0]
m_test = X_test.shape[0]

print ("Dataset dimensions:")
print ("Number of training examples = " + str(m_train))
print ("Number of testing examples = " + str(m_test))
print ("Height/Width of each image: num_px = " + str(num_px))
print ("Each image is of size: <" + str(num_px) + ", " + str(num_px) + ">")
print ("X train shape: " + str(X_train.shape))
print ("y train shape: " + str(y_train.shape))


I am working on the file "02-Classification/digits.ipynb".
The first task is:
The second part of the training set is a 5000-dimensional vector y that contains labels for the training set.
Can you write Python code for it?

y = dataset['y'] # the labels
y.shape

y[499]


I am working on the file "02-Classification/digits.ipynb".
The first task is:
Visualise the data
Each line of X is an array representing an image. You can visualize an example by running the following code. Feel free also to change the indexImage value and re-run to see other images.
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

import numpy as np
    # Example of a picture
indexImage = 4000  # try any index between 0 and 4999. They are sorted, from 1 to 10 (=0)
renderImage = np.reshape(dataset['X'][indexImage], (20,20))
labelImage = dataset['y'][indexImage]

plt.imshow(renderImage, cmap='gray')
print ("Label: this is a ", labelImage)


I am working on the file "02-Classification/forest.ipynb".
The first task is:
Example prediction
Let's use the wrong row - that we extracted earlier from the dataset - as a prediction example.
but first we need to correct it:
Can you write Python code for it?

outputClassPredictionExample = wrongRow['class']

forPredictionExample = wrongRow.drop(labels=['class','user']) # remove class and user
forPredictionExample.z4 = -144 # correct the value

print("We use this example for prediction later:")
print(forPredictionExample)
print("The class shall be: ", outputClassPredictionExample)

model.predict(forPredictionExample.values.reshape(1, -1))


I am working on the file "02-Classification/forest.ipynb".
The first task is:
Extract the target values
Can you write Python code for it?

# Activity to predict is in "class" column
# Encode 'y' value as a dummies version of dataset's "class" column
#
y = pd.get_dummies(X['class'].copy())
# this produces a 5 column wide dummies dataframe as the y value

#
# Get rid of the user and class columns in X
#
X.drop(['class','user'], axis=1, inplace=True)

print (X.head(2))

print (y.head())


I am working on the file "02-Classification/forest.ipynb".
The first task is:
Great, no NaNs here. Let's go on.
Can you write Python code for it?

#
# Encode the gender column: 0 as male, 1 as female
#
X.gender  = X.gender.map({'Woman':1, 'Man':0})

#
# Clean up any column with commas in it
# so that they're properly represented as decimals instead
#
X.how_tall_in_meters = X.how_tall_in_meters.str.replace(',','.').astype(float)
X.body_mass_index = X.body_mass_index.str.replace(',','.').astype(float)

#
# Check data types
print (X.dtypes)

# column z4 is type "object". Something is wrong with the dataset.

#
# Convert that column into numeric
# Use errors='raise'. This will alert you if something ends up being
# problematic
#

#
# INFO: There is an error raised ... you will find it if you try the method
#
# print (X[pd.isnull(X).any(axis=1)])
# 122076 --> z4 =    -14420-11-2011 04:50:23.713
#
# !! The data point #122076 is a wrong coded record,
# change it or drop it before calling the to_numeric methods:
#
#X.at[122076, 'z4'] = -144   // change to correct value

# I keep this value for later and drop it from the dataset
wrongRow = X.loc[122076]
X.drop(X.index[[122076]], inplace=True)

X.z4 = pd.to_numeric(X.z4, errors='raise')

print (X.dtypes)
# everything ok now


I am working on the file "02-Classification/forest.ipynb".
The first task is:
Note that it takes a much longer time to train a forest than a single decision tree.
This is the score based on the test dataset that we split earlier. Note how good it is.
Can you write Python code for it?

print ("Scoring...")
s = time.time()

score = model.score(X_test, y_test)

print ("Score: ", round(score*100, 3))
print ("Scoring completed in: ", time.time() - s)


I am working on the file "02-Classification/forest.ipynb".
The first task is:
Pre-processing the data
What we want to do is to predict the activity class based on the accelerometer's data from the wearables.
Can you write Python code for it?

#
# An easy way to show which rows have NaNs in them:
print (X[pd.isnull(X).any(axis=1)])


I am working on the file "02-Classification/forest.ipynb".
The first task is:
Split the dataset into training and test
Can you write Python code for it?

#
# Split  data into test / train sets
#
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
                                                    random_state=7)


I am working on the file "02-Classification/forest.ipynb".
The first task is:
The fourth one is "standing up". Seems that the model predicted correctly.
OutOfBag error instead of splitting into train and test
Since each tree within the forest is only trained using a subset of the overall training set, the forest ensemble has the ability to error test itself.
It does this by scoring each tree's predictions against that tree's out-of-bag samples. A tree's out of bag samples are those forest training samples that were withheld from a specific tree during training.
One of the advantages of using the out of bag (OOB) error is that eliminates the need to split your data into a training / testing before feeding it into the forest model, since that's part of the forest algorithm. However using the OOB error metric often underestimates the actual performance improvement and the optimal number of training iterations.
Can you write Python code for it?

modelOOB = RandomForestClassifier(n_estimators=30, max_depth= 20, random_state=0,
                               oob_score=True)

print ("Fitting...")
s = time.time()

modelOOB.fit(X, y)

print("completed in: ", time.time() - s, "seconds")


I am working on the file "02-Classification/forest.ipynb".
The first task is:
These are the top 5 features used in the classification.
They are all related to the movements, no gender or age.
Can you write Python code for it?

# Extract feature importances
fi = pd.DataFrame({'feature': list(X_train.columns),
                   'importance': model.feature_importances_}).\
                    sort_values('importance', ascending = False)

# Display
fi.head()


I am working on the file "02-Classification/forest.ipynb".
The first task is:
You can check the SKlearn documentation to see all possible parameters.
The ones used here:

n_estimators: integer, optional (default=100)
The number of trees in the forest. Note that this number changed from 10 to 100 (following the progress in computing performance and memory)
max_depth: integer or None, optional (default=None)
The maximum depth of the tree. If None, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples.
Setting a limit helps with the computing time and memory needed.Not setting a max depth will lead to have unpruned and fully grown trees which - depending on the dataset - will require large memory footprint.
oob_score: bool (default=False)
Whether to use out-of-bag samples to estimate the generalization accuracy.
random_state: int, RandomState instance or None, optional (default=None)
Controls both the randomness of the bootstrapping of the samples used when building trees (if bootstrap=True) and the sampling of the features to consider

And other useful / important:

criterion: string, optional (default=”gini”)
The function to measure the quality of a split. Supported criteria are “gini” for the Gini impurity and “entropy” for the information gain.
Same as for the Trees.
bootstrap: boolean, optional (default=True)
Whether bootstrap samples are used when building trees. If False, the whole datset is used to build each tree.
Can you write Python code for it?

print ("Fitting...")
s = time.time()

model.fit(X_train, y_train)

print("completed in: ", time.time() - s, "seconds")


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
After adding in all the other features and properly encoding those that
need to be, the separation between CKD and Non-CKD
patients increases significantly: by taking many secondary features and combining them, machine
 learning is usually able to come up with more informative descriptions of
 the data.
PCA Operates based on variance. The variable with the greatest variance will dominate.
Let's peek into the data and check the variance of every feature in the dataset.
Can you write Python code for it?


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
Between linear PCA and the non-linear Isomap, the latter algorithm is better able to capture the true nature of the faces dataset when reduced to two components.
Each coordinate axis of the 3D manifold  correlate highly with one degree of freedom from the original, underlying data.
In the isomap plot of the first two components (0 and 1), above, you can see that's the Left and Right Head Position 'degree of freedom' was encoded onto the first component (the X-axis). In other words, what varies as you move horizontally in the manifold rendering.
Alter the code to graph the second and third components (index=1 and 2) instead of the 0th and 1st, for both PCA and Isomap.
Can you write Python code for it?

Plot2D(T, "PCA transformation", 1, 2, num_to_plot=40)

Plot2D(manifold, "ISO transformation", 1, 2, num_to_plot=40)


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
Decompose to 2D using PCA
Can you write Python code for it?

from sklearn.decomposition import PCA

def do_PCA(armadillo):
  #
  # import the libraries required for PCA.
  # Then, train your PCA on the armadillo dataframe. Finally,
  # drop one dimension (reduce it down to 2D) and project the
  # armadillo down to the 2D principal component feature space.
  #

  pca = PCA(n_components=2)
  pca.fit(armadillo)

  reducedArmadillo = pca.transform(armadillo)

  return reducedArmadillo


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
Let's plot the features with their vectors:
Can you write Python code for it?

import math

def drawVectors(transformed_features, components_, columns, plt):

  num_columns = len(columns)

  # This funtion will project your *original* feature (columns)
  # onto your principal component feature-space, so that you can
  # visualize how "important" each one was in the
  # multi-dimensional scaling

  # Scale the principal components by the max value in
  # the transformed set belonging to that component
  xvector = components_[0] * max(transformed_features[:,0])
  yvector = components_[1] * max(transformed_features[:,1])

  # Sort each column by its length. These are your *original*
  # columns, not the principal components.
  important_features = { columns[i] : math.sqrt(xvector[i]**2 + yvector[i]**2) for i in range(num_columns) }
  important_features = sorted(zip(important_features.values(), important_features.keys()), reverse=True)
  print ("Features by importance:\n", important_features)

  ## visualize projections

  ax = plt.axes()

  for i in range(num_columns):
    # Use an arrow to project each original feature as a
    # labeled vector on your principal component axes
    plt.arrow(0, 0, xvector[i], yvector[i], color='b', width=0.0005, head_width=0.02, alpha=0.75)
    plt.text(xvector[i]*1.2, yvector[i]*1.2, list(columns)[i], color='b', alpha=0.75)

  return ax

# Plot the transformed data as a scatter plot. Recall that transforming
# the data will result in a NumPy NDArray. Therefore we convert it to DataFrame and have pandas
# do the plot.
#
# Since we transformed via PCA, we no longer have column names. We know we
# are in P.C. space, so we'll just define the coordinates accordingly:

ax = drawVectors(T, pcaModel.components_, df.columns.values, plt)
T = pd.DataFrame(T)
T.columns = ['PC 1', 'PC 2']
T.plot.scatter(x='PC 1', y='PC 2', marker='o', c=labelsCol, alpha=0.75, ax=ax);


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
Look closely at the Isomap plot. Can you tell what 'degree of freedom' the X axis represents?
It's the Down and Up Head Position
Draw the dataframes in 3D
Can you write Python code for it?

def Plot3D(T, title, x, y, z):

  fig = plt.figure()
  ax = fig.add_subplot(111, projection = '3d')

  ax.set_title(title)
  ax.set_xlabel('Component: {0}'.format(x))
  ax.set_ylabel('Component: {0}'.format(y))
  ax.set_zlabel('Component: {0}'.format(z))

  ax.scatter(T[:,x],T[:,y],T[:,z], marker='.', c='red')

Plot3D(T, "PCA transformed data 3D", 0, 1, 2)

Plot3D(manifold, "Isomap transformed data 3D", 0, 1, 2)


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
Next we split the data into separate training and testing sets, using the usual 70-30 proportions.
Can you write Python code for it?

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, labels, test_size=0.3, random_state=0)


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
Note that scaling the features did affect their variances.
After scaling, the green patients without chronic kidney disease are more cleanly separable from the red patients with chronic kidney disease.
Armadillo: From 3D rendering to 2D plot
The goal is to reduce the dimensionality of a 3D scan from three to two using PCA to cast a shadow of the data onto its two most
important principal components. Then render the resulting 2D scatter plot.
The scan is a real life armadillo sculpture scanned using a Cyberware 3030 MS 3D scanner at Stanford University.
The sculpture is available as part of their 3D Scanning Repository - where the PLY file is available - and is a very dense 3D mesh consisting of 172974 vertices!
Read the data
Every 100 data samples, we save 1.
If things run too slow, try increasing this number.
If things run too fast, try decreasing it... =)
Can you write Python code for it?

reduce_factor = 100

from plyfile import PlyData

# Load up the scanned armadillo
plyfile = PlyData.read('../Datasets/stanford_armadillo.ply')
armadillo = pd.DataFrame({
  'x':plyfile['vertex']['z'][::reduce_factor],
  'y':plyfile['vertex']['x'][::reduce_factor],
  'z':plyfile['vertex']['y'][::reduce_factor]
})


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
Now we will standardise the Wheat data to unit variance using SKlearn.
SKLearn has many different methods for transforming features by scaling them (this is a type of pre-processing) : RobustScaler, Normalizer, MinMaxScaler, MaxAbsScaler, StandardScaler...
However in order to be effective at PCA, there are a few requirements that must be met, and which will drive the selection of the scaler.
PCA required that the data is standardized -- in other words its mean is equal to 0, and it has ~unit variance.
SKLearn's regular Normalizer doesn't zero out the mean of your data, it only clamps it, so it's inappropriate to use here (depending on the data).
MinMaxScaler and MaxAbsScaler both fail to set a unit variance, so we won't be using them either.
RobustScaler can work, again depending on the data (but watch for outliers).
For these reasons we're going to use the StandardScaler.
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler

sc = StandardScaler()

X_train_std = sc.fit_transform(X_train)

X_test_std = sc.transform(X_test)


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
Pictures are rotated.
Rotate them, so we don't have to crane our necks:
Can you write Python code for it?

for i in range(num_images):
  df.loc[i,:] = df.loc[i,:].values.reshape(num_pixels, num_pixels).T.reshape(-1)


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
Prepare the function to plot the 2D images representation
Can you write Python code for it?

import random

# The format is: Plot2D(T, title, x, y, num_to_plot=40):
# T is the transformed data, NDArray.
# title is the chart title
# x is the principal component you want displayed on the x-axis, Can be 0 or 1
# y is the principal component you want displayed on the y-axis, Can be 1 or 2

def Plot2D(T, title, x, y, num_to_plot=40):
  # This method picks a bunch of random samples (images in our case)
  # to plot onto the chart:
  fig = plt.figure()
  ax = fig.add_subplot(111)

  ax.set_title(title)
  ax.set_xlabel('Component: {0}'.format(x))
  ax.set_ylabel('Component: {0}'.format(y))

  x_size = (max(T[:,x]) - min(T[:,x])) * 0.08
  y_size = (max(T[:,y]) - min(T[:,y])) * 0.08

  for i in range(num_to_plot):
    img_num = int(random.random() * num_images)
    x0, y0 = T[img_num,x]-x_size/2., T[img_num,y]-y_size/2.
    x1, y1 = T[img_num,x]+x_size/2., T[img_num,y]+y_size/2.
    img = df.iloc[img_num,:].values.reshape(num_pixels, num_pixels)
    ax.imshow(img, aspect='auto', cmap=plt.cm.gray, interpolation='nearest', zorder=100000, extent=(x0, x1, y0, y1))

  # It also plots the full scatter:
  ax.scatter(T[:,x],T[:,y], marker='.',alpha=0.7)


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
Render the Original Armadillo
Can you write Python code for it?

from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.set_title('Armadillo 3D')
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel('Z')
ax.scatter(armadillo.x, armadillo.y, armadillo.z, c='green', marker='.', alpha=0.75);


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
Render the newly transformed RandomizedPCA armadillo!
Can you write Python code for it?

if not rpca is None:
  fig = plt.figure()
  ax = fig.add_subplot(111)
  ax.set_title('RandomizedPCA, build time: ' + str(time_delta))
  ax.scatter(rpca[:,0], rpca[:,1], c='red', marker='.', alpha=0.75)


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
Since we want to reduce the dimensionality of our dataset by compressing it onto a new feature subspace, we only select the subset of the eigenvectors (principal components) that contains most of the information (variance). As the eigenvalues define the magnitude of the eigenvectors, we can sort the eigenvalues by decreasing magnitude and we are interested in the top  eigenvectors based on the values of their corresponding eigenvalues.
Can you write Python code for it?

tot = sum(eigenvalues)
varianceExplained = [(i / tot) for i in sorted(eigenvalues, reverse=True)]
cumVarianceExp = np.cumsum(varianceExplained)

varianceExplained


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
The first thing PCA does is center the dataset about its mean by subtracting the mean from each sample. Looking at the .describe() output of the dataset, particularly the min, max, and mean readings per feature, some features will dominate the X axis (likely wc) and some the Y axis (likely bgr).
Now we use PCA from SKLearn
Can you write Python code for it?

from sklearn.decomposition import PCA

# Run PCA on the dataset and reduce it to 2 components
#

pcaModel = PCA(n_components=2)
pcaModel.fit(df) # fit the model

T = pcaModel.transform(df) # T is an array of len 158 (one for each df's entry) and two components

T[0]

pcaModel.explained_variance_ratio_

pcaModel.explained_variance_

pcaModel.singular_values_


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
The resulting plot indicates that the first principal component alone accounts for 70 percent of the variance. Also, we can see that the first two principal components combined explain almost 90 percent of the variance in the data.
After we have successfully decomposed the covariance matrix into eigenpairs, let's now proceed with the last steps to transform the Wheat dataset onto the new principal component axes. In this section, we will sort the eigenpairs by descending order of the eigenvalues, construct a projection matrix from the selected eigenvectors and use the projection matrix to transform the data onto the lower-dimensional subspace.
Can you write Python code for it?

eigenpairs =[(np.abs(eigenvalues[i]),eigenvectors[:,i]) for i in range(len(eigenvalues))]
eigenpairs.sort(reverse=True)


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
There are 158 entries.
Each one has the classification target column, plus 24 other classes: 11 numeric and 14 nominal.
See the site page for an explanation of each class.
Can you write Python code for it?

# Create some color coded labels; the actual label feature
# will be removed prior to executing PCA, since it's unsupervised.
# we're only labeling by color so you can see the effects of PCA

labelsCol = ['red' if i=='ckd' else 'green' for i in df.classification]

# Get rid of nominal columns
#

df.drop(labels=['classification'], axis=1, inplace=True)

df = pd.get_dummies(df,columns=['rbc', 'pc', 'pcc', 'ba', 'htn',
                                'dm', 'cad', 'appet', 'pe', 'ane'])

print(df.dtypes)

# Print out and check the dataframe's dtypes.
#

df.wc = pd.to_numeric(df.wc, errors='coerce')
df.rc = pd.to_numeric(df.rc, errors='coerce')
df.pcv = pd.to_numeric(df.pcv, errors='coerce')

print(df.dtypes)


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
There are some NaN. Let's drop them.
Can you write Python code for it?


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
Time the execution of PCA 5000x
PCA is ran 5000x in order to help decrease the potential of rogue processes altering the speed of execution.
Can you write Python code for it?

import datetime

t1 = datetime.datetime.now()
for i in range(5000): pca = do_PCA(armadillo)
time_delta = datetime.datetime.now() - t1


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
Time the execution of randomized PCA 5000x
Can you write Python code for it?

t1 = datetime.datetime.now()
for i in range(5000): rpca = do_RandomizedPCA(armadillo)
time_delta = datetime.datetime.now() - t1


I am working on the file "02-Classification/isomap-pca.ipynb".
The first task is:
We can run PCA also changing the solver algorithm, specifically the Randomized (truncated) method for the SVD solver.
To find out how to use it, check out the full docs
Can you write Python code for it?

def do_RandomizedPCA(armadillo):

  pca = PCA(n_components=2, svd_solver='randomized')
  pca.fit(armadillo)

  reducedArmadillo = pca.transform(armadillo)

  return reducedArmadillo


I am working on the file "02-Classification/jhu.ipynb".
The first task is:
And now we can again apply the OneVsAll model but this time to the reduced dataset:
Can you write Python code for it?

print ("Training the One Vs All model on reduced 4-components dataset ...")
s = time.time()

modelOneVsAll.fit(XtrainLDAred, ytrain)

print("Done! Completed in: ", time.time() - s, "seconds")

modelOneVsAll.score(XtestLDAred, ytest)


I am working on the file "02-Classification/jhu.ipynb".
The first task is:
Another 6 features removed.
A final reduction could be to remove all the features that have a high correlation between them, which I skip.
Normalisation
LDA (and in general other classification models) works better if the data set is normalised.
Can you write Python code for it?

from sklearn import preprocessing

normaliser = preprocessing.Normalizer()

  # this will keep the columns names
xNormalised = pd.DataFrame(normaliser.fit_transform(X_hv), columns = X_hv.columns)


I am working on the file "02-Classification/jhu.ipynb".
The first task is:
Baseline prediction
A baseline is needed to see if any model trained is really that useful.  Can it beat the simplest baseline?
Our baseline will be deciding randomly the "classe" category based on the frequency in the training set (classe A is 28% of the times, classe B is 19% and so on …)
The baseline accuracy would be around 0.2, as there are five classes to choose.
Any model with a higher accuracy than the baseline is a better model.
Can you write Python code for it?

y.value_counts() # frequency of each category


I am working on the file "02-Classification/jhu.ipynb".
The first task is:
Classes are almost uniformly distributed, with a small prevalence for A-class.
Preprocess the data
Can you write Python code for it?

y = X.classe.copy() # copy “y” column (the target) values out

X.drop(['classe'], axis=1, inplace=True) # then, drop y column


I am working on the file "02-Classification/jhu.ipynb".
The first task is:
Copy all "high variance" features into a new dataframe x_hv:
Can you write Python code for it?

X_hv = X.loc[:, mask == True]

X_hv.shape

totalObs = X_hv.shape[0] # this will soon come useful
totalObs


I am working on the file "02-Classification/jhu.ipynb".
The first task is:
Explore the data
Can you write Python code for it?


I am working on the file "02-Classification/jhu.ipynb".
The first task is:
Feature extraction
160 features are computationally expensive for the model training, so we aim to reduce them.
Remove user- and time-dependent features
First of all, it’s clear from the summary above that the first six variables have no use, since they are user-dependent and time-dependent; so we remove them:
Can you write Python code for it?

columnsToDelete = ['ID','user_name', 'raw_timestamp_part_1', 'raw_timestamp_part_2','cvtd_timestamp', 'new_window', 'num_window']

X.drop(columnsToDelete, axis = 1, inplace=True)


I am working on the file "02-Classification/jhu.ipynb".
The first task is:
Find all features with variance larger than 0.5 (you can tune it):
Can you write Python code for it?

mask = selector.variances_ > 0.5  # arbitrary value 0.5


I am working on the file "02-Classification/jhu.ipynb".
The first task is:
From the statement above, some variables have missing values (NaN).
We could substitute them with the average for that variable but there are many missing values and this is not improving the model accuracy (I tried).
Instead of less-accurate imputation of missing data, I just remove all predictors with NA values. Hard but fair …
Can you write Python code for it?


I am working on the file "02-Classification/jhu.ipynb".
The first task is:
Great! The LDA model is much better than the baseline and similar to the OneVsAll results plus it was quite fast.
Note: 70% of correct predictions is not exactly perfect but it was not the goal of this exercise: you can get almost perfect results using decision trees or random forests.
Can you write Python code for it?


I am working on the file "02-Classification/jhu.ipynb".
The first task is:
Re-train the OneVsAll model with only the top four features.
Now that we have the top 4 features, we can train again the model only with them:
Can you write Python code for it?

top4features = ['pitch_forearm', 'yaw_belt', 'magnet_dumbbell_z', 'roll_dumbbell']

Xtrain_reduced = Xtrain[top4features].copy()

Xtrain_reduced.shape

Xtest_reduced = Xtest[top4features].copy()

print ("Training the OneVsAll model on 4 features ...")
s = time.time()

modelOneVsAll.fit(Xtrain_reduced, ytrain)

print("Done! Completed in: ", time.time() - s, "seconds")

modelOneVsAll.score(Xtest_reduced, ytest)


I am working on the file "02-Classification/jhu.ipynb".
The first task is:
Remove features with missing values
Can you write Python code for it?


I am working on the file "02-Classification/jhu.ipynb".
The first task is:
Split the data into training and testing
For each model I will measure the out of sample error that is the error rate you get on a new data set.
The purpose of using a different data set than training is model checking. I want to validate how well the model got trained.
I will calculate the out of sample error by looking at the accuracy.
Can you write Python code for it?

from sklearn.model_selection import train_test_split

Xtrain, Xtest, ytrain, ytest = train_test_split(xNormalised, y, test_size=0.2, random_state=7)


I am working on the file "02-Classification/jhu.ipynb".
The first task is:
The model scores a better accuracy than the baseline but it took a long time.
Using a Logistic multinomial regression hold similar accuracy results.
Reduce even more the features
One possibility would be to apply the LR model to a very reduced subset of features.
To do that we need first to find out which are the most important features.
As we remember a good way to have them is to use a decision tree. Let's use the tree as a shortcut to find the most important features.
Train a decision tree
Can you write Python code for it?

from sklearn.ensemble import ExtraTreesClassifier
tree = ExtraTreesClassifier()

tree.fit(Xtrain, ytrain)

feat_importances = pd.Series(tree.feature_importances_, index=Xtest.columns)
feat_importances.nlargest(4)