I am working on the file "Deep Learning/Naive Version of Neural Network.ipynb".
The first task is:
Neural Network
BP Algorithms
Weights: $ \omega(i,j)$ means the weight in $i$-th node of $j$-th layer, $b_j$ means bias weight
Error:
$$ Err(\omega,b) = \frac{1}{2}(y - y_{pred})^2 $$
$$ x_j = f\left(\sum_{i=1}^{M} \omega_{i,j} x_i + b_j\right) $$
$$ \frac{\partial Err(\omega,b)}{\partial \omega_{i,j}} = (y-y_{pred}) \frac{\partial y_{pred}} {\partial \omega_{i,j}}$$
$$ \frac{\partial y_{pred}} {\partial \omega_{i,j}} = f'(S_{j}) \frac{\partial S_{j}} {\partial \omega_{i,j}} = f'(S_{j}) x_i$$
where $ S_{j} = \sum_{i=1}^{M} \omega_{i,j} x_i + b_j$. In same way, we can have:
$$ \frac{\partial y_{pred}} {\partial b_j} = f'(S_{j}) \frac{\partial S_{j}} {\partial b_j} = f'(S_{j}) $$
So, we have:
$$ \frac{\partial Err(\omega,b)}{\partial \omega_{i,j}} = \delta_{i,j} x_i$$
$$ \frac{\partial Err(\omega,b)}{\partial b_j} = \delta_{i,j}$$
where $\delta_{i,j} = (y-y_{pred})f'(S_{j})$. Meanwhile, according to gradient theroem, we can update weight by the following rule:
$$ \Delta\omega_{i,j} = - \eta \frac{\partial Err(\omega,b)}{\partial \omega_{i,j}} $$
where $\eta$ is the learning rate.
Can you write Python code for it?

class NeuralNetwork:
    def __init__(self,layers,activation='tanh'):
        if activation == 'logistic':
            self.activation = logistic
            self.activation_deriv = logistic_deriv
        elif activation == 'tanh':
            self.activation = tanh
            self.activation_deriv = tanh_deriv

        self.weights = []
        for i in range(1, len(layers) - 1):
            self.weights.append(\
                (2*np.random.random((layers[i - 1] + 1, layers[i] + 1))-1)*0.25)
            self.weights.append(\
                (2*np.random.random((layers[i] + 1, layers[i + 1]))-1)*0.25)
    def fit(self,X,y,learning_rate=0.2,epochs=10000):
        X = np.atleast_2d(X) #Make sure that arrays with at least two dimensions
        temp = np.ones([X.shape[0],X.shape[1]+1])
        temp[:,0:-1] = X # Add bias terms
        X = temp
        y = np.array(y)

        for k in range(epochs):
            # Pick one instance to train network randomly
            # Stochastic gradient
            i = np.random.randint(X.shape[0])
            a = [X[i]]

            for layer in range(len(self.weights)):
                a.append(self.activation(np.dot(a[layer],self.weights[layer])))
            error = y[i] - a[-1]
            deltas = [error*self.activation_deriv(a[-1])]

            for layer in range(len(a)-2,0,-1):
                deltas.append(\
                    deltas[-1].dot(self.weights[layer].T)*self.activation_deriv(a[layer]))
            deltas.reverse()

            for i in range(len(self.weights)):
                layer = np.atleast_2d(a[i])
                delta = np.atleast_2d(deltas[i])
                self.weights[i] += learning_rate*layer.T.dot(delta)

    def predict(self,x):
        x = np.array(x)
        temp = np.ones(x.shape[0]+1)
        temp[0:-1] = x
        a = temp
        for layer in range(0,len(self.weights)):
            a = self.activation(np.dot(a,self.weights[layer]))
        return a


I am working on the file "Deep Learning/Tensorflow Basics.ipynb".
The first task is:
Now let's see an example of a more complex operation, using Matrix Multiplication. First we need to create the matrices:
Can you write Python code for it?

import numpy as np
# Make sure to use floats here, int64 will cause an error.
a = np.array([[5.0,5.0]])
b = np.array([[2.0],[2.0]])

a

a.shape

b

b.shape

mat1 = tf.constant(a)

mat2 = tf.constant(b)


I am working on the file "Deep Learning/Tensorflow Basics.ipynb".
The first task is:
Operations
You can line up multiple Tensorflow operations in to be run during a session:
Can you write Python code for it?

x = tf.constant(2)
y = tf.constant(3)

with tf.Session() as sess:
    print('Operations with Constants')
    print('Addition',sess.run(x+y))
    print('Subtraction',sess.run(x-y))
    print('Multiplication',sess.run(x*y))
    print('Division',sess.run(x/y))


I am working on the file "Deep Learning/Tensorflow Basics.ipynb".
The first task is:
Placeholder
You may not always have the constants right away, and you may be waiting for a constant to appear after a cycle of operations. tf.placeholder is a tool for this. It inserts a placeholder for a tensor that will be always fed.
Important: This tensor will produce an error if evaluated. Its value must be fed using the feed_dict optional argument to Session.run(),
Tensor.eval(), or Operation.run(). For example, for a placeholder of a matrix of floating point numbers:
x = tf.placeholder(tf.float32, shape=(1024, 1024))

Here is an example for integer placeholders:
Can you write Python code for it?

x = tf.placeholder(tf.int32)
y = tf.placeholder(tf.int32)

x

type(x)


I am working on the file "Deep Learning/Tensorflow Basics.ipynb".
The first task is:
Running operations with variable input:
Can you write Python code for it?

d = {x:20,y:30}

with tf.Session() as sess:
    print('Operations with Constants')
    print('Addition',sess.run(add,feed_dict=d))
    print('Subtraction',sess.run(sub,feed_dict=d))
    print('Multiplication',sess.run(mul,feed_dict=d))


I am working on the file "Deep Learning/Tensorflow Basics.ipynb".
The first task is:
Simple Constants
Let's show how to create a simple constant with Tensorflow, which TF stores as a tensor object:
Can you write Python code for it?

hello = tf.constant('Hello World')

type(hello)

x = tf.constant(100)

type(x)


I am working on the file "Deep Learning/Tensorflow Basics.ipynb".
The first task is:
The matrix multiplication operation:
Can you write Python code for it?


I am working on the file "Deep Neural Network Application Image Classification/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:
$12,288$ equals $64 \times 64 \times 3$ which is the size of one reshaped image vector.
3 - Architecture of your model
Now that you are familiar with the dataset, it is time to build a deep neural network to distinguish cat images from non-cat images.
You will build two different models:
- A 2-layer neural network
- An L-layer deep neural network
You will then compare the performance of these models, and also try out different values for $L$.
Let's look at the two architectures.
3.1 - 2-layer neural network

 Figure 2: 2-layer neural network.
 The model can be summarized as: INPUT -> LINEAR -> RELU -> LINEAR -> SIGMOID -> OUTPUT.

Detailed Architecture of figure 2:
- The input is a (64,64,3) image which is flattened to a vector of size $(12288,1)$.
- The corresponding vector: $[x_0,x_1,...,x_{12287}]^T$ is then multiplied by the weight matrix $W^{[1]}$ of size $(n^{[1]}, 12288)$.
- You then add a bias term and take its relu to get the following vector: $[a_0^{[1]}, a_1^{[1]},..., a_{n^{[1]}-1}^{[1]}]^T$.
- You then repeat the same process.
- You multiply the resulting vector by $W^{[2]}$ and add your intercept (bias).
- Finally, you take the sigmoid of the result. If it is greater than 0.5, you classify it to be a cat.
3.2 - L-layer deep neural network
It is hard to represent an L-layer deep neural network with the above representation. However, here is a simplified network representation:

 Figure 3: L-layer neural network.
 The model can be summarized as: [LINEAR -> RELU] $\times$ (L-1) -> LINEAR -> SIGMOID

Detailed Architecture of figure 3:
- The input is a (64,64,3) image which is flattened to a vector of size (12288,1).
- The corresponding vector: $[x_0,x_1,...,x_{12287}]^T$ is then multiplied by the weight matrix $W^{[1]}$ and then you add the intercept...
Can you write Python code for it?

### CONSTANTS DEFINING THE MODEL ####
n_x = 12288     # num_px * num_px * 3
n_h = 7
n_y = 1
layers_dims = (n_x, n_h, n_y)

# GRADED FUNCTION: two_layer_model

def two_layer_model(X, Y, layers_dims, learning_rate = 0.0075, num_iterations = 3000, print_cost=False):
    """
    Implements a two-layer neural network: LINEAR->RELU->LINEAR->SIGMOID.

    Arguments:
    X -- input data, of shape (n_x, number of examples)
    Y -- true "label" vector (containing 0 if cat, 1 if non-cat), of shape (1, number of examples)
    layers_dims -- dimensions of the layers (n_x, n_h, n_y)
    num_iterations -- number of iterations of the optimization loop
    learning_rate -- learning rate of the gradient descent update rule
    print_cost -- If set to True, this will print the cost every 100 iterations

    Returns:
    parameters -- a dictionary containing W1, W2, b1, and b2
    """

    np.random.seed(1)
    grads = {}
    costs = []                              # to keep track of the cost
    m = X.shape[1]                           # number of examples
    (n_x, n_h, n_y) = layers_dims

    # Initialize parameters dictionary, by calling one of the functions you'd previously implemented
    ### START CODE HERE ### (≈ 1 line of code)
    parameters = initialize_parameters(n_x, n_h, n_y)
    ### END CODE HERE ###

    # Get W1, b1, W2 and b2 from the dictionary parameters.
    W1 = parameters["W1"]
    b1 = parameters["b1"]
    W2 = parameters["W2"]
    b2 = parameters["b2"]

    # Loop (gradient descent)

    for i in range(0, num_iterations):

        # Forward propagation: LINEAR -> RELU -> LINEAR -> SIGMOID. Inputs: "X, W1, b1". Output: "A1, cache1, A2, cache2".
        ### START CODE HERE ### (≈ 2 lines of code)
        A1, cache1 = linear_activation_forward(X, W1, b1, "relu")
        A2, cache2 = linear_activation_forward(A1, W2, b2, "sigmoid")
        ### END CODE HERE ###

        # Compute cost
        ### START CODE HERE ### (≈ 1 line of code)
        cost = compute_cost(A2, Y)
     ...


I am working on the file "Deep Neural Network Application Image Classification/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:
2 - Dataset
You will use the same "Cat vs non-Cat" dataset as in "Logistic Regression as a Neural Network" (Assignment 2). The model you had built had 70% test accuracy on classifying cats vs non-cats images. Hopefully, your new model will perform a better!
Problem Statement: You are given a dataset ("data.h5") containing:
    - a training set of m_train images labelled as cat (1) or non-cat (0)
    - a test set of m_test images labelled as cat and non-cat
    - each image is of shape (num_px, num_px, 3) where 3 is for the 3 channels (RGB).
Let's get more familiar with the dataset. Load the data by running the cell below.
Can you write Python code for it?

train_x_orig, train_y, test_x_orig, test_y, classes = load_data()


I am working on the file "Deep Neural Network Application Image Classification/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:

    **Train Accuracy**

    0.985645933014

Can you write Python code for it?

pred_test = predict(test_x, test_y, parameters)


I am working on the file "Deep Neural Network Application Image Classification/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:
As usual, you reshape and standardize the images before feeding them to the network. The code is given in the cell below.

 Figure 1: Image to vector conversion.

Can you write Python code for it?

# Reshape the training and test examples
train_x_flatten = train_x_orig.reshape(train_x_orig.shape[0], -1).T   # The "-1" makes reshape flatten the remaining dimensions
test_x_flatten = test_x_orig.reshape(test_x_orig.shape[0], -1).T

# Standardize data to have feature values between 0 and 1.
train_x = train_x_flatten/255.
test_x = test_x_flatten/255.

print ("train_x's shape: " + str(train_x.shape))
print ("test_x's shape: " + str(test_x.shape))


I am working on the file "Deep Neural Network Application Image Classification/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:
Expected Output:

         **Accuracy**
         0.72

Note: You may notice that running the model on fewer iterations (say 1500) gives better accuracy on the test set. This is called "early stopping" and we will talk about it in the next course. Early stopping is a way to prevent overfitting.
Congratulations! It seems that your 2-layer neural network has better performance (72%) than the logistic regression implementation (70%, assignment week 2). Let's see if you can do even better with an $L$-layer model.
5 - L-layer Neural Network
Question: Use the helper functions you have implemented previously to build an $L$-layer neural network with the following structure: [LINEAR -> RELU]$\times$(L-1) -> LINEAR -> SIGMOID. The functions you may need and their inputs are:
python
def initialize_parameters_deep(layer_dims):
    ...
    return parameters
def L_model_forward(X, parameters):
    ...
    return AL, caches
def compute_cost(AL, Y):
    ...
    return cost
def L_model_backward(AL, Y, caches):
    ...
    return grads
def update_parameters(parameters, grads, learning_rate):
    ...
    return parameters
Can you write Python code for it?

### CONSTANTS ###
layers_dims = [12288, 20, 7, 5, 1] #  5-layer model

# GRADED FUNCTION: L_layer_model

def L_layer_model(X, Y, layers_dims, learning_rate = 0.0075, num_iterations = 3000, print_cost=False):#lr was 0.009
    """
    Implements a L-layer neural network: [LINEAR->RELU]*(L-1)->LINEAR->SIGMOID.

    Arguments:
    X -- data, numpy array of shape (number of examples, num_px * num_px * 3)
    Y -- true "label" vector (containing 0 if cat, 1 if non-cat), of shape (1, number of examples)
    layers_dims -- list containing the input size and each layer size, of length (number of layers + 1).
    learning_rate -- learning rate of the gradient descent update rule
    num_iterations -- number of iterations of the optimization loop
    print_cost -- if True, it prints the cost every 100 steps

    Returns:
    parameters -- parameters learnt by the model. They can then be used to predict.
    """

    np.random.seed(1)
    costs = []                         # keep track of cost

    # Parameters initialization.
    ### START CODE HERE ###
    parameters = initialize_parameters_deep(layers_dims)
    ### END CODE HERE ###

    # Loop (gradient descent)
    for i in range(0, num_iterations):

        # Forward propagation: [LINEAR -> RELU]*(L-1) -> LINEAR -> SIGMOID.
        ### START CODE HERE ### (≈ 1 line of code)
        AL, caches = L_model_forward(X, parameters)
        ### END CODE HERE ###

        # Compute cost.
        ### START CODE HERE ### (≈ 1 line of code)
        cost = compute_cost(AL, Y)
        ### END CODE HERE ###

        # Backward propagation.
        ### START CODE HERE ### (≈ 1 line of code)
        grads = L_model_backward(AL, Y, caches)
        ### END CODE HERE ###

        # Update parameters.
        ### START CODE HERE ### (≈ 1 line of code)
        parameters = update_parameters(parameters, grads, learning_rate)
        ### END CODE HERE ###

        # Print the cost every 100 training example
        if print_cost and i % 100 ==...


I am working on the file "Deep Neural Network Application Image Classification/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:
Expected Output:

         **Accuracy**
         1.0

Can you write Python code for it?

predictions_test = predict(test_x, test_y, parameters)


I am working on the file "Deep Neural Network Application Image Classification/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:
Expected Output:

         **Cost after iteration 0**
         0.6930497356599888

         **Cost after iteration 100**
         0.6464320953428849

         **...**
         ...

         **Cost after iteration 2400**
         0.048554785628770206

Good thing you built a vectorized implementation! Otherwise it might have taken 10 times longer to train this.
Now, you can use the trained parameters to classify images from the dataset. To see your predictions on the training and test sets, run the cell below.
Can you write Python code for it?

predictions_train = predict(train_x, train_y, parameters)


I am working on the file "Deep Neural Network Application Image Classification/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:
Run the cell below to train your parameters. See if your model runs. The cost should be decreasing. It may take up to 5 minutes to run 2500 iterations. Check if the "Cost after iteration 0" matches the expected output below, if not click on the square (⬛) on the upper bar of the notebook to stop the cell and try to find your error.
Can you write Python code for it?

parameters = two_layer_model(train_x, train_y, layers_dims = (n_x, n_h, n_y), num_iterations = 2500, print_cost=True)


I am working on the file "Deep Neural Network Application Image Classification/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:
The following code will show you an image in the dataset. Feel free to change the index and re-run the cell multiple times to see other images.
Can you write Python code for it?

# Example of a picture
index = 13
plt.imshow(train_x_orig[index])
print ("y = " + str(train_y[0,index]) + ". It's a " + classes[train_y[0,index]].decode("utf-8") +  " picture.")

# Explore your dataset
m_train = train_x_orig.shape[0]
num_px = train_x_orig.shape[1]
m_test = test_x_orig.shape[0]

print ("Number of training examples: " + str(m_train))
print ("Number of testing examples: " + str(m_test))
print ("Each image is of size: (" + str(num_px) + ", " + str(num_px) + ", 3)")
print ("train_x_orig shape: " + str(train_x_orig.shape))
print ("train_y shape: " + str(train_y.shape))
print ("test_x_orig shape: " + str(test_x_orig.shape))
print ("test_y shape: " + str(test_y.shape))


I am working on the file "Deep Neural Network Application Image Classification/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:
You will now train the model as a 5-layer neural network.
Run the cell below to train your model. The cost should decrease on every iteration. It may take up to 5 minutes to run 2500 iterations. Check if the "Cost after iteration 0" matches the expected output below, if not click on the square (⬛) on the upper bar of the notebook to stop the cell and try to find your error.
Can you write Python code for it?

parameters = L_layer_model(train_x, train_y, layers_dims, num_iterations = 2500, print_cost = True)


I am working on the file "Deep Neural Networks/tf-deep-neural-networks.ipynb".
The first task is:
Deep Neural Network in TensorFlow
We've seen how to build a logistic classifier using TensorFlow. Now we're going to see how to use the logistic classifier to build a deep neural network.
In the following walkthrough, we'll step through TensorFlow code written to classify the letters in the MNIST database.
Code
TensorFlow MNIST
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets(".", one_hot=True, reshape=False)


I am working on the file "Deep Neural Networks/tf-deep-neural-networks.ipynb".
The first task is:
Deep neural networks use multiple layers with each layer requiring it's own weight and bias. The 'hidden_layer' weight and bias is for the hidden layer. The 'out' weight and bias is for the output layer. If the neural network were deeper, there would be weights and biases for each additional layer.
Input
Can you write Python code for it?

# tf Graph input
x = tf.placeholder("float", [None, 28, 28, 1])
y = tf.placeholder("float", [None, n_classes])

x_flat = tf.reshape(x, [-1, n_input])


I am working on the file "Deep Neural Networks/tf-deep-neural-networks.ipynb".
The first task is:
Hidden Layer Parameters
Can you write Python code for it?

n_hidden_layer = 256 # layer number of features


I am working on the file "Deep Neural Networks/tf-deep-neural-networks.ipynb".
The first task is:
Let's train that model, then save the weights:
Can you write Python code for it?

import math

save_file = 'train_model.ckpt'
batch_size = 128
n_epochs = 100

saver = tf.train.Saver()

# Launch the graph
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    # Training cycle
    for epoch in range(n_epochs):
        total_batch = math.ceil(mnist.train.num_examples / batch_size)

        # Loop over all batches
        for i in range(total_batch):
            batch_features, batch_labels = mnist.train.next_batch(batch_size)
            sess.run(
                optimizer,
                feed_dict={features: batch_features, labels: batch_labels})

        # Print status for every 10 epochs
        if epoch % 10 == 0:
            valid_accuracy = sess.run(
                accuracy,
                feed_dict={
                    features: mnist.validation.images,
                    labels: mnist.validation.labels})
            print('Epoch {:<3} - Validation Accuracy: {}'.format(
                epoch,
                valid_accuracy))

    # Save the model
    saver.save(sess, save_file)
    print('Trained Model Saved.')


I am working on the file "Deep Neural Networks/tf-deep-neural-networks.ipynb".
The first task is:
Load a Trained Model
Let's load the weights and bias from memory, then check the test accuracy.
Can you write Python code for it?

saver = tf.train.Saver()

# Launch the graph
with tf.Session() as sess:
    saver.restore(sess, save_file)

    test_accuracy = sess.run(
        accuracy,
        feed_dict={features: mnist.test.images, labels: mnist.test.labels})

print('Test Accuracy: {}'.format(test_accuracy))


I am working on the file "Deep Neural Networks/tf-deep-neural-networks.ipynb".
The first task is:
TensorFlow Dropout

Dropout is a regularization technique for reducing overfitting. The technique temporarily drops units (artificial neurons) from the network, along with all of those units' incoming and outgoing connections. Figure 1 illustrates how dropout works.
TensorFlow provides the tf.nn.dropout() function, which you can use to implement dropout.
Let's look at an example of how to use tf.nn.dropout().
Can you write Python code for it?

keep_prob = tf.placeholder(tf.float32) # probability to keep units

hidden_layer = tf.add(tf.matmul(features, weights[0]), biases[0])
hidden_layer = tf.nn.relu(hidden_layer)
hidden_layer = tf.nn.dropout(hidden_layer, keep_prob)

logits = tf.add(tf.matmul(hidden_layer, weights[1]), biases[1])


I am working on the file "Deep Neural Networks/tf-deep-neural-networks.ipynb".
The first task is:
That's it! You now know how to save and load a trained model in TensorFlow. Let's look at loading weights and biases into modified models in the next section.
Loading the Weights and Biases into a New Model
Sometimes you might want to adjust, or "finetune" a model that you have already trained and saved.
However, loading saved Variables directly into a modified model can generate errors. Let's go over how to avoid these problems.
Naming Error
TensorFlow uses a string identifier for Tensors and Operations called name. If a name is not given, TensorFlow will create one automatically. TensorFlow will give the first node the name <Type>, and then give the name <Type>_<number> for the subsequent nodes. Let's see how this can affect loading a model with a different order of weights and bias:
Can you write Python code for it?

import tensorflow as tf

# Remove the previous weights and bias
tf.reset_default_graph()

save_file = 'model.ckpt'

# Two Tensor Variables: weights and bias
weights = tf.Variable(tf.truncated_normal([2, 3]))
bias = tf.Variable(tf.truncated_normal([3]))

saver = tf.train.Saver()

# Print the name of Weights and Bias
print('Save Weights: {}'.format(weights.name))
print('Save Bias: {}'.format(bias.name))

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    saver.save(sess, save_file)

# Remove the previous weights and bias
tf.reset_default_graph()

# Two Variables: weights and bias
bias = tf.Variable(tf.truncated_normal([3]))
weights = tf.Variable(tf.truncated_normal([2, 3]))

saver = tf.train.Saver()

# Print the name of Weights and Bias
print('Load Weights: {}'.format(weights.name))
print('Load Bias: {}'.format(bias.name))

with tf.Session() as sess:
    # Load the weights and bias - ERROR
    saver.restore(sess, save_file)


I am working on the file "Deep Neural Networks/tf-deep-neural-networks.ipynb".
The first task is:
The MNIST data is made up of 28px by 28px images with a single [channel](https://en.wikipedia.org/wiki/Channel_(digital_image%29). The tf.reshape() function above reshapes the 28px by 28px matrices in x into vectors of 784px by 1px
Multilayer Perceptron
Can you write Python code for it?

# Hidden layer with RELU activation
layer_1 = tf.add(tf.matmul(x_flat, weights['hidden_layer']), biases['hidden_layer'])
layer_1 = tf.nn.relu(layer_1)
# Output layer with linear activation
logits = tf.add(tf.matmul(layer_1, weights['out']), biases['out'])


I am working on the file "Deep Neural Networks/tf-deep-neural-networks.ipynb".
The first task is:
The MNIST library in TensorFlow provides the ability to receive the dataset in batches. Calling the mnist.train.next_batch() function returns a subset of the training data.
Deeper Neural Network

That's it! Going from one layer to two is easy. Adding more layers to the network allows you to solve more complicated problems.
Save and Restore TensorFlow Models
Training a model can take hours. But once you close your TensorFlow session, you lose all the trained weights and biases. If you were to reuse the model in the future, you would have to train it all over again!
Fortunately, TensorFlow gives you the ability to save your progress using a class called tf.train.Saver. This class provides the functionality to save any tf.Variable to your file system.
Saving Variables
Let's start with a simple example of saving weights and bias Tensors. For the first example we'll just save two variables. Later examples we'll save all the weights in a practical model.
Can you write Python code for it?

import tensorflow as tf

# The file path to save the data
save_file = 'model.ckpt'

# Two Tensor Variables: weights and bias
weights = tf.Variable(tf.truncated_normal([2, 3]))
bias = tf.Variable(tf.truncated_normal([3]))

# Class used to save and/or restore Tensor Variables
saver = tf.train.Saver()

with tf.Session() as sess:
    # Initialize all the Variables
    sess.run(tf.global_variables_initializer())

    # Show the values of weights and bias
    print('Weights:')
    print(sess.run(weights))
    print('Bias:')
    print(sess.run(bias))

    # Save the model
    saver.save(sess, save_file)


I am working on the file "Deep Neural Networks/tf-deep-neural-networks.ipynb".
The first task is:
The Tensors weights and bias are set to random values using the tf.truncated_normal() function. The values are then saved to the save_file location, "model.ckpt", using the tf.train.Saver.save() function. (The ".ckpt" extension stands for "checkpoint".)
Loading Variables
Now that the Tensor Variables are saved, let's load them back into a new model.
Can you write Python code for it?

# Remove the previous weights and bias
tf.reset_default_graph()

# Two Variables: weights and bias
weights = tf.Variable(tf.truncated_normal([2, 3]))
bias = tf.Variable(tf.truncated_normal([3]))

# Class used to save and/or restore Tensor Variables
saver = tf.train.Saver()

with tf.Session() as sess:
    # Load the weights and bias
    saver.restore(sess, save_file)

    # Show the values of weights and bias
    print('Weight:')
    print(sess.run(weights))
    print('Bias:')
    print(sess.run(bias))


I am working on the file "Deep Neural Networks/tf-deep-neural-networks.ipynb".
The first task is:
The above code applies the tf.nn.relu() function to the hidden_layer, effectively turning off any negative weights and acting like an on/off switch. Adding additional layers, like the output layer, after an activation function turns the model into a nonlinear function. This nonlinearity allows the network to solve more complex problems.
Quiz
Below you'll use the ReLU function to turn a linear single layer network into a non-linear multilayer network.
Can you write Python code for it?

import tensorflow as tf

output = None
hidden_layer_weights = [
    [0.1, 0.2, 0.4],
    [0.4, 0.6, 0.6],
    [0.5, 0.9, 0.1],
    [0.8, 0.2, 0.8]]
out_weights = [
    [0.1, 0.6],
    [0.2, 0.1],
    [0.7, 0.9]]

# Weights and biases
weights = [
    tf.Variable(hidden_layer_weights),
    tf.Variable(out_weights)]
biases = [
    tf.Variable(tf.zeros(3)),
    tf.Variable(tf.zeros(2))]

# Input
features = tf.Variable([[1.0, 2.0, 3.0, 4.0], [-1.0, -2.0, -3.0, -4.0], [11.0, 12.0, 13.0, 14.0]])

# TODO: Create Model
hidden_layer = tf.add(tf.matmul(features, weights[0]), biases[0])
hidden_layer = tf.nn.relu(hidden_layer)

output = tf.add(tf.matmul(hidden_layer, weights[1]), biases[1])

# TODO: Print session results
init = tf.global_variables_initializer()

with tf.Session() as session:
    session.run(init)
    session.run(hidden_layer)
    output = session.run(output)
    print(output)


I am working on the file "Deep Neural Networks/tf-deep-neural-networks.ipynb".
The first task is:
The code above illustrates how to apply dropout to a neural network.
The tf.nn.dropout() function takes in two parameters:

hidden_layer: the tensor to which you would like to apply dropout
keep_prob: the probability of keeping (i.e. not dropping) any given unit

keep_prob allows you to adjust the number of units to drop. In order to compensate for dropped units, tf.nn.dropout() multiplies all units that are kept (i.e. not dropped) by 1/keep_prob.
During training, a good starting value for keep_prob is 0.5.
During testing, use a keep_prob value of 1.0 to keep all units and maximize the power of the model.
Quiz
This quiz will be starting with the code from the ReLU Quiz and applying a dropout layer. Build a model with a ReLU layer and dropout layer using the keep_prob placeholder to pass in a probability of 0.5. Print the logits from the model.
Note: Output will be different every time the code is run. This is caused by dropout randomizing the units it drops.
Can you write Python code for it?

# Solution is available in the other "solution.py" tab
import tensorflow as tf

hidden_layer_weights = [
    [0.1, 0.2, 0.4],
    [0.4, 0.6, 0.6],
    [0.5, 0.9, 0.1],
    [0.8, 0.2, 0.8]]
out_weights = [
    [0.1, 0.6],
    [0.2, 0.1],
    [0.7, 0.9]]

# Weights and biases
weights = [
    tf.Variable(hidden_layer_weights),
    tf.Variable(out_weights)]
biases = [
    tf.Variable(tf.zeros(3)),
    tf.Variable(tf.zeros(2))]

# Input
features = tf.Variable([[0.0, 2.0, 3.0, 4.0], [0.1, 0.2, 0.3, 0.4], [11.0, 12.0, 13.0, 14.0]])

# TODO: Create Model with Dropout
keep_prob = tf.placeholder(tf.float32)

hidden_layer = tf.add(tf.matmul(features, weights[0]), biases[0])
hidden_layer = tf.nn.relu(hidden_layer)
hidden_layer = tf.nn.dropout(hidden_layer, keep_prob)

logits = tf.add(tf.matmul(hidden_layer, weights[1]), biases[1])

# TODO: Print logits from a session
with tf.Session() as session:
    session.run(tf.global_variables_initializer())

    output = session.run(logits, feed_dict={keep_prob: 0.5})
    print(output)


I am working on the file "Deep Neural Networks/tf-deep-neural-networks.ipynb".
The first task is:
The variable n_hidden_layer determines the size of the hidden layer in the neural network. This is also known as the width of a layer.
Weights and Biases
Can you write Python code for it?

# Store layers weight & bias
weights = {
    'hidden_layer': tf.Variable(tf.random_normal([n_input, n_hidden_layer])),
    'out': tf.Variable(tf.random_normal([n_hidden_layer, n_classes]))
}
biases = {
    'hidden_layer': tf.Variable(tf.random_normal([n_hidden_layer])),
    'out': tf.Variable(tf.random_normal([n_classes]))
}


I am working on the file "Deep Neural Networks/tf-deep-neural-networks.ipynb".
The first task is:
This is the same optimization technique used in the Intro to TensorFLow lab.
Session
Can you write Python code for it?

# Initializing the variables
init = tf.global_variables_initializer()

# Launch the graph
with tf.Session() as sess:
    sess.run(init)
    # Training cycle
    for epoch in range(training_epochs):
        total_batch = int(mnist.train.num_examples/batch_size)
        # Loop over all batches
        for i in range(total_batch):
            batch_x, batch_y = mnist.train.next_batch(batch_size)
            # Run optimization op (backprop) and cost op (to get loss value)
            sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})


I am working on the file "Deep Neural Networks/tf-deep-neural-networks.ipynb".
The first task is:
We've seen the linear function tf.add(tf.matmul(x_flat, weights['hidden_layer']), biases['hidden_layer']) before, also known as xw + b. Combining linear functions together using a ReLU will give you a two layer network.
Optimizer
Can you write Python code for it?

# Define loss and optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))
optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(cost)


I am working on the file "Deep Neural Networks/tf-deep-neural-networks.ipynb".
The first task is:
You'll notice that the name properties for weights and bias are different than when you saved the model. This is why the code produces the "Assign requires shapes of both tensors to match" error. The code saver.restore(sess, save_file) is trying to load weight data into bias and bias data into weights.
Instead of letting TensorFlow set the name property, let's set it manually:
Can you write Python code for it?

import tensorflow as tf

tf.reset_default_graph()

save_file = 'model.ckpt'

# Two Tensor Variables: weights and bias
weights = tf.Variable(tf.truncated_normal([2, 3]), name='weights_0')
bias = tf.Variable(tf.truncated_normal([3]), name='bias_0')

saver = tf.train.Saver()

# Print the name of Weights and Bias
print('Save Weights: {}'.format(weights.name))
print('Save Bias: {}'.format(bias.name))

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    saver.save(sess, save_file)

# Remove the previous weights and bias
tf.reset_default_graph()

# Two Variables: weights and bias
bias = tf.Variable(tf.truncated_normal([3]), name='bias_0')
weights = tf.Variable(tf.truncated_normal([2, 3]) ,name='weights_0')

saver = tf.train.Saver()

# Print the name of Weights and Bias
print('Load Weights: {}'.format(weights.name))
print('Load Bias: {}'.format(bias.name))

with tf.Session() as sess:
    # Load the weights and bias - No Error
    saver.restore(sess, save_file)

print('Loaded Weights and Bias successfully.')


I am working on the file "Deep XOR.ipynb".
The first task is:
Gradient of the hidden layer
We have computed the partial derivative with respect to our last layer, now let's do the same with respect to our hidden layer. This will help us when obtaining the partial derivatives to our trainable coefficients, which is ultimately what we want to obtain.
Again, let's apply the chain-rule to derive this:
$$\begin{split}
\frac{\partial L}{\partial H} & = \frac{\partial L}{\partial \hat{Y}} \frac{\partial \hat{Y}}{\partial Z} \frac{\partial Z}{\partial H} \
& = \frac{\hat{Y} - Y}{N} \frac{\partial Z}{\partial H} \
& = \frac{\hat{Y} - Y}{N} \frac{\partial}{\partial H} H W_2 + b_2 \
& = \frac{\hat{Y} - Y}{N} W_2^T\
\end{split}$$
Including the ReLU activation function $\sigma$ from the first layer will simplify the math later for backpropagation, let's do it:
$$\begin{split}
\frac{\partial L}{\partial \sigma} & = \frac{\partial L}{\partial \hat{Y}} \frac{\partial \hat{Y}}{\partial Z} \frac{\partial Z}{\partial H}\frac{\partial H}{\partial \sigma} \
& = \frac{\hat{Y} - Y}{N} W_2^T \frac{\partial H}{\partial \sigma}\
& = \frac{\hat{Y} - Y}{N} W_2^T \frac{\partial}{\partial \sigma}\sigma(X W_1 + b_1)\
& = \frac{\hat{Y} - Y}{N} W_2^T \frac{\partial}{\partial \sigma}\text{max}{\mathbf{0}, X W_1 + b_1}\
& = \frac{\hat{Y} - Y}{N} W_2^T \mathbb{1}{X W_1 + b_1 > 0}\
\end{split}$$
As you can see, the derivative of a ReLu function $\mbox{max}{0, x}$ is simply the indicator $\mathbb{1}{x > 0}$. If this is not obvious, here's a further explanation.
Since we know that this $\mathbb{1}{X W_1 + b_1 > 0}$ will always be true as long as $\mathbb{1}{H > 0}$ is true (since $H$ is already rectified), we can further simplify the formula by:
$$\begin{split}
\frac{\partial L}{\partial \sigma} = \frac{\hat{Y} - Y}{N} W_2^T \mathbb{1}{X W_1 + b_1 > 0} = \frac{\hat{Y} - Y}{N} W_2^T \mathbb{1}{H > 0}\
\end{split}$$
Let's put this derivative in a function.
Can you write Python code for it?

def compute_hidden_diff(H, W2, dloss):
    """Computes the derivative of the first hidden layer, including it's activation.

    Parameters
    ----------
    H: np.ndarray((N, n_hidden))
        Output of the hidden layer.
    W2: np.ndarray((n_hidden, 2))
        Coefficients of the hidden to output layer.
    dloss: np.ndarray((N, 2))
        Derivative of the output layer.

    Returns
    -------
    dhidden: np.ndarray((N, n_hidden))
        Derivative of the hidden layer.
    """
    dhidden = np.dot(dloss, W2.T)
    dhidden[H <= 0] = 0  # ReLU backprop
    return dhidden


I am working on the file "Deep XOR.ipynb".
The first task is:
Gradient of the trainable coefficients
Now that we have the derivative of the loss function with respect to the output (second layer), we can easily compute the gradient of the trainable coefficients $W_2$ and $b_2$ using backpropagation. Let's do it:
$$\begin{split}
\frac{\partial L}{\partial W_2} & = \frac{\partial L}{\partial \hat{Y}} \frac{\partial \hat{Y}}{\partial Z} \frac{\partial Z}{\partial W_2} \
& = \frac{\hat{Y} - Y}{N} \frac{\partial Z}{\partial W_2} \
& = \frac{\hat{Y} - Y}{N} \frac{\partial}{\partial W_2} H W_2 + b_2 \
& = H^T \frac{\hat{Y} - Y}{N}\
\
\frac{\partial L}{\partial b_2} & = \frac{\partial L}{\partial \hat{Y}} \frac{\partial \hat{Y}}{\partial Z} \frac{\partial Z}{\partial b_2} \
& = \frac{\hat{Y} - Y}{N} \frac{\partial}{\partial b_2} H W_2 + b_2 \
& = \frac{\hat{Y} - Y}{N}\
\end{split}$$
Analogously, since we have computed the derivative of the loss function with respect to the hidden layer, we can now compute the one with respect to the rest of trainable coefficients $W_1$ and $b_1$:
$$\begin{split}
\frac{\partial L}{\partial W_1} & = \frac{\partial L}{\partial \hat{Y}} \frac{\partial \hat{Y}}{\partial Z} \frac{\partial Z}{\partial H}\frac{\partial H}{\partial \sigma}\frac{\partial \sigma}{\partial W_1} \
& = \frac{\partial L}{\partial \sigma} \frac{\partial}{\partial W_1} X W_1 + b_1\
& = X^T \frac{\partial L}{\partial \sigma}\
& = X^T \frac{\hat{Y} - Y}{N} W_2^T \mathbb{1}{H > 0}\
\
\frac{\partial L}{\partial b_1} & = \frac{\partial L}{\partial \hat{Y}} \frac{\partial \hat{Y}}{\partial Z} \frac{\partial Z}{\partial H}\frac{\partial H}{\partial \sigma}\frac{\partial \sigma}{\partial b_1} \
& = \frac{\partial L}{\partial \sigma} \frac{\partial}{\partial b_1} X W_1 + b_1\
& = \frac{\partial L}{\partial \sigma}\
& = \frac{\hat{Y} - Y}{N} W_2^T \mathbb{1}{H > 0}\
\end{split}$$
All of which could be modularized and implemented in a backpropagation function like this:
Can you write Python code for it?

def backprop(x, dx):
    """Computes the backpropagation for a given layer.

    Parameters
    ----------
    x: np.ndarray((N, n_in))
        Input layer.
    dx: np.ndarray((N, n_out))
        Derivative of the input layer.

    Returns
    -------
    dW: np.ndarray((n_in, n_out))
        Derivative of the loss with respect to the W matrix.
    db: np.ndarray((1, n_out))
        Derivative of the loss with respect to b.
    """
    dW = np.dot(x.T, dx)
    db = np.sum(dx, axis=0, keepdims=True)
    return dW, db


I am working on the file "Deep XOR.ipynb".
The first task is:
The End
So that's it. We have trained a feedwordard neural network with a hidden layer to model an XOR gate. We have implemented it using numpy. And it worked (at least most of the times).
I'm pretty sure there might be some bugs in the code / math. But that's why this is here, to learn with you, titans. Thus, pull requests are welcome :)
Bonus: XOR, Y U Not Work With Linear Models?
As explained in Section 6.2 of the Deep Learning book, the XOR operation is not linear. Therefore, it is not possible to model it with simple linear regression. To illustrate this, we can actually try to fit one of these simple models and find the results analytically.
We are gonna frame this as regression (instead of classification), so let's say that $\mathbf{y} \in \mathbb{R}^N$ are the labels (notice how this is now a simple vector instead of an $N \times 2$ matrix) and $X \in \mathbb{R}^{N \times 2}$ the observations set (as before). The model we want to approximate is:
$$ \mathbf{\hat{y}} = X \mathbf{w} + b $$
where $\mathbf{w} \in \mathbb{R}^2$ and $b \in \mathbb{R}$ are the trainable coefficients.
To simplify the math we can include the bias as an extra coefficient in $\mathbf{w}$ by adding a column of ones in $X$, resulting in:
$$ \mathbf{\hat{y}} = X \mathbf{w}$$
where $X \in \mathbb{R}^{N \times 3}$ and $\mathbf{w} \in \mathbb{R}^3$.
Let the "loss function" in this case be the Mean Squared Error (MSE), which is standard for regression problems:
$$\text{MSE} = \frac{1}{N} \sum_i^N (y_i - \hat{y}_i)^2 = \frac{1}{N} (\mathbf{y} - \mathbf{\hat{y}})^2$$
We can derive the set of normal equations to solve this optimization problem analytically:
$$\begin{equation}
\text{argmin}_{\mathbf{w}} \text{MSE}\
\frac{\partial \text{MSE}}{\partial \mathbf{w}} = 0\
\frac{\partial}{\partial \mathbf{w}} \frac{1}{N} (\mathbf{y} - (X \mathbf{w}))^2 = 0 \
\frac{\partial}{\partial \mathbf{w}} \mathbf{y}^T\mathbf{y} - 2 \mathbf{y}^TX \mathbf{w} + (X\mathbf{w})^2 = 0 \
-2 X^T\mathbf{y} + 2X^TX\mathbf{w} = 0 \
-X^T\mathbf{y} + X^T X\mathbf{w} = 0...
Can you write Python code for it?

# Normal equations, solving with the four unique examples:
X = np.array([[1, 0, 0], [1, 0, 1], [1, 1, 0], [1, 1, 1]])  # Added the bias as ones in the first column
y = np.array([0, 1, 1, 0])
b, w1, w2 = np.dot(np.linalg.inv(np.dot(X.T, X)), np.dot(X.T, y))
print("b = {}, w = [{}, {}]".format(b, w1, w2))


I am working on the file "Deep XOR.ipynb".
The first task is:
The Loss Function
Next step is to define an appropriate differentiable loss function that allows us to train our network. The loss should be small when we are obtaining successful results with the current parameters, and high otherwise.
The cross-entropy function is commonly used in classification problems, since its derivative is quite simple and yet successfully captures the cost of the network for a given set of parameters.
The average cross-entropy is defined as:
$$L = - \frac{1}{N}\sum_{i}^N \sum_{j}^2 y_{ij} \log(\hat{y}_{ij}) $$
Let's quickly inspect this function. If we have a perfect prediction (e.g., $\mathbf{y}_i = {0, 1}$ and $\mathbf{\hat{y}}_i = {0, 1}$), then $L_i$ is 0, as desired. Whereas, if we have completely wrong one (e.g., $\mathbf{y}_i = {0, 1}$ and $\mathbf{\hat{y}}_i = {1, 0}$), then we get $L_i = \infty$, also as required. Alrighten.
We can now implement this function in numpy. To avoid the instability of computing $\log(0)$, we can simply multiply each of the predicted results with the target value, and sum across them (one will be zero). This is essentially like "selecting" the likelihood of the correct class:
$$L = - \frac{1}{N}\sum_{i}^N y_{ik} \log(\hat{y}{ik}) + (1 - y{ik}) \log(\hat{y}{in}) = - \frac{1}{N}\sum{i}^N \log(\hat{y}{ik}) \qquad\text{where}\qquad y{ik} = 1$$
And here its implementation:
Can you write Python code for it?

def compute_loss(Y, Y_est):
    """Computes the averate cross-entropy loss.

    Parameters
    ----------
    Y: np.ndarray((N, 2))
        Target labels (XOR correct results).
    Y_est: np.ndarray((N, 2))
        Estimated XOR values.

    Returns
    -------
    loss: float
        Loss value: the average cross-entropy.
    """
    N = Y.shape[0]
    return - np.sum(np.log(np.sum(Y * Y_est, axis=1))) / float(N)


I am working on the file "Deep XOR.ipynb".
The first task is:
The Training
As it is common in neural networks, we use gradient descent to train our set of weights $W_1, W2$ and biases $b_1, b_2$.
This method makes use of the partial derivates of the whole model to identify the steepest direction (the gradient) of each trainable variable in order to minimize, step by step, the given loss function $L$.
The chain-rule formula will help us in this process, backpropagating the gradient from the output of the network to its input. So, let us begin.
Gradient of the output layer
The (fun) derivative of our loss function with respect to the softmax output of our model is:
$$\begin{split}
\frac{\partial L}{\partial z_{ij}} & = \frac{\partial L}{\partial \hat{y}{ij}} \frac{\partial \hat{y}{ij}}{\partial z_{ij}} \
& = \frac{\partial}{\partial\hat{y}{ij}} \left( - \sum{j}^2 y_{ij} \log(\hat{y}{ij}) \right) \frac{\partial \hat{y}{ij}}{\partial z_{ij}}\
& = - \sum_{j}^2 y_{ij} \frac{1}{\hat{y}{ij}} \frac{\partial \hat{y}{ij}}{\partial z_{ij}} \
& = - \sum_{j}^2 y_{ij} \frac{1}{\hat{y}{ij}} \frac{\partial}{\partial z{ij}} \frac{e^{z_{ij}}}{\sum_m^2e^{z_{im}}} \
& = - \sum_{j}^2 y_{ij} \frac{1}{\hat{y}{ij}} \frac{e^{z{ij}}\sum_m^2 e^{z_{im}} - e^{z_{ij}}e^{z_{ij}}}{(\sum_m^2e^{z_{im}})^2} \
& = - y_{ik} \frac{1}{\hat{y}{ik}} \frac{e^{z{ik}}\sum_m^2 e^{z_{im}} - e^{z_{ik}}e^{z_{ik}}}{(\sum_m^2e^{z_{im}})^2} - (1 - y_{ik}) \frac{1}{\hat{y}{in}} \frac{e^{z{in}}\sum_m^2 e^{z_{im}} - e^{z_{in}}e^{z_{in}}}{(\sum_m^2e^{z_{im}})^2}\
& = - y_{ik} \frac{1}{\hat{y}{ik}} \frac{e^{z{ik}}}{\sum_m^2e^{z_{im}}} \frac{\sum_m^2 e^{z_{im}} - e^{z_{ik}}}{\sum_m^2e^{z_{im}}}\
& = - y_{ik} \frac{1}{\hat{y}{ik}} \hat{y}{ik} \frac{\sum_m^2 e^{z_{im}} - e^{z_{ik}}}{\sum_m^2e^{z_{im}}} \
& = - y_{ik} \left( \frac{\sum_m^2 e^{z_{im}}}{\sum_m^2e^{z_{im}}} - \frac{e^{z_{ik}}}{\sum_m^2e^{z_{im}}} \right)\
& = - y_{ik} ( 1 - \hat{y}{ik})\
& = - y{ik} + y_{ik}\hat{y}{ik}\
& = - y{ij} + 1\hat{y}{ij}\
& = \hat{y}{ij} - y_{ij}\
\end{split}$$
Applied to all our datapoints, the result would...
Can you write Python code for it?

def compute_loss_diff(Y, Y_est):
    """Computes the averate cross-entropy loss derivative.

    Parameters
    ----------
    Y: np.ndarray((N, 2))
        Target labels (XOR correct results).
    Y_est: np.ndarray((N, 2))
        Estimated XOR values.

    Returns
    -------
    dloss: np.ndarray((N, 2))
        Loss value: the average cross-entropy derivative.
    """
    return (Y_est - Y) / float(Y.shape[0])


I am working on the file "Deep XOR.ipynb".
The first task is:
The Whole Thing
Alrighten. The last thing to do is to put all the pieces together. We will do so in a single training loop, that will go over a few thousand epochs.
It may be interesting to play around with the step variable, to see how much faster (but more likely to fall into bad local minima) the whole thing will be if set high enough; and how slow (but less likely to fall into those terrible places) it would be with a sufficiently small step.
In the last line of this implementation I update the whole training set with new randomly generated data. While this is not necessary (and makes the whole training noticeably slower), it seems to help to obtain better results. Feel free to play with it (i.e., remove it or just update the set after a few epochs).
Another interesting parameter is the number of training observations N. While it often converges with 100 observations, that's not always the case. I guess the more observations, the more local minima around the gradient we'll find, but it might result in a faster convergence too (when it does converge). Also notice how sensible this parameter is in terms of computation efficiency.
Can you write Python code for it?

# Create some training data
N = 100
X, Y = gen_data(N)

# Initialize the coefficients randomly
W1, b1, W2, b2 = init_coefficients()

# Learning parameters
n_epochs = 10000
step = 0.01

# Main training loop
for n_epoch in range(n_epochs):
    # Forward pass of the network, catching the two layers' outputs
    H, Y_est = forward_pass(X, W1, W2, b1, b2)

    # Average cross-entropy loss
    loss = compute_loss(Y, Y_est)

    # Gradient of the second (output) layer
    dloss = compute_loss_diff(Y, Y_est)

    # Gradient of the hidden layer
    dhidden = compute_hidden_diff(H, W2, dloss)

    # Backpropagation over all trainable coefficients
    dW2, db2 = backprop(H, dloss)
    dW1, db1 = backprop(X, dhidden)

    # Update step
    W1 = update_step(W1, dW1, step)
    b1 = update_step(b1, db1, step)
    W2 = update_step(W2, dW2, step)
    b2 = update_step(b2, db2, step)

    # Printout
    if n_epoch % 1000 == 0:
        acc = np.sum(Y[np.arange(N), np.argmax(Y_est, axis=1)]) / float(N) # Accuracy
        print("\tEpoch %d: loss %f\t acc %.1f%%" % (n_epoch, loss, (acc * 100)))

    # Generate more data randomly (this might prevent falling into bad local minima)
    X, Y = gen_data(N)


I am working on the file "Deep learning loss functions.ipynb".
The first task is:
Apply:
Can you write Python code for it?

# Compute the range of values that we wish to explore
xs = np.arange(-5.0, 5.001, 1.0/128.0).astype(np.float32)

# Build an array of logit vector, where each logit vector is for a 2-class problem with the values [0, x[i]]
logits = np.stack([np.zeros_like(xs), xs], axis=1)

# Use softmax to compute predicted probabilities:
clf_q = f_softmax(logits)

# Compute negative log loss of softmax:
clf_loss = f_clf_loss(logits)

# Compute gradient of negative log loss of softmax with respect to the logits:
clf_loss_grad = f_clf_loss_grad(logits, np.ones_like(xs))

plt.figure(figsize=(5, 5))
plt.xlim(-5.0, 5.0)
plt.ylim(-5.0, 5.0)
line_p, = plt.plot(xs, np.ones_like(xs), label=r'$p$')
line_q, = plt.plot(xs, clf_q[:, 1], label=r'$q = softmax(X)$')
line_loss, = plt.plot(xs, clf_loss[:, 1], label=r'loss $c =-ln(q)$')
line_loss_grad, = plt.plot(xs, clf_loss_grad[:, 1], label=r'grad loss $\frac{dc}{dX_1}$')
plt.legend(handles=[line_p, line_q, line_loss, line_loss_grad])
plt.xlabel(r'$X_1$')
plt.show()

if SAVE_PLOTS:
    plt.figure(figsize=(5, 5))
    plt.xlim(-5.0, 5.0)
    plt.ylim(-5.0, 5.0)
    line_p, = plt.plot(xs, np.ones_like(xs), label=r'$p$')
    line_q, = plt.plot(xs, clf_q[:, 1], label=r'$q = softmax(X)$')
    plt.legend(handles=[line_p, line_q])
    plt.xlabel(r'$X_1$')
    plt.savefig('clf_loss_0.png', dpi=600)
    plt.close()

    plt.figure(figsize=(5, 5))
    plt.xlim(-5.0, 5.0)
    plt.ylim(-5.0, 5.0)
    line_p, = plt.plot(xs, np.ones_like(xs), label=r'$p$')
    line_q, = plt.plot(xs, clf_q[:, 1], label=r'$q = softmax(X)$')
    line_loss, = plt.plot(xs, clf_loss[:, 1], label=r'loss $c =-ln(q)$')
    plt.legend(handles=[line_p, line_q, line_loss])
    plt.xlabel(r'$X_1$')
    plt.savefig('clf_loss_1.png', dpi=600)
    plt.close()

    plt.figure(figsize=(5, 5))
    plt.xlim(-5.0, 5.0)
    plt.ylim(-5.0, 5.0)
    line_p, = plt.plot(xs, np.ones_like(xs), label=r'$p$')
    line_q, = plt.plot(xs, clf_q[:, 1], label=r'$q = softmax(X)$')
    line_loss, = plt.plot(xs, clf_loss[:, 1], label=r'loss $c =-ln(q)$')
   ...


I am working on the file "Deep learning loss functions.ipynb".
The first task is:
Classification: softmax non-linearity with negative log loss
Define a convenience function for computing the gradient of negative log loss with softmax. We use PyTorch to do it as it handles computing the gradient for us.
Can you write Python code for it?

# Softmax function
def f_softmax(logits, axis=1):
    ex = np.exp(logits)
    return ex / ex.sum(axis=axis, keepdims=True)

# Classification loss: negative log of softmax
def f_clf_loss(logits, axis=1):
    t_logits = torch.tensor(logits, requires_grad=True)
    # Compute negative log-softmax
    return -F.log_softmax(t_logits, dim=axis).detach().numpy()

# Gadient of classification loss
def f_clf_loss_grad(logits, target, axis=1):
    t_logits = torch.tensor(logits, requires_grad=True)
    t_targets = torch.tensor(target, dtype=torch.int64)
    # Compute cross_entropy loss
    loss = F.cross_entropy(t_logits, t_targets, reduction='sum')
    # Sum and compute gradient
    loss.backward()
    return t_logits.grad.detach().numpy()


I am working on the file "Deep learning loss functions.ipynb".
The first task is:
Regression: no non-linearity and Huber loss
Can you write Python code for it?

# Use PyTorch `smooth_l1_loss`
def f_regr_huber_loss(predictions, targets, delta=1.0):
    t_predictions = torch.tensor(predictions, requires_grad=True)
    t_targets = torch.tensor(targets, requires_grad=True)
    # Compute squared error
    return F.smooth_l1_loss(t_predictions, t_targets)

def f_regr_huber_loss_grad(predictions, targets, delta=1.0):
    t_predictions = torch.tensor(predictions, requires_grad=True)
    t_targets = torch.tensor(targets, requires_grad=True)
    # Compute squared error
    loss = F.smooth_l1_loss(t_predictions, t_targets)
    # Sum and compute gradient
    loss.sum().backward()
    return t_predictions.grad.detach().numpy()

# Compute the range of values that we wish to explore
xs = np.arange(-5.0, 5.0, 0.01).astype(np.float32)

# Use Huber loss:
regr_sqr_loss = [f_regr_huber_loss(x, 0.0) for x in xs]

# Compute gradient of Huber loss with respect to x-hat
regr_sqr_loss_grad = [f_regr_huber_loss_grad(x, 0.0) for x in xs]

plt.figure(figsize=(5,5))
plt.xlim(-5.0, 5.0)
plt.ylim(-5.0, 5.0)
line_loss, = plt.plot(xs, regr_sqr_loss, label=r'loss $c = huber(x, \hat{x})$')
line_loss_grad, = plt.plot(xs, regr_sqr_loss_grad, label=r'grad loss $\frac{dc}{dx}$')
plt.legend(handles=[line_loss, line_loss_grad])
plt.xlabel(r'$x$')
plt.show()

if SAVE_PLOTS:
    plt.figure(figsize=(5, 5))
    plt.xlim(-5.0, 5.0)
    plt.ylim(-5.0, 5.0)
    line_loss, = plt.plot(xs, regr_sqr_loss, label=r'loss $c = huber(x, \hat{x})$')
    line_loss_grad, = plt.plot(xs, regr_sqr_loss_grad, label=r'grad loss $\frac{dc}{dx}$')
    plt.legend(handles=[line_loss, line_loss_grad])
    plt.xlabel(r'$x$')
    plt.savefig('regr_huber_loss_2.png', dpi=600)
    plt.close()


I am working on the file "Deep learning loss functions.ipynb".
The first task is:
Regression: no non-linearity and squared error loss
Can you write Python code for it?

# Function for computing squared error loss
def f_regr_sqr_loss(a, b):
    return (a - b)**2

# Gadient of squared error loss
def f_regr_sqr_loss_grad(x_hat, x):
    t_x_hat = torch.tensor(x_hat, requires_grad=True)
    t_x = torch.tensor(x, requires_grad=True)
    # Compute squared error
    loss = -(t_x_hat - t_x)**2
    # Sum and compute gradient
    loss.sum().backward()
    return t_x.grad.detach().numpy()

# Compute the range of values that we wish to explore
xs = np.arange(-5.0, 5.0, 0.01).astype(np.float32)

# Use squared error loss:
regr_sqr_loss = [f_regr_sqr_loss(x, 0.0) for x in xs]

# Compute gradient of squared error with respect to x-hat
regr_sqr_loss_grad = [f_regr_sqr_loss_grad(x, 0.0) for x in xs]

plt.figure(figsize=(5,5))
plt.xlim(-5.0, 5.0)
plt.ylim(-5.0, 5.0)
line_loss, = plt.plot(xs, regr_sqr_loss, label=r'loss $c = (x - \hat{x})^2$')
line_loss_grad, = plt.plot(xs, regr_sqr_loss_grad, label=r'grad loss $\frac{dc}{dx}$')
plt.legend(handles=[line_loss, line_loss_grad])
plt.xlabel(r'$x$')
plt.show()

if SAVE_PLOTS:
    plt.figure(figsize=(5, 5))
    plt.xlim(-5.0, 5.0)
    plt.ylim(-5.0, 5.0)
    line_loss, = plt.plot(xs, regr_sqr_loss, label=r'loss $c = (x - \hat{x})^2$')
    line_loss_grad, = plt.plot(xs, regr_sqr_loss_grad, label=r'grad loss $\frac{dc}{dx}$')
    plt.legend(handles=[line_loss, line_loss_grad])
    plt.xlabel(r'$x$')
    plt.savefig('regr_sqr_loss_2.png', dpi=600)
    plt.close()


I am working on the file "Deep learning loss functions.ipynb".
The first task is:
We want to use Theano so that we can use it's auto-differentiation, since I'm too lazy to work out the derivatives of these functions by hand! :)
We also want to avoid the overhead of using the GPU for such small tasks, so tell Theano to use the CPU:
Can you write Python code for it?

import os

import numpy as np
import pandas as pd
import torch, torch.nn as nn, torch.nn.functional as F
from matplotlib import pyplot as plt
import seaborn as sns
sns.set()

EPSILON = 1.0e-12

SAVE_PLOTS = True


I am working on the file "Deep-Learning-Book-Review/Linear_Algebra_with_Python_and_NumPy.ipynb".
The first task is:
Euclidean norm of vector is returned by method numpy.linalg.norm
Can you write Python code for it?

norm = linalg.norm(a)      # Euclidean norm of vector a
print('a =', a)
print('norm(a) =', norm)

x = a/linalg.norm(a)       # Make normalized/unit vector from a
print('x =', x)
print('norm(x) =', linalg.norm(x))


I am working on the file "Deep-Learning-Book-Review/Linear_Algebra_with_Python_and_NumPy.ipynb".
The first task is:
For merging matrices or vectors methods numpy.hstack and numpy.vstack can be used.
Can you write Python code for it?

print(vstack([ A, ones([1,3]) ]))  # Add row vector to matrix
print(hstack([ A, ones([3,1]) ]))  # Add column vector to matrix
print(hstack([ A, eye(3) ]))       # Merge two matrices horizontally


I am working on the file "Deep-Learning-Book-Review/Linear_Algebra_with_Python_and_NumPy.ipynb".
The first task is:
If a vector was defined as 2D array of type matrix, transportation is not a problem.
Can you write Python code for it?

x = b.T           # Make column vector from vector b (defined as matrix)
print(x)
print(x.shape)    # Now size of column vector is 3x1
print(x.T)        # Make row vector by transpostion of column vector


I am working on the file "Deep-Learning-Book-Review/Linear_Algebra_with_Python_and_NumPy.ipynb".
The first task is:
Matrices can be defined as 2D arrays of type array or matrix (there is no problem with transposition with any type).
Can you write Python code for it?

A = array([[11,12,13], [21,22,23], [31,32,33]])  # Define matrix of size 3x3 as 2D 'array-type'
print(A)
print(A.shape)

B = matrix('11 12 13; 21 22 23; 31 32 33')  # Define matrix of size 3x3 as 'matrix-type'
print(B)
print(B.shape)

print(B[0,1])    # Get matrix element at row 0, column 1
print(B[0,:])    # Get 1st row of matrix (A[0] returns also 1st row)
print(B[:,0])    # Get 1st column of matrix

print(A[:,0])   # Note that column from 'array-type' matrix is returned as 1D array
print(B[:,0])   # Column from 'matrix-type' matrix is returned as true column as expected


I am working on the file "Deep-Learning-Book-Review/Linear_Algebra_with_Python_and_NumPy.ipynb".
The first task is:
Note that matrix and array elements in Python are indexed from 0, in contrast to Matlab where indexing starts from 1.
Can you write Python code for it?

print(c[:])     # Get all elements
print(c[0])     # The first element
print(c[-1])    # The last element
print(c[:3])    # The first 3 elements
print(c[-3:])   # The last 3 elemnets
print(c[2:4])   # 2:4 selects elements of indexes 2 and 3


I am working on the file "Deep-Learning-Book-Review/Linear_Algebra_with_Python_and_NumPy.ipynb".
The first task is:
Operations with Matrices
Matrix transposition is obtained by attribute .T
Can you write Python code for it?

X = ones([2,5])     # Generate 2x5 matrix full of ones
Y = X.T             # Obtain transpose of matrix X

print('Matrix X of size', X.shape, ':\n', X)
print('\nMatrix Y=X.T of size', Y.shape, ':\n', Y)


I am working on the file "Deep-Learning-Book-Review/Linear_Algebra_with_Python_and_NumPy.ipynb".
The first task is:
There are also methods for essential matrix features like Frobenius norm, rank or determinant.
Can you write Python code for it?

print('Matrix A of size', A.shape, ':\n', A)

# Frobenius norm of matrix
print('\nFrobenius norm: ||A|| =', linalg.norm(A))

# Rank of matrix
print('rank(A) =', linalg.matrix_rank(A))

# Determinant of matrix
print('det(A) =',  linalg.det(A))


I am working on the file "Deep-Learning-Book-Review/Linear_Algebra_with_Python_and_NumPy.ipynb".
The first task is:
Transposition of vectors is not so intuitive as in Matlab, especially if a vector is defined as 1D array and you cannot distinguish between row and column vector. However, using the keyword newaxis it's possible to shape the vector into 2D array (as matrix of size $1 \times n$ or $n \times 1$), where transposition makes sense and can be obtained by attribute .T.
Can you write Python code for it?

x = a[:,newaxis]  # Make column vector from vector a (defined as array)
print(x)
print(x.shape)    # Now size of column vector is 3x1
print(x.T)        # Make row vector by transpostion of column vector


I am working on the file "Deep-Learning-Book-Review/Linear_Algebra_with_Python_and_NumPy.ipynb".
The first task is:
Vectors and Matrices
Using NumPy we can define vectors and matrices with both real or complex elements. Although, in contrast to Matlab, where matrix is the default type, in Python we need to define vectors and matrices as array or matrix type from NumPy package.
<img  src="http://www.math.cornell.edu/~mec/Winter2009/RalucaRemus/Lecture1/Images/matrix.gif"/>
Can you write Python code for it?

a = array([10,20,30])   # Define a vector of size 3 using type 'array'
print(a)
print(a.shape)          # Size/shape of vector

b = matrix('10 20 30')  # Define a vector of size 3 using type 'matrix'
print(b)
print(b.shape)          # Size/shape of vector

c = linspace(10,20,6)   # Define vector as 6 values evenly spaced from 10 to 20
print(c)


I am working on the file "Deep-MNIST-for-Experts.ipynb".
The first task is:
Build a Multilayer Convolutional Network

Getting 92% accuracy on MNIST is bad. It's almost embarrassingly bad. In this section, we'll fix that, jumping from a very simple model to something moderately sophisticated: a small convolutional neural network. This will get us to around 99.2% accuracy -- not state of the art, but respectable.
Weight Initialization
To create this model, we're going to need to create a lot of weights and biases. One should generally initialize weights with a small amount of noise for symmetry breaking, and to prevent 0 gradients. Since we're using ReLU neurons, it is also good practice to initialize them with a slightly positive initial bias to avoid "dead neurons". Instead of doing this repeatedly while we build the model, let's create two handy functions to do it for us.
Can you write Python code for it?

def weight_variable(shape):
    initial = tf.truncated_normal(shape, stddev=0.1)
    return tf.Variable(initial)

def bias_variable(shape):
    initial = tf.constant(0.1, shape=shape)
    return tf.Variable(initial)


I am working on the file "Deep-MNIST-for-Experts.ipynb".
The first task is:
Dropout
To reduce overfitting, we will apply dropout before the readout layer. We create a placeholder for the probability that a neuron's output is kept during dropout. This allows us to turn dropout on during training, and turn it off during testing. TensorFlow's tf.nn.dropout op automatically handles scaling neuron outputs in addition to masking them, so dropout just works without any additional scaling.1
Can you write Python code for it?

keep_prob = tf.placeholder(tf.float32)
h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)


I am working on the file "Deep-MNIST-for-Experts.ipynb".
The first task is:
Here x and y_ aren't specific values. Rather, they are each a placeholder -- a value that we'll input when we ask TensorFlow to run a computation.
The input images x will consist of a 2d tensor of floating point numbers. Here we assign it a shape of [None, 784], where 784 is the dimensionality of a single flattened 28 by 28 pixel MNIST image, and None indicates that the first dimension, corresponding to the batch size, can be of any size. The target output classes y_ will also consist of a 2d tensor, where each row is a one-hot 10-dimensional vector indicating which digit class (zero through nine) the corresponding MNIST image belongs to.
The shape argument to placeholder is optional, but it allows TensorFlow to automatically catch bugs stemming from inconsistent tensor shapes.
Variables
We now define the weights W and biases b for our model. We could imagine treating these like additional inputs, but TensorFlow has an even better way to handle them: Variable. A Variable is a value that lives in TensorFlow's computation graph. It can be used and even modified by the computation. In machine learning applications, one generally has the model parameters be Variables.
Can you write Python code for it?

W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))


I am working on the file "Deep-MNIST-for-Experts.ipynb".
The first task is:
Note that tf.nn.softmax_cross_entropy_with_logits internally applies the softmax on the model's unnormalized model prediction and sums across all classes, and tf.reduce_mean takes the average over these sums.
Train the Model

Now that we have defined our model and training loss function, it is straightforward to train using TensorFlow. Because TensorFlow knows the entire computation graph, it can use automatic differentiation to find the gradients of the loss with respect to each of the variables. TensorFlow has a variety of built-in optimization algorithms. For this example, we will use steepest gradient descent, with a step length of 0.5, to descend the cross entropy.
Can you write Python code for it?


I am working on the file "Deep-MNIST-for-Experts.ipynb".
The first task is:
Predicted Class and Loss Function
We can now implement our regression model. It only takes one line! We multiply the vectorized input images x by the weight matrix W, add the bias b.
Can you write Python code for it?


I am working on the file "Deep-MNIST-for-Experts.ipynb".
The first task is:
Readout Layer
Finally, we add a layer, just like for the one layer softmax regression above.
Can you write Python code for it?

W_fc2 = weight_variable([1024, 10])
b_fc2 = bias_variable([10])

y_conv = tf.matmul(h_fc1_drop, W_fc2) + b_fc2


I am working on the file "Deep-MNIST-for-Experts.ipynb".
The first task is:
To apply the layer, we first reshape x to a 4d tensor, with the second and third dimensions corresponding to image width and height, and the final dimension corresponding to the number of color channels.
Can you write Python code for it?


I am working on the file "Deep-MNIST-for-Experts.ipynb".
The first task is:
We can specify a loss function just as easily. Loss indicates how bad the model's prediction was on a single example; we try to minimize that while training across all the examples. Here, our loss function is the cross-entropy between the target and the softmax activation function applied to the model's prediction. As in the beginners tutorial, we use the stable formulation:
Can you write Python code for it?

cross_entropy = tf.reduce_mean(
    tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y))


I am working on the file "Deep-MNIST-for-Experts.ipynb".
The first task is:
We load 100 training examples in each training iteration. We then run the train_step operation, using feed_dict to replace the placeholder tensors x and y_ with the training examples. Note that you can replace any tensor in your computation graph using feed_dict -- it's not restricted to just placeholders.
Evaluate the Model
How well did our model do?
First we'll figure out where we predicted the correct label. tf.argmax is an extremely useful function which gives you the index of the highest entry in a tensor along some axis. For example, tf.argmax(y,1) is the label our model thinks is most likely for each input, while tf.argmax(y_,1) is the true label. We can use tf.equal to check if our prediction matches the truth.
Can you write Python code for it?

correct_prediction = tf.equal(tf.argmax(y,1), tf.arg_max(y_,1))


I am working on the file "Deep-MNIST-for-Experts.ipynb".
The first task is:
We then convolve x_image with the weight tensor, add the bias, apply the ReLU function, and finally max pool. The max_pool_2x2 method will reduce the image size to 14x14.
Can you write Python code for it?

h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
h_pool1 = max_pool_2x2(h_conv1)


I am working on the file "Deep-MNIST-for-Experts.ipynb".
The first task is:
What TensorFlow actually did in that single line was to add new operations to the computation graph. These operations included ones to compute gradients, compute parameter update steps, and apply update steps to the parameters.
The returned operation train_step, when run, will apply the gradient descent updates to the parameters. Training the model can therefore be accomplished by repeatedly running train_step.
Can you write Python code for it?

for _ in range(1000):
    batch = mnist.train.next_batch(100)
    train_step.run(feed_dict={x: batch[0], y_: batch[1]})


I am working on the file "DeepLearning/3-ConvolutionalNeuralNetwork/2-DeepCNN_CaseStudy/KerasTutorial/Keras+-+Tutorial+-+Happy+House+v1.ipynb".
The first task is:
5 - Other useful functions in Keras (Optional)
Two other basic features of Keras that you'll find useful are:
- model.summary(): prints the details of your layers in a table with the sizes of its inputs/outputs
- plot_model(): plots your graph in a nice layout. You can even save it as ".png" using SVG() if you'd like to share it on social media ;). It is saved in "File" then "Open..." in the upper bar of the notebook.
Run the following code.
Can you write Python code for it?

happyModel.summary()

plot_model(happyModel, to_file='HappyModel.png')
SVG(model_to_dot(happyModel).create(prog='dot', format='svg'))


I am working on the file "DeepLearning/3-ConvolutionalNeuralNetwork/2-DeepCNN_CaseStudy/KerasTutorial/Keras+-+Tutorial+-+Happy+House+v1.ipynb".
The first task is:
Details of the "Happy" dataset:
- Images are of shape (64,64,3)
- Training: 600 pictures
- Test: 150 pictures
It is now time to solve the "Happy" Challenge.
2 - Building a model in Keras
Keras is very good for rapid prototyping. In just a short time you will be able to build a model that achieves outstanding results.
Here is an example of a model in Keras:
```python
def model(input_shape):
    # Define the input placeholder as a tensor with shape input_shape. Think of this as your input image!
    X_input = Input(input_shape)
# Zero-Padding: pads the border of X_input with zeroes
X = ZeroPadding2D((3, 3))(X_input)

# CONV -> BN -> RELU Block applied to X
X = Conv2D(32, (7, 7), strides = (1, 1), name = 'conv0')(X)
X = BatchNormalization(axis = 3, name = 'bn0')(X)
X = Activation('relu')(X)

# MAXPOOL
X = MaxPooling2D((2, 2), name='max_pool')(X)

# FLATTEN X (means convert it to a vector) + FULLYCONNECTED
X = Flatten()(X)
X = Dense(1, activation='sigmoid', name='fc')(X)

# Create model. This creates your Keras model instance, you'll use this instance to train/test the model.
model = Model(inputs = X_input, outputs = X, name='HappyModel')

return model

```
Note that Keras uses a different convention with variable names than we've previously used with numpy and TensorFlow. In particular, rather than creating and assigning a new variable on each step of forward propagation such as X, Z1, A1, Z2, A2, etc. for the computations for the different layers, in Keras code each line above just reassigns X to a new value using X = .... In other words, during each step of forward propagation, we are just writing the latest value in the commputation into the same variable X. The only exception was X_input, which we kept separate and did not overwrite, since we needed it at the end to create the Keras model instance (model = Model(inputs = X_input, ...) above).
Exercise: Implement a HappyModel(). This assignment is more open-ended than most. We suggest that you start by implementing a model using the architecture we...
Can you write Python code for it?

# GRADED FUNCTION: HappyModel

def HappyModel(input_shape):
    """
    Implementation of the HappyModel.

    Arguments:
    input_shape -- shape of the images of the dataset

    Returns:
    model -- a Model() instance in Keras
    """

    ### START CODE HERE ###
    # Feel free to use the suggested outline in the text above to get started, and run through the whole
    # exercise (including the later portions of this notebook) once. The come back also try out other
    # network architectures as well.

    ### END CODE HERE ###

    return model


I am working on the file "DeepLearning/3-ConvolutionalNeuralNetwork/2-DeepCNN_CaseStudy/KerasTutorial/Keras+-+Tutorial+-+Happy+House+v1.ipynb".
The first task is:
Exercise: Implement step 2, i.e. compile the model to configure the learning process. Choose the 3 arguments of compile() wisely. Hint: the Happy Challenge is a binary classification problem.
Can you write Python code for it?

### START CODE HERE ### (1 line)
None
### END CODE HERE ###


I am working on the file "DeepLearning/3-ConvolutionalNeuralNetwork/2-DeepCNN_CaseStudy/KerasTutorial/Keras+-+Tutorial+-+Happy+House+v1.ipynb".
The first task is:
If your happyModel() function worked, you should have observed much better than random-guessing (50%) accuracy on the train and test sets. To pass this assignment, you have to get at least 75% accuracy.
To give you a point of comparison, our model gets around 95% test accuracy in 40 epochs (and 99% train accuracy) with a mini batch size of 16 and "adam" optimizer. But our model gets decent accuracy after just 2-5 epochs, so if you're comparing different models you can also train a variety of models on just a few epochs and see how they compare.
If you have not yet achieved 75% accuracy, here're some things you can play around with to try to achieve it:

Try using blocks of CONV->BATCHNORM->RELU such as:
python
X = Conv2D(32, (3, 3), strides = (1, 1), name = 'conv0')(X)
X = BatchNormalization(axis = 3, name = 'bn0')(X)
X = Activation('relu')(X)
until your height and width dimensions are quite low and your number of channels quite large (≈32 for example). You are encoding useful information in a volume with a lot of channels. You can then flatten the volume and use a fully-connected layer.
You can use MAXPOOL after such blocks. It will help you lower the dimension in height and width.
Change your optimizer. We find Adam works well.
If the model is struggling to run and you get memory issues, lower your batch_size (12 is usually a good compromise)
Run on more epochs, until you see the train accuracy plateauing.

Even if you have achieved 75% accuracy, please feel free to keep playing with your model to try to get even better results.
Note: If you perform hyperparameter tuning on your model, the test set actually becomes a dev set, and your model might end up overfitting to the test (dev) set. But just for the purpose of this assignment, we won't worry about that here.
3 - Conclusion
Congratulations, you have solved the Happy House challenge!
Now, you just need to link this model to the front-door camera of your house. We unfortunately won't go into the details of how to do that here.
<font...
Can you write Python code for it?

### START CODE HERE ###
img_path = 'images/my_image.jpg'
### END CODE HERE ###
img = image.load_img(img_path, target_size=(64, 64))
imshow(img)

x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

print(happyModel.predict(x))


I am working on the file "DeepLearning/3-ConvolutionalNeuralNetwork/2-DeepCNN_CaseStudy/KerasTutorial/Keras+-+Tutorial+-+Happy+House+v1.ipynb".
The first task is:
Note that if you run fit() again, the model will continue to train with the parameters it has already learnt instead of reinitializing them.
Exercise: Implement step 4, i.e. test/evaluate the model.
Can you write Python code for it?

### START CODE HERE ### (1 line)
preds = None
### END CODE HERE ###
print()
print ("Loss = " + str(preds[0]))
print ("Test Accuracy = " + str(preds[1]))


I am working on the file "DeepLearning/3-ConvolutionalNeuralNetwork/2-DeepCNN_CaseStudy/KerasTutorial/Keras+-+Tutorial+-+Happy+House+v1.ipynb".
The first task is:
Note: As you can see, we've imported a lot of functions from Keras. You can use them easily just by calling them directly in the notebook. Ex: X = Input(...) or X = ZeroPadding2D(...).
1 - The Happy House
For your next vacation, you decided to spend a week with five of your friends from school. It is a very convenient house with many things to do nearby. But the most important benefit is that everybody has commited to be happy when they are in the house. So anyone wanting to enter the house must prove their current state of happiness.

   Figure 1   : the Happy House

As a deep learning expert, to make sure the "Happy" rule is strictly applied, you are going to build an algorithm which that uses pictures from the front door camera to check if the person is happy or not. The door should open only if the person is happy.
You have gathered pictures of your friends and yourself, taken by the front-door camera. The dataset is labbeled.

Run the following code to normalize the dataset and learn about its shapes.
Can you write Python code for it?

X_train_orig, Y_train_orig, X_test_orig, Y_test_orig, classes = load_dataset()

# Normalize image vectors
X_train = X_train_orig/255.
X_test = X_test_orig/255.

# Reshape
Y_train = Y_train_orig.T
Y_test = Y_test_orig.T

print ("number of training examples = " + str(X_train.shape[0]))
print ("number of test examples = " + str(X_test.shape[0]))
print ("X_train shape: " + str(X_train.shape))
print ("Y_train shape: " + str(Y_train.shape))
print ("X_test shape: " + str(X_test.shape))
print ("Y_test shape: " + str(Y_test.shape))


I am working on the file "DeepLearning/3-ConvolutionalNeuralNetwork/2-DeepCNN_CaseStudy/KerasTutorial/Keras+-+Tutorial+-+Happy+House+v1.ipynb".
The first task is:
You have now built a function to describe your model. To train and test this model, there are four steps in Keras:
1. Create the model by calling the function above
2. Compile the model by calling model.compile(optimizer = "...", loss = "...", metrics = ["accuracy"])
3. Train the model on train data by calling model.fit(x = ..., y = ..., epochs = ..., batch_size = ...)
4. Test the model on test data by calling model.evaluate(x = ..., y = ...)
If you want to know more about model.compile(), model.fit(), model.evaluate() and their arguments, refer to the official Keras documentation.
Exercise: Implement step 1, i.e. create the model.
Can you write Python code for it?

### START CODE HERE ### (1 line)
happyModel = None
### END CODE HERE ###


I am working on the file "DeepLearning/dog_app.ipynb".
The first task is:
(IMPLEMENTATION) Assess the Human Face Detector
Question 1: Use the code cell below to test the performance of the face_detector function.
- What percentage of the first 100 images in human_files have a detected human face?
- What percentage of the first 100 images in dog_files have a detected human face?
Ideally, we would like 100% of human images with a detected face and 0% of dog images with a detected face.  You will see that our algorithm falls short of this goal, but still gives acceptable performance.  We extract the file paths for the first 100 images from each of the datasets and store them in the numpy arrays human_files_short and dog_files_short.
Answer:
* Of the first 100 images in 'human_files', the face detector detects a human face in each of the 100 files, hence the accuracy is 1.0 and 100% of the images have a human face detected in them.
* This detector has been trained on human faces and sometimes misclassifies dog faces as human faces. It does so for 11 of the first 100 images in 'dog_files'. 11% of the dog images has a human face detected in them.
Can you write Python code for it?

human_files_short = human_files[:100]
dog_files_short = train_files[:100]
# Do NOT modify the code above this line.

## TODO: Test the performance of the face_detector algorithm
## on the images in human_files_short and dog_files_short.

humans = 0
dogs = 0

for img in human_files_short:
    if face_detector(img) :
        humans = humans + 1

for img in dog_files_short:
    if face_detector(img) :
        dogs = dogs + 1

accuracy_on_humans = humans / 100.0
accuracy_on_dogs = dogs / 100.0

print ("Accuracy on humans is " + str(accuracy_on_humans))
print ("Accuracy on dogs is " + str(accuracy_on_dogs))


I am working on the file "DeepLearning/dog_app.ipynb".
The first task is:
(IMPLEMENTATION) Predict Dog Breed with the Model
Write a function that takes an image path as input and returns the dog breed (Affenpinscher, Afghan_hound, etc) that is predicted by your model.
Similar to the analogous function in Step 5, your function should have three steps:
1. Extract the bottleneck features corresponding to the chosen CNN model.
2. Supply the bottleneck features as input to the model to return the predicted vector.  Note that the argmax of this prediction vector gives the index of the predicted dog breed.
3. Use the dog_names array defined in Step 0 of this notebook to return the corresponding breed.
The functions to extract the bottleneck features can be found in extract_bottleneck_features.py, and they have been imported in an earlier code cell.  To obtain the bottleneck features corresponding to your chosen CNN architecture, you need to use the function
extract_{network}

where {network}, in the above filename, should be one of VGG19, Resnet50, InceptionV3, or Xception.
Can you write Python code for it?

### TODO: Write a function that takes a path to an image as input
### and returns the dog breed that is predicted by the model.
def Xception_predict_breed(img_path):
    # extract bottleneck features
    bottleneck_feature = extract_Xception(path_to_tensor(img_path))
    # obtain predicted vector
    predicted_vector = Xception_model.predict(bottleneck_feature)
    # return dog breed that is predicted by the model
    return (dog_names[np.argmax(predicted_vector)], predicted_vector[0][np.argmax(predicted_vector)])

import glob
from PIL import Image
from io import BytesIO
from IPython.display import HTML

def get_thumbnail(path):
    i = Image.open(path).convert('RGB')
    i.thumbnail((150, 150), Image.LANCZOS)
    return i

def image_base64(im):
    if isinstance(im, str):
        im = get_thumbnail(im)
    with BytesIO() as buffer:
        im.save(buffer, 'jpeg')
        return base64.b64encode(buffer.getvalue()).decode()

def image_formatter(im):
    return f''

def file_formatter(file_name):
    return f''

predictions_and_images = dict()
for filename in glob.iglob('dogs/*'):
    label, score = Xception_predict_breed(filename)
    print("Predicting that image in  {0}, depicts a {1}, with a score {2}".format(filename, label, score))
    predictions_and_images[filename] = label

import pandas as pd
import base64
from IPython.display import HTML
predictions_and_images
predictions_df = pd.DataFrame.from_dict(predictions_and_images, orient='index')
predictions_df.reset_index(inplace=True)
predictions_df.columns = ['file', 'label']
predictions_df['image'] = predictions_df.file.map(lambda f: get_thumbnail(f))
predictions_df
HTML(predictions_df[['label', 'file']].to_html(formatters={'file': file_formatter}, escape=False))


I am working on the file "DeepLearning/dog_app.ipynb".
The first task is:
(IMPLEMENTATION) Test the Model
Try out your model on the test dataset of dog images. Ensure that your test accuracy is greater than 60%.
Can you write Python code for it?

### TODO: Calculate classification accuracy on the test dataset.
# get index of predicted dog breed for each image in test set
Xception_predictions = [np.argmax(Xception_model.predict(np.expand_dims(feature, axis=0))) for feature in test_Xception]

# report test accuracy
test_accuracy = 100*np.sum(np.array(Xception_predictions)==np.argmax(test_targets, axis=1))/len(Xception_predictions)
print('Test accuracy: %.4f%%' % test_accuracy)


I am working on the file "DeepLearning/dog_app.ipynb".
The first task is:
(IMPLEMENTATION) Train the Model
Train your model in the code cell below.  Use model checkpointing to save the model that attains the best validation loss.
You are welcome to augment the training data, but this is not a requirement.
Can you write Python code for it?

from keras.callbacks import ModelCheckpoint

### TODO: specify the number of epochs that you would like to use to train the model.

epochs = 20

### Do NOT modify the code below this line.

checkpointer = ModelCheckpoint(filepath='saved_models/weights.best.from_scratch.hdf5',
                               verbose=1, save_best_only=True)

model.fit(train_tensors, train_targets,
          validation_data=(valid_tensors, valid_targets),
          epochs=epochs, batch_size=20, callbacks=[checkpointer], verbose=1)


from keras.callbacks import ModelCheckpoint
### TODO: Train the model.
checkpointer = ModelCheckpoint(filepath='saved_models/weights.best.Xception.hdf5',
                               verbose=1, save_best_only=True)

history = Xception_model.fit(train_Xception, train_targets,
          validation_data=(valid_Xception, valid_targets),
          epochs=50, batch_size=100, callbacks=[checkpointer], verbose=1)


I am working on the file "DeepLearning/dog_app.ipynb".
The first task is:

Step 2: Detect Dogs
In this section, we use a pre-trained ResNet-50 model to detect dogs in images.  Our first line of code downloads the ResNet-50 model, along with weights that have been trained on ImageNet, a very large, very popular dataset used for image classification and other vision tasks.  ImageNet contains over 10 million URLs, each linking to an image containing an object from one of 1000 categories.  Given an image, this pre-trained ResNet-50 model returns a prediction (derived from the available categories in ImageNet) for the object that is contained in the image.
Can you write Python code for it?

from keras.applications.resnet50 import ResNet50

# define ResNet50 model
ResNet50_model = ResNet50(weights='imagenet')


I am working on the file "DeepLearning/dog_app.ipynb".
The first task is:

Step 3: Create a CNN to Classify Dog Breeds (from Scratch)
Now that we have functions for detecting humans and dogs in images, we need a way to predict breed from images.  In this step, you will create a CNN that classifies dog breeds.  You must create your CNN from scratch (so, you can't use transfer learning yet!), and you must attain a test accuracy of at least 1%.  In Step 5 of this notebook, you will have the opportunity to use transfer learning to create a CNN that attains greatly improved accuracy.
Be careful with adding too many trainable layers!  More parameters means longer training, which means you are more likely to need a GPU to accelerate the training process.  Thankfully, Keras provides a handy estimate of the time that each epoch is likely to take; you can extrapolate this estimate to figure out how long it will take for your algorithm to train.
We mention that the task of assigning breed to dogs from images is considered exceptionally challenging.  To see why, consider that even a human would have great difficulty in distinguishing between a Brittany and a Welsh Springer Spaniel.
Brittany | Welsh Springer Spaniel
- | -
 |
It is not difficult to find other dog breed pairs with minimal inter-class variation (for instance, Curly-Coated Retrievers and American Water Spaniels).
Curly-Coated Retriever | American Water Spaniel
- | -
 |
Likewise, recall that labradors come in yellow, chocolate, and black.  Your vision-based algorithm will have to conquer this high intra-class variation to determine how to classify all of these different shades as the same breed.
Yellow Labrador | Chocolate Labrador | Black Labrador
- | -
 | <img src="images/Labrador_retriever_06455.jpg"...
Can you write Python code for it?

from PIL import ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True

# pre-process the data for Keras
train_tensors = paths_to_tensor(train_files).astype('float32')/255
valid_tensors = paths_to_tensor(valid_files).astype('float32')/255
test_tensors = paths_to_tensor(test_files).astype('float32')/255


I am working on the file "DeepLearning/dog_app.ipynb".
The first task is:

Step 4: Use a CNN to Classify Dog Breeds
To reduce training time without sacrificing accuracy, we show you how to train a CNN using transfer learning.  In the following step, you will get a chance to use transfer learning to train your own CNN.
Obtain Bottleneck Features
Can you write Python code for it?

bottleneck_features = np.load('/data/bottleneck_features/DogVGG16Data.npz')
train_VGG16 = bottleneck_features['train']
valid_VGG16 = bottleneck_features['valid']
test_VGG16 = bottleneck_features['test']


I am working on the file "DeepLearning/dog_app.ipynb".
The first task is:

Step 6: Write your Algorithm
Write an algorithm that accepts a file path to an image and first determines whether the image contains a human, dog, or neither.  Then,
- if a dog is detected in the image, return the predicted breed.
- if a human is detected in the image, return the resembling dog breed.
- if neither is detected in the image, provide output that indicates an error.
You are welcome to write your own functions for detecting humans and dogs in images, but feel free to use the face_detector and dog_detector functions developed above.  You are required to use your CNN from Step 5 to predict dog breed.
Some sample output for our algorithm is provided below, but feel free to design your own user experience!

(IMPLEMENTATION) Write your Algorithm
Can you write Python code for it?

### TODO: Write your algorithm.
### Feel free to use as many code cells as needed.

def predict_for_image(filename):
    label, score = Xception_predict_breed(filename)
    if (score > 0.95) :
        predictions_and_images[filename] = ('dog', label, score)
    elif face_detector(filename):
        predictions_and_images[filename] = ('human', label, score)
    else:
        predictions_and_images[filename] = ('neither', label, score)
    return predictions_and_images


I am working on the file "DeepLearning/dog_app.ipynb".
The first task is:
Before using any of the face detectors, it is standard procedure to convert the images to grayscale.  The detectMultiScale function executes the classifier stored in face_cascade and takes the grayscale image as a parameter.
In the above code, faces is a numpy array of detected faces, where each row corresponds to a detected face.  Each detected face is a 1D array with four entries that specifies the bounding box of the detected face.  The first two entries in the array (extracted in the above code as x and y) specify the horizontal and vertical positions of the top left corner of the bounding box.  The last two entries in the array (extracted here as w and h) specify the width and height of the box.
Write a Human Face Detector
We can use this procedure to write a function that returns True if a human face is detected in an image and False otherwise.  This function, aptly named face_detector, takes a string-valued file path to an image as input and appears in the code block below.
Can you write Python code for it?

# returns "True" if face is detected in image stored at img_path
def face_detector(img_path):
    img = cv2.imread(img_path)
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    faces = face_cascade.detectMultiScale(gray)
    return len(faces) > 0


I am working on the file "DeepLearning/dog_app.ipynb".
The first task is:
Compile the Model
Can you write Python code for it?


I am working on the file "DeepLearning/dog_app.ipynb".
The first task is:
Import Human Dataset
In the code cell below, we import a dataset of human images, where the file paths are stored in the numpy array human_files.
Can you write Python code for it?

import random
random.seed(8675309)

# load filenames in shuffled human dataset
human_files = np.array(glob("/data/lfw/*/*"))
random.shuffle(human_files)

# print statistics about the dataset
print('There are %d total human images.' % len(human_files))


I am working on the file "DeepLearning/dog_app.ipynb".
The first task is:
Load the Model with the Best Validation Loss
Can you write Python code for it?


I am working on the file "DeepLearning/dog_app.ipynb".
The first task is:
Making Predictions with ResNet-50
Getting the 4D tensor ready for ResNet-50, and for any other pre-trained model in Keras, requires some additional processing.  First, the RGB image is converted to BGR by reordering the channels.  All pre-trained models have the additional normalization step that the mean pixel (expressed in RGB as $[103.939, 116.779, 123.68]$ and calculated from all pixels in all images in ImageNet) must be subtracted from every pixel in each image.  This is implemented in the imported function preprocess_input.  If you're curious, you can check the code for preprocess_input here.
Now that we have a way to format our image for supplying to ResNet-50, we are now ready to use the model to extract the predictions.  This is accomplished with the predict method, which returns an array whose $i$-th entry is the model's predicted probability that the image belongs to the $i$-th ImageNet category.  This is implemented in the ResNet50_predict_labels function below.
By taking the argmax of the predicted probability vector, we obtain an integer corresponding to the model's predicted object class, which we can identify with an object category through the use of this dictionary.
Can you write Python code for it?

from keras.applications.resnet50 import preprocess_input, decode_predictions

def ResNet50_predict_labels(img_path):
    # returns prediction vector for image located at img_path
    img = preprocess_input(path_to_tensor(img_path))
    return np.argmax(ResNet50_model.predict(img))


I am working on the file "DeepLearning/dog_app.ipynb".
The first task is:
Model Architecture
The model uses the the pre-trained VGG-16 model as a fixed feature extractor, where the last convolutional output of VGG-16 is fed as input to our model.  We only add a global average pooling layer and a fully connected layer, where the latter contains one node for each dog category and is equipped with a softmax.
Can you write Python code for it?

VGG16_model = Sequential()
VGG16_model.add(GlobalAveragePooling2D(input_shape=train_VGG16.shape[1:]))
VGG16_model.add(Dense(133, activation='softmax'))

VGG16_model.summary()


I am working on the file "DeepLearning/dog_app.ipynb".
The first task is:
Predict Dog Breed with the Model
Can you write Python code for it?

from extract_bottleneck_features import *

def VGG16_predict_breed(img_path):
    # extract bottleneck features
    bottleneck_feature = extract_VGG16(path_to_tensor(img_path))
    # obtain predicted vector
    predicted_vector = VGG16_model.predict(bottleneck_feature)
    # return dog breed that is predicted by the model
    return dog_names[np.argmax(predicted_vector)]


I am working on the file "DeepLearning/dog_app.ipynb".
The first task is:
Train the Model
Can you write Python code for it?

checkpointer = ModelCheckpoint(filepath='saved_models/weights.best.VGG16.hdf5',
                               verbose=1, save_best_only=True)

VGG16_model.fit(train_VGG16, train_targets,
          validation_data=(valid_VGG16, valid_targets),
          epochs=20, batch_size=20, callbacks=[checkpointer], verbose=1)


I am working on the file "DeepLearning/dog_app.ipynb".
The first task is:
Write a Dog Detector
While looking at the dictionary, you will notice that the categories corresponding to dogs appear in an uninterrupted sequence and correspond to dictionary keys 151-268, inclusive, to include all categories from 'Chihuahua' to 'Mexican hairless'.  Thus, in order to check to see if an image is predicted to contain a dog by the pre-trained ResNet-50 model, we need only check if the ResNet50_predict_labels function above returns a value between 151 and 268 (inclusive).
We use these ideas to complete the dog_detector function below, which returns True if a dog is detected in an image (and False if not).
Can you write Python code for it?

### returns "True" if a dog is detected in the image stored at img_path
def dog_detector(img_path):
    prediction = ResNet50_predict_labels(img_path)
    return ((prediction <= 268) & (prediction >= 151))


I am working on the file "DeepLearningA-Z/02-supervised-deep-learning/01-Artificial-Neural-Networks-ANN/.ipynb_checkpoints/02-Building-an-ANN-checkpoint.ipynb".
The first task is:
Building an ANN
Can you write Python code for it?

# Importing the Keras libraries and packages
import keras
from keras.models import Sequential
from keras.layers import Dense

# Initialising the ANN
classifier = Sequential()

# Adding the input layer and the first hidden layer
classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu', input_dim = 11))

# Adding the second hidden layer
classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu'))

# Adding the output layer
classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))

# Compiling the ANN
classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])

# Fitting the ANN to the Training set
classifier.fit(X_train, y_train, batch_size = 10, epochs = 100)


I am working on the file "DeepLearningA-Z/02-supervised-deep-learning/01-Artificial-Neural-Networks-ANN/.ipynb_checkpoints/02-Building-an-ANN-checkpoint.ipynb".
The first task is:
Evaluating, Improving and Tuning the ANN

Using K-Fold Cross validation with Keras
Can you write Python code for it?

# Evaluating the ANN
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import cross_val_score
from keras.models import Sequential
from keras.layers import Dense
def build_classifier():
    classifier = Sequential()
    classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu', input_dim = 11))
    classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu'))
    classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))
    classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])
    return classifier
classifier = KerasClassifier(build_fn = build_classifier, batch_size = 10, epochs = 100)
accuracies = cross_val_score(estimator = classifier, X = X_train, y = y_train, cv = 10, n_jobs = -1)
mean = accuracies.mean()
variance = accuracies.std()

# Improving the ANN
# Dropout Regularization to reduce overfitting if needed

# Tuning the ANN
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import GridSearchCV
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout

def build_classifier(optimizer):
    classifier = Sequential()
    classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu', input_dim = 11))
    # classifier.add(Dropout(p = 0.1))
    classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu'))
    # classifier.add(Dropout(p = 0.1))
    classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))
    classifier.compile(optimizer = optimizer, loss = 'binary_crossentropy', metrics = ['accuracy'])
    return classifier
classifier = KerasClassifier(build_fn = build_classifier)
parameters = {'batch_size': [25, 32],
              'epochs': [100, 500],
              'optimizer': ['adam', 'rmsprop']}
grid_search = GridSearchCV(estimator = classifier,
                           param_grid = parameters,
      ...