I am working on the file "Neural Networks And Deep Learning/Logistic Regression with a Neural Network mindset.ipynb".
The first task is:
Many software bugs in deep learning come from having matrix/vector dimensions that don't fit. If you can keep your matrix/vector dimensions straight you will go a long way toward eliminating many bugs.
Exercise: Find the values for:
    - m_train (number of training examples)
    - m_test (number of test examples)
    - num_px (= height = width of a training image)
Remember that train_set_x_orig is a numpy-array of shape (m_train, num_px, num_px, 3). For instance, you can access m_train by writing train_set_x_orig.shape[0].
Can you write Python code for it?

### START CODE HERE ### (≈ 3 lines of code)
m_train = train_set_y.shape[1]
m_test = test_set_y.shape[1]
num_px = train_set_x_orig.shape[1]
### END CODE HERE ###

print ("Number of training examples: m_train = " + str(m_train))
print ("Number of testing examples: m_test = " + str(m_test))
print ("Height/Width of each image: num_px = " + str(num_px))
print ("Each image is of size: (" + str(num_px) + ", " + str(num_px) + ", 3)")
print ("train_set_x shape: " + str(train_set_x_orig.shape))
print ("train_set_y shape: " + str(train_set_y.shape))
print ("test_set_x shape: " + str(test_set_x_orig.shape))
print ("test_set_y shape: " + str(test_set_y.shape))


I am working on the file "Neural Networks And Deep Learning/Logistic Regression with a Neural Network mindset.ipynb".
The first task is:
Run the following cell to train your model.
Can you write Python code for it?

d = model(train_set_x, train_set_y, test_set_x, test_set_y, num_iterations = 2000, learning_rate = 0.005, print_cost = True)


I am working on the file "Neural Networks And Deep Learning/Logistic Regression with a Neural Network mindset.ipynb".
The first task is:
We added "_orig" at the end of image datasets (train and test) because we are going to preprocess them. After preprocessing, we will end up with train_set_x and test_set_x (the labels train_set_y and test_set_y don't need any preprocessing).
Each line of your train_set_x_orig and test_set_x_orig is an array representing an image. You can visualize an example by running the following code. Feel free also to change the index value and re-run to see other images.
Can you write Python code for it?

# Example of a picture
index = 25
plt.imshow(train_set_x_orig[index])
print ("y = " + str(train_set_y[:,index]) + ", it's a '" + classes[np.squeeze(train_set_y[:,index])].decode("utf-8") +  "' picture.")


I am working on the file "Neural Networks and Deep Learning/Building+your+Deep+Neural+Network+-+Step+by+Step+v3.ipynb".
The first task is:
2 - Outline of the Assignment
To build your neural network, you will be implementing several "helper functions". These helper functions will be used in the next assignment to build a two-layer neural network and an L-layer neural network. Each small helper function you will implement will have detailed instructions that will walk you through the necessary steps. Here is an outline of this assignment, you will:

Initialize the parameters for a two-layer network and for an $L$-layer neural network.
Implement the forward propagation module (shown in purple in the figure below).
Complete the LINEAR part of a layer's forward propagation step (resulting in $Z^{[l]}$).
We give you the ACTIVATION function (relu/sigmoid).
Combine the previous two steps into a new [LINEAR->ACTIVATION] forward function.
Stack the [LINEAR->RELU] forward function L-1 time (for layers 1 through L-1) and add a [LINEAR->SIGMOID] at the end (for the final layer $L$). This gives you a new L_model_forward function.

Compute the loss.
Implement the backward propagation module (denoted in red in the figure below).
Complete the LINEAR part of a layer's backward propagation step.
We give you the gradient of the ACTIVATE function (relu_backward/sigmoid_backward)
Combine the previous two steps into a new [LINEAR->ACTIVATION] backward function.
Stack [LINEAR->RELU] backward L-1 times and add [LINEAR->SIGMOID] backward in a new L_model_backward function

Finally update the parameters.

 Figure 1

Note that for every forward function, there is a corresponding backward function. That is why at every step of your forward module you will be storing some values in a cache. The cached values are useful for computing gradients. In the backpropagation module you will then use the cache to calculate the gradients. This assignment will show you exactly how to carry out each of these steps.
3 - Initialization
You will write two helper functions that...
Can you write Python code for it?

# GRADED FUNCTION: initialize_parameters

def initialize_parameters(n_x, n_h, n_y):
    """
    Argument:
    n_x -- size of the input layer
    n_h -- size of the hidden layer
    n_y -- size of the output layer

    Returns:
    parameters -- python dictionary containing your parameters:
                    W1 -- weight matrix of shape (n_h, n_x)
                    b1 -- bias vector of shape (n_h, 1)
                    W2 -- weight matrix of shape (n_y, n_h)
                    b2 -- bias vector of shape (n_y, 1)
    """

    np.random.seed(1)

    ### START CODE HERE ### (≈ 4 lines of code)
    W1 = np.random.randn(n_h,n_x)*0.01
    b1 = np.zeros((n_h,1))
    W2 = np.random.randn(n_y,n_h)*0.01
    b2 = np.zeros((n_y,1))
    ### END CODE HERE ###

    assert(W1.shape == (n_h, n_x))
    assert(b1.shape == (n_h, 1))
    assert(W2.shape == (n_y, n_h))
    assert(b2.shape == (n_y, 1))

    parameters = {"W1": W1,
                  "b1": b1,
                  "W2": W2,
                  "b2": b2}

    return parameters

parameters = initialize_parameters(2,2,1)
print("W1 = " + str(parameters["W1"]))
print("b1 = " + str(parameters["b1"]))
print("W2 = " + str(parameters["W2"]))
print("b2 = " + str(parameters["b2"]))


I am working on the file "Neural Networks and Deep Learning/Building+your+Deep+Neural+Network+-+Step+by+Step+v3.ipynb".
The first task is:

     **AL**
     [[ 0.17007265  0.2524272 ]]

     **Length of caches list **
     2

Great! Now you have a full forward propagation that takes the input X and outputs a row vector $A^{[L]}$ containing your predictions. It also records all intermediate values in "caches". Using $A^{[L]}$, you can compute the cost of your predictions.
5 - Cost function
Now you will implement forward and backward propagation. You need to compute the cost, because you want to check if your model is actually learning.
Exercise: Compute the cross-entropy cost $J$, using the following formula: $$-\frac{1}{m} \sum\limits_{i = 1}^{m} (y^{(i)}\log\left(a^{[L] (i)}\right) + (1-y^{(i)})\log\left(1- a^{L}\right)) \tag{7}$$
Can you write Python code for it?

# GRADED FUNCTION: compute_cost

def compute_cost(AL, Y):
    """
    Implement the cost function defined by equation (7).

    Arguments:
    AL -- probability vector corresponding to your label predictions, shape (1, number of examples)
    Y -- true "label" vector (for example: containing 0 if non-cat, 1 if cat), shape (1, number of examples)

    Returns:
    cost -- cross-entropy cost
    """

    m = Y.shape[1]

    # Compute loss from aL and y.
    ### START CODE HERE ### (≈ 1 lines of code)
    cost = -np.sum(np.dot(Y,np.log(AL).T)+np.dot((1-Y),np.log(1-AL).T))/m
    ### END CODE HERE ###

    cost = np.squeeze(cost)      # To make sure your cost's shape is what we expect (e.g. this turns [[17]] into 17).
    assert(cost.shape == ())

    return cost

Y, AL = compute_cost_test_case()

print("cost = " + str(compute_cost(AL, Y)))


I am working on the file "Neural Networks and Deep Learning/Building+your+Deep+Neural+Network+-+Step+by+Step+v3.ipynb".
The first task is:
Expected Output

     dW1
            [[ 0.41010002  0.07807203  0.13798444  0.10502167]
 [ 0.          0.          0.          0.        ]
 [ 0.05283652  0.01005865  0.01777766  0.0135308 ]]

     db1
            [[-0.22007063]
 [ 0.        ]
 [-0.02835349]]

   dA1
            [[ 0.          0.52257901]
 [ 0.         -0.3269206 ]
 [ 0.         -0.32070404]
 [ 0.         -0.74079187]]

6.4 - Update Parameters
In this section you will update the parameters of the model, using gradient descent:
$$ W^{[l]} = W^{[l]} - \alpha \text{ } dW^{[l]} \tag{16}$$
$$ b^{[l]} = b^{[l]} - \alpha \text{ } db^{[l]} \tag{17}$$
where $\alpha$ is the learning rate. After computing the updated parameters, store them in the parameters dictionary.
Exercise: Implement update_parameters() to update your parameters using gradient descent.
Instructions:
Update parameters using gradient descent on every $W^{[l]}$ and $b^{[l]}$ for $l = 1, 2, ..., L$.
Can you write Python code for it?

# GRADED FUNCTION: update_parameters

def update_parameters(parameters, grads, learning_rate):
    """
    Update parameters using gradient descent

    Arguments:
    parameters -- python dictionary containing your parameters
    grads -- python dictionary containing your gradients, output of L_model_backward

    Returns:
    parameters -- python dictionary containing your updated parameters
                  parameters["W" + str(l)] = ...
                  parameters["b" + str(l)] = ...
    """

    L = len(parameters) // 2 # number of layers in the neural network

    # Update rule for each parameter. Use a for loop.
    ### START CODE HERE ### (≈ 3 lines of code)
    for l in range(L):
        parameters["W" + str(l+1)] = parameters["W" + str(l+1)] - grads['dW'+str(l+1)]*learning_rate
        parameters["b" + str(l+1)] = parameters["b" + str(l+1)] - grads['db'+str(l+1)]*learning_rate
    ### END CODE HERE ###

    return parameters

parameters, grads = update_parameters_test_case()
parameters = update_parameters(parameters, grads, 0.1)

print ("W1 = "+ str(parameters["W1"]))
print ("b1 = "+ str(parameters["b1"]))
print ("W2 = "+ str(parameters["W2"]))
print ("b2 = "+ str(parameters["b2"]))


I am working on the file "Neural Networks and Deep Learning/Building+your+Deep+Neural+Network+-+Step+by+Step+v3.ipynb".
The first task is:
Expected Output:

    **cost**
     0.41493159961539694

6 - Backward propagation module
Just like with forward propagation, you will implement helper functions for backpropagation. Remember that back propagation is used to calculate the gradient of the loss function with respect to the parameters.
Reminder:

 Figure 3 : Forward and Backward propagation for LINEAR->RELU->LINEAR->SIGMOID
 The purple blocks represent the forward propagation, and the red blocks represent the backward propagation.

Now, similar to forward propagation, you are going to build the backward propagation in three steps:
- LINEAR backward
- LINEAR -> ACTIVATION backward where ACTIVATION computes the derivative of either the ReLU or sigmoid activation
- [LINEAR -> RELU] $\times$ (L-1) -> LINEAR -> SIGMOID backward (whole...
Can you write Python code for it?

# GRADED FUNCTION: linear_backward

def linear_backward(dZ, cache):
    """
    Implement the linear portion of backward propagation for a single layer (layer l)

    Arguments:
    dZ -- Gradient of the cost with respect to the linear output (of current layer l)
    cache -- tuple of values (A_prev, W, b) coming from the forward propagation in the current layer

    Returns:
    dA_prev -- Gradient of the cost with respect to the activation (of the previous layer l-1), same shape as A_prev
    dW -- Gradient of the cost with respect to W (current layer l), same shape as W
    db -- Gradient of the cost with respect to b (current layer l), same shape as b
    """
    A_prev, W, b = cache
    m = A_prev.shape[1]

    ### START CODE HERE ### (≈ 3 lines of code)
    dW = np.dot(dZ,A_prev.T)/m
    db = np.sum(dZ,axis=1,keepdims=True)/m
    dA_prev = np.dot(W.T,dZ)
    ### END CODE HERE ###

    assert (dA_prev.shape == A_prev.shape)
    assert (dW.shape == W.shape)
    assert (db.shape == b.shape)

    return dA_prev, dW, db

# Set up some test inputs
dZ, linear_cache = linear_backward_test_case()

dA_prev, dW, db = linear_backward(dZ, linear_cache)
print ("dA_prev = "+ str(dA_prev))
print ("dW = " + str(dW))
print ("db = " + str(db))


I am working on the file "Neural Networks and Deep Learning/Building+your+Deep+Neural+Network+-+Step+by+Step+v3.ipynb".
The first task is:
Expected Output:

     **dA_prev**
     [[ 0.51822968 -0.19517421]
 [-0.40506361  0.15255393]
 [ 2.37496825 -0.89445391]]

         **dW**
         [[-0.10076895  1.40685096  1.64992505]]

         **db**
         [[ 0.50629448]]

6.2 - Linear-Activation backward
Next, you will create a function that merges the two helper functions: linear_backward and the backward step for the activation linear_activation_backward.
To help you implement linear_activation_backward, we provided two backward functions:
- sigmoid_backward: Implements the backward propagation for SIGMOID unit. You can call it as follows:
python
dZ = sigmoid_backward(dA, activation_cache)

relu_backward: Implements the backward propagation for RELU unit. You can call it as follows:

python
dZ = relu_backward(dA, activation_cache)
If $g(.)$ is the activation function,
sigmoid_backward and relu_backward compute $$dZ^{[l]} = dA^{[l]} * g'(Z^{[l]}) \tag{11}$$.
Exercise: Implement the backpropagation for the LINEAR->ACTIVATION layer.
Can you write Python code for it?

# GRADED FUNCTION: linear_activation_backward

def linear_activation_backward(dA, cache, activation):
    """
    Implement the backward propagation for the LINEAR->ACTIVATION layer.

    Arguments:
    dA -- post-activation gradient for current layer l
    cache -- tuple of values (linear_cache, activation_cache) we store for computing backward propagation efficiently
    activation -- the activation to be used in this layer, stored as a text string: "sigmoid" or "relu"

    Returns:
    dA_prev -- Gradient of the cost with respect to the activation (of the previous layer l-1), same shape as A_prev
    dW -- Gradient of the cost with respect to W (current layer l), same shape as W
    db -- Gradient of the cost with respect to b (current layer l), same shape as b
    """
    linear_cache, activation_cache = cache

    if activation == "relu":
        ### START CODE HERE ### (≈ 2 lines of code)
        dZ = relu_backward(dA, activation_cache)
        dA_prev, dW, db = linear_backward(dZ, linear_cache)
        ### END CODE HERE ###

    elif activation == "sigmoid":
        ### START CODE HERE ### (≈ 2 lines of code)
        dZ = sigmoid_backward(dA, activation_cache)
        dA_prev, dW, db = linear_backward(dZ, linear_cache)
        ### END CODE HERE ###

    return dA_prev, dW, db

AL, linear_activation_cache = linear_activation_backward_test_case()

dA_prev, dW, db = linear_activation_backward(AL, linear_activation_cache, activation = "sigmoid")
print ("sigmoid:")
print ("dA_prev = "+ str(dA_prev))
print ("dW = " + str(dW))
print ("db = " + str(db) + "\n")

dA_prev, dW, db = linear_activation_backward(AL, linear_activation_cache, activation = "relu")
print ("relu:")
print ("dA_prev = "+ str(dA_prev))
print ("dW = " + str(dW))
print ("db = " + str(db))


I am working on the file "Neural Networks and Deep Learning/Building+your+Deep+Neural+Network+-+Step+by+Step+v3.ipynb".
The first task is:
Expected output:

     **With sigmoid: A **
     [[ 0.96890023  0.11013289]]

     **With ReLU: A **
     [[ 3.43896131  0.        ]]

Note: In deep learning, the "[LINEAR->ACTIVATION]" computation is counted as a single layer in the neural network, not two layers.
d) L-Layer Model
For even more convenience when implementing the $L$-layer Neural Net, you will need a function that replicates the previous one (linear_activation_forward with RELU) $L-1$ times, then follows that with one linear_activation_forward with SIGMOID.

 Figure 2 : [LINEAR -> RELU] $\times$ (L-1) -> LINEAR -> SIGMOID model

Exercise: Implement the forward propagation of the above model.
Instruction: In the code below, the variable AL will denote $A^{[L]} = \sigma(Z^{[L]}) = \sigma(W^{[L]} A^{[L-1]} + b^{[L]})$. (This is sometimes also called Yhat, i.e., this is $\hat{Y}$.)
Tips:
- Use the functions you had previously written
- Use a for loop to replicate [LINEAR->RELU] (L-1) times
- Don't forget to keep track of the caches in the "caches" list. To add a new value c to a list, you can use list.append(c).
Can you write Python code for it?

# GRADED FUNCTION: L_model_forward

def L_model_forward(X, parameters):
    """
    Implement forward propagation for the [LINEAR->RELU]*(L-1)->LINEAR->SIGMOID computation

    Arguments:
    X -- data, numpy array of shape (input size, number of examples)
    parameters -- output of initialize_parameters_deep()

    Returns:
    AL -- last post-activation value
    caches -- list of caches containing:
                every cache of linear_relu_forward() (there are L-1 of them, indexed from 0 to L-2)
                the cache of linear_sigmoid_forward() (there is one, indexed L-1)
    """

    caches = []
    A = X
    L = len(parameters) // 2                  # number of layers in the neural network

    # Implement [LINEAR -> RELU]*(L-1). Add "cache" to the "caches" list.
    for l in range(1, L):
        A_prev = A
        ### START CODE HERE ### (≈ 2 lines of code)
        A, cache = linear_activation_forward(A_prev, parameters['W'+str(l)], parameters['b'+str(l)], activation='relu')
        caches.append(cache)
        ### END CODE HERE ###

    # Implement LINEAR -> SIGMOID. Add "cache" to the "caches" list.
    ### START CODE HERE ### (≈ 2 lines of code)
    AL, cache = linear_activation_forward(A, parameters['W'+str(l+1)], parameters['b'+str(l+1)], activation='sigmoid')
    caches.append(cache)
    ### END CODE HERE ###

    assert(AL.shape == (1,X.shape[1]))

    return AL, caches

X, parameters = L_model_forward_test_case()
AL, caches = L_model_forward(X, parameters)
print("AL = " + str(AL))
print("Length of caches list = " + str(len(caches)))


I am working on the file "Neural Networks and Deep Learning/Building+your+Deep+Neural+Network+-+Step+by+Step+v3.ipynb".
The first task is:
Expected output:

     **W1**
     [[ 0.01624345 -0.00611756]
 [-0.00528172 -0.01072969]]

     **b1**
    [[ 0.]
 [ 0.]]

    **W2**
     [[ 0.00865408 -0.02301539]]

     **b2**
     [[ 0.]]

3.2 - L-layer Neural Network
The initialization for a deeper L-layer neural network is more complicated because there are many more weight matrices and bias vectors. When completing the initialize_parameters_deep, you should make sure that your dimensions match between each layer. Recall that $n^{[l]}$ is the number of units in layer $l$. Thus for example if the size of our input $X$ is $(12288, 209)$ (with $m=209$ examples) then:

         **Shape of W**
         **Shape of b**
         **Activation**
         **Shape of Activation**

         **Layer 1**
         $(n^{[1]},12288)$
         $(n^{[1]},1)$
         $Z^{[1]} = W^{[1]}  X + b^{[1]} $

         $(n^{[1]},209)$

         **Layer 2**
         $(n^{[2]}, n^{[1]})$
         $(n^{[2]},1)$
        $Z^{[2]} = W^{[2]} A^{[1]} + b^{[2]}$
         $(n^{[2]}, 209)$

         $\vdots$
         $\vdots$
         $\vdots$
         $\vdots$
         $\vdots$

         **Layer L-1**
         $(n^{[L-1]}, n^{[L-2]})$
         $(n^{[L-1]}, 1)$
        $Z^{[L-1]} =  W^{[L-1]} A^{[L-2]} + b^{[L-1]}$
         $(n^{[L-1]}, 209)$

         **Layer L**
         $(n^{[L]}, n^{[L-1]})$
         $(n^{[L]}, 1)$
         $Z^{[L]} =  W^{[L]} A^{[L-1]} + b^{[L]}$
  ...
Can you write Python code for it?

# GRADED FUNCTION: initialize_parameters_deep

def initialize_parameters_deep(layer_dims):
    """
    Arguments:
    layer_dims -- python array (list) containing the dimensions of each layer in our network

    Returns:
    parameters -- python dictionary containing your parameters "W1", "b1", ..., "WL", "bL":
                    Wl -- weight matrix of shape (layer_dims[l], layer_dims[l-1])
                    bl -- bias vector of shape (layer_dims[l], 1)
    """

    np.random.seed(3)
    parameters = {}
    L = len(layer_dims)            # number of layers in the network

    for l in range(1, L):
        ### START CODE HERE ### (≈ 2 lines of code)
        parameters['W' + str(l)] = np.random.randn(layer_dims[l],layer_dims[l-1])*0.01
        parameters['b' + str(l)] = np.zeros((layer_dims[l],1))
        ### END CODE HERE ###

        assert(parameters['W' + str(l)].shape == (layer_dims[l], layer_dims[l-1]))
        assert(parameters['b' + str(l)].shape == (layer_dims[l], 1))

    return parameters

parameters = initialize_parameters_deep([5,4,3])
print("W1 = " + str(parameters["W1"]))
print("b1 = " + str(parameters["b1"]))
print("W2 = " + str(parameters["W2"]))
print("b2 = " + str(parameters["b2"]))


I am working on the file "Neural Networks and Deep Learning/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:
$12,288$ equals $64 \times 64 \times 3$ which is the size of one reshaped image vector.
3 - Architecture of your model
Now that you are familiar with the dataset, it is time to build a deep neural network to distinguish cat images from non-cat images.
You will build two different models:
- A 2-layer neural network
- An L-layer deep neural network
You will then compare the performance of these models, and also try out different values for $L$.
Let's look at the two architectures.
3.1 - 2-layer neural network

 Figure 2: 2-layer neural network.
 The model can be summarized as: INPUT -> LINEAR -> RELU -> LINEAR -> SIGMOID -> OUTPUT.

Detailed Architecture of figure 2:
- The input is a (64,64,3) image which is flattened to a vector of size $(12288,1)$.
- The corresponding vector: $[x_0,x_1,...,x_{12287}]^T$ is then multiplied by the weight matrix $W^{[1]}$ of size $(n^{[1]}, 12288)$.
- You then add a bias term and take its relu to get the following vector: $[a_0^{[1]}, a_1^{[1]},..., a_{n^{[1]}-1}^{[1]}]^T$.
- You then repeat the same process.
- You multiply the resulting vector by $W^{[2]}$ and add your intercept (bias).
- Finally, you take the sigmoid of the result. If it is greater than 0.5, you classify it to be a cat.
3.2 - L-layer deep neural network
It is hard to represent an L-layer deep neural network with the above representation. However, here is a simplified network representation:

 Figure 3: L-layer neural network.
 The model can be summarized as: [LINEAR -> RELU] $\times$ (L-1) -> LINEAR -> SIGMOID

Detailed Architecture of figure 3:
- The input is a (64,64,3) image which is flattened to a vector of size (12288,1).
- The corresponding vector: $[x_0,x_1,...,x_{12287}]^T$ is then multiplied by the weight matrix $W^{[1]}$ and then you add the intercept...
Can you write Python code for it?

### CONSTANTS DEFINING THE MODEL ####
n_x = 12288     # num_px * num_px * 3
n_h = 7
n_y = 1
layers_dims = (n_x, n_h, n_y)

# GRADED FUNCTION: two_layer_model

def two_layer_model(X, Y, layers_dims, learning_rate = 0.0075, num_iterations = 3000, print_cost=False):
    """
    Implements a two-layer neural network: LINEAR->RELU->LINEAR->SIGMOID.

    Arguments:
    X -- input data, of shape (n_x, number of examples)
    Y -- true "label" vector (containing 0 if cat, 1 if non-cat), of shape (1, number of examples)
    layers_dims -- dimensions of the layers (n_x, n_h, n_y)
    num_iterations -- number of iterations of the optimization loop
    learning_rate -- learning rate of the gradient descent update rule
    print_cost -- If set to True, this will print the cost every 100 iterations

    Returns:
    parameters -- a dictionary containing W1, W2, b1, and b2
    """

    np.random.seed(1)
    grads = {}
    costs = []                              # to keep track of the cost
    m = X.shape[1]                           # number of examples
    (n_x, n_h, n_y) = layers_dims

    # Initialize parameters dictionary, by calling one of the functions you'd previously implemented
    ### START CODE HERE ### (≈ 1 line of code)
    parameters = initialize_parameters(n_x, n_h, n_y)
    ### END CODE HERE ###

    # Get W1, b1, W2 and b2 from the dictionary parameters.
    W1 = parameters["W1"]
    b1 = parameters["b1"]
    W2 = parameters["W2"]
    b2 = parameters["b2"]

    # Loop (gradient descent)

    for i in range(0, num_iterations):

        # Forward propagation: LINEAR -> RELU -> LINEAR -> SIGMOID. Inputs: "X, W1, b1". Output: "A1, cache1, A2, cache2".
        ### START CODE HERE ### (≈ 2 lines of code)
        A1, cache1 = linear_activation_forward(X, W1, b1, activation='relu')
        A2, cache2 = linear_activation_forward(A1, W2, b2, activation='sigmoid')
        ### END CODE HERE ###

        # Compute cost
        ### START CODE HERE ### (≈ 1 line of code)
        cost =...


I am working on the file "Neural Networks and Deep Learning/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:

    **Train Accuracy**

    0.985645933014

Can you write Python code for it?

pred_test = predict(test_x, test_y, parameters)


I am working on the file "Neural Networks and Deep Learning/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:
As usual, you reshape and standardize the images before feeding them to the network. The code is given in the cell below.

 Figure 1: Image to vector conversion.

Can you write Python code for it?

# Reshape the training and test examples
train_x_flatten = train_x_orig.reshape(train_x_orig.shape[0], -1).T   # The "-1" makes reshape flatten the remaining dimensions
test_x_flatten = test_x_orig.reshape(test_x_orig.shape[0], -1).T

# Standardize data to have feature values between 0 and 1.
train_x = train_x_flatten/255.
test_x = test_x_flatten/255.

print ("train_x's shape: " + str(train_x.shape))
print ("test_x's shape: " + str(test_x.shape))


I am working on the file "Neural Networks and Deep Learning/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:
Expected Output:

         **Accuracy**
         1.0

Can you write Python code for it?

predictions_test = predict(test_x, test_y, parameters)


I am working on the file "Neural Networks and Deep Learning/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:
Expected Output:

         **Cost after iteration 0**
         0.6930497356599888

         **Cost after iteration 100**
         0.6464320953428849

         **...**
         ...

         **Cost after iteration 2400**
         0.048554785628770206

Good thing you built a vectorized implementation! Otherwise it might have taken 10 times longer to train this.
Now, you can use the trained parameters to classify images from the dataset. To see your predictions on the training and test sets, run the cell below.
Can you write Python code for it?

predictions_train = predict(train_x, train_y, parameters)


I am working on the file "Neural Networks and Deep Learning/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:
Expected Output:

         **Cost after iteration 0**
         0.771749

         **Cost after iteration 100**
         0.672053

         **...**
         ...

         **Cost after iteration 2400**
         0.092878

Can you write Python code for it?

pred_train = predict(train_x, train_y, parameters)


I am working on the file "Neural Networks and Deep Learning/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:
Run the cell below to train your parameters. See if your model runs. The cost should be decreasing. It may take up to 5 minutes to run 2500 iterations. Check if the "Cost after iteration 0" matches the expected output below, if not click on the square (⬛) on the upper bar of the notebook to stop the cell and try to find your error.
Can you write Python code for it?

parameters = two_layer_model(train_x, train_y, layers_dims = (n_x, n_h, n_y), num_iterations = 2500, print_cost=True)


I am working on the file "Neural Networks and Deep Learning/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:
The following code will show you an image in the dataset. Feel free to change the index and re-run the cell multiple times to see other images.
Can you write Python code for it?

# Example of a picture
index = 1
plt.imshow(train_x_orig[index])
print ("y = " + str(train_y[0,index]) + ". It's a " + classes[train_y[0,index]].decode("utf-8") +  " picture.")

# Explore your dataset
m_train = train_x_orig.shape[0]
num_px = train_x_orig.shape[1]
m_test = test_x_orig.shape[0]

print ("Number of training examples: " + str(m_train))
print ("Number of testing examples: " + str(m_test))
print ("Each image is of size: (" + str(num_px) + ", " + str(num_px) + ", 3)")
print ("train_x_orig shape: " + str(train_x_orig.shape))
print ("train_y shape: " + str(train_y.shape))
print ("test_x_orig shape: " + str(test_x_orig.shape))
print ("test_y shape: " + str(test_y.shape))


I am working on the file "Neural Networks and Deep Learning/Deep+Neural+Network+-+Application+v3.ipynb".
The first task is:
You will now train the model as a 5-layer neural network.
Run the cell below to train your model. The cost should decrease on every iteration. It may take up to 5 minutes to run 2500 iterations. Check if the "Cost after iteration 0" matches the expected output below, if not click on the square (⬛) on the upper bar of the notebook to stop the cell and try to find your error.
Can you write Python code for it?

parameters = L_layer_model(train_x, train_y, layers_dims, num_iterations = 2500, print_cost = True)


I am working on the file "Neural Networks and Deep Learning/Logistic_Regression_with_a_Neural_Network_mindset_v3.ipynb".
The first task is:
Expected Output:

          ** dw **
         [[ 0.99993216]
 [ 1.99980262]]

          ** db **
         0.499935230625

          ** cost **
         6.000064773192205

d) Optimization

You have initialized your parameters.
You are also able to compute a cost function and its gradient.
Now, you want to update the parameters using gradient descent.

Exercise: Write down the optimization function. The goal is to learn $w$ and $b$ by minimizing the cost function $J$. For a parameter $\theta$, the update rule is $ \theta = \theta - \alpha \text{ } d\theta$, where $\alpha$ is the learning rate.
Can you write Python code for it?

# GRADED FUNCTION: optimize

def optimize(w, b, X, Y, num_iterations, learning_rate, print_cost = False):
    """
    This function optimizes w and b by running a gradient descent algorithm

    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of shape (num_px * num_px * 3, number of examples)
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat), of shape (1, number of examples)
    num_iterations -- number of iterations of the optimization loop
    learning_rate -- learning rate of the gradient descent update rule
    print_cost -- True to print the loss every 100 steps

    Returns:
    params -- dictionary containing the weights w and bias b
    grads -- dictionary containing the gradients of the weights and bias with respect to the cost function
    costs -- list of all the costs computed during the optimization, this will be used to plot the learning curve.

    Tips:
    You basically need to write down two steps and iterate through them:
        1) Calculate the cost and the gradient for the current parameters. Use propagate().
        2) Update the parameters using gradient descent rule for w and b.
    """

    costs = []

    for i in range(num_iterations):

        # Cost and gradient calculation (≈ 1-4 lines of code)
        ### START CODE HERE ###
        grads, cost =  propagate(w, b, X, Y)
        ### END CODE HERE ###

        # Retrieve derivatives from grads
        dw = grads["dw"]
        db = grads["db"]

        # update rule (≈ 2 lines of code)
        ### START CODE HERE ###
        w = w - np.dot(learning_rate, dw)
        b = b -  learning_rate* db
        ### END CODE HERE ###

        # Record the costs
        if i % 100 == 0:
            costs.append(cost)

        # Print the cost every 100 training examples
        if print_cost and i % 100 == 0:
            print ("Cost after iteration %i: %f" %(i, cost))

    params = {"w": w,
...


I am working on the file "Neural Networks and Deep Learning/Logistic_Regression_with_a_Neural_Network_mindset_v3.ipynb".
The first task is:
Expected Output:

    **train_set_x_flatten shape**
     (12288, 209)

    **train_set_y shape**
    (1, 209)

    **test_set_x_flatten shape**
    (12288, 50)

    **test_set_y shape**
    (1, 50)

  **sanity check after reshaping**
  [17 31 56 22 33]

To represent color images, the red, green and blue channels (RGB) must be specified for each pixel, and so the pixel value is actually a vector of three numbers ranging from 0 to 255.
One common preprocessing step in machine learning is to center and standardize your dataset, meaning that you substract the mean of the whole numpy array from each example, and then divide each example by the standard deviation of the whole numpy array. But for picture datasets, it is simpler and more convenient and works almost as well to just divide every row of the dataset by 255 (the maximum value of a pixel channel).

Let's standardize our dataset.
Can you write Python code for it?

train_set_x = train_set_x_flatten/255.
test_set_x = test_set_x_flatten/255.


I am working on the file "Neural Networks and Deep Learning/Logistic_Regression_with_a_Neural_Network_mindset_v3.ipynb".
The first task is:
Expected Output:

        **w**
       [[ 0.1124579 ]
 [ 0.23106775]]

        **b**
        1.55930492484

        **dw**
        [[ 0.90158428]
 [ 1.76250842]]

        **db**
        0.430462071679

Exercise: The previous function will output the learned w and b. We are able to use w and b to predict the labels for a dataset X. Implement the predict() function. There is two steps to computing predictions:

Calculate $\hat{Y} = A = \sigma(w^T X + b)$

Convert the entries of a into 0 (if activation <= 0.5) or 1 (if activation > 0.5), stores the predictions in a vector Y_prediction. If you wish, you can use an if/else statement in a for loop (though there is also a way to vectorize this).
Can you write Python code for it?

# GRADED FUNCTION: predict

def predict(w, b, X):
    '''
    Predict whether the label is 0 or 1 using learned logistic regression parameters (w, b)

    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of size (num_px * num_px * 3, number of examples)

    Returns:
    Y_prediction -- a numpy array (vector) containing all predictions (0/1) for the examples in X
    '''

    m = X.shape[1]
    Y_prediction = np.zeros((1,m))
    w = w.reshape(X.shape[0], 1)

    # Compute vector "A" predicting the probabilities of a cat being present in the picture
    ### START CODE HERE ### (≈ 1 line of code)
    A = sigmoid(np.dot(w.T, X)+b)
    ### END CODE HERE ###

    for i in range(A.shape[1]):

        # Convert probabilities A[0,i] to actual predictions p[0,i]
        ### START CODE HERE ### (≈ 4 lines of code)
        if A[0][i]>0.5:
            Y_prediction[0][i]=1
        else:
            Y_prediction[0][i]=0

        ### END CODE HERE ###

    assert(Y_prediction.shape == (1, m))

    return Y_prediction

print ("predictions = " + str(predict(w, b, X)))


I am working on the file "Neural Networks and Deep Learning/Logistic_Regression_with_a_Neural_Network_mindset_v3.ipynb".
The first task is:
Expected Output:

         **Train Accuracy**
         99.04306220095694 %

        **Test Accuracy**
         70.0 %

Comment: Training accuracy is close to 100%. This is a good sanity check: your model is working and has high enough capacity to fit the training data. Test error is 68%. It is actually not bad for this simple model, given the small dataset we used and that logistic regression is a linear classifier. But no worries, you'll build an even better classifier next week!
Also, you see that the model is clearly overfitting the training data. Later in this specialization you will learn how to reduce overfitting, for example by using regularization. Using the code below (and changing the index variable) you can look at predictions on pictures of the test set.
Can you write Python code for it?

num_px=64
#test_set_x[:,1].reshape((num_px, num_px, 3))

# Example of a picture that was wrongly classified.
index = 1
plt.imshow(test_set_x[:,index].reshape((num_px, num_px, 3)))
print ("y = " + str(test_set_y[0, index]) + ", you predicted that it is a \"" + classes[d["Y_prediction_test"][0, index]].decode("utf-8") +  "\" picture.")


I am working on the file "Neural Networks and Deep Learning/Logistic_Regression_with_a_Neural_Network_mindset_v3.ipynb".
The first task is:
Expected Output:

    **sigmoid([0, 2])**
     [ 0.5         0.88079708]

4.2 - Initializing parameters
Exercise: Implement parameter initialization in the cell below. You have to initialize w as a vector of zeros. If you don't know what numpy function to use, look up np.zeros() in the Numpy library's documentation.
Can you write Python code for it?

# GRADED FUNCTION: initialize_with_zeros

def initialize_with_zeros(dim):
    """
    This function creates a vector of zeros of shape (dim, 1) for w and initializes b to 0.

    Argument:
    dim -- size of the w vector we want (or number of parameters in this case)

    Returns:
    w -- initialized vector of shape (dim, 1)
    b -- initialized scalar (corresponds to the bias)
    """

    ### START CODE HERE ### (≈ 1 line of code)
    w = np.zeros((dim,1))
    b = 0
    ### END CODE HERE ###

    assert(w.shape == (dim, 1))
    assert(isinstance(b, float) or isinstance(b, int))

    return w, b

dim = 2
w, b = initialize_with_zeros(dim)
print ("w = " + str(w))
print ("b = " + str(b))


I am working on the file "Neural Networks and Deep Learning/Logistic_Regression_with_a_Neural_Network_mindset_v3.ipynb".
The first task is:
Interpretation:
- Different learning rates give different costs and thus different predictions results.
- If the learning rate is too large (0.01), the cost may oscillate up and down. It may even diverge (though in this example, using 0.01 still eventually ends up at a good value for the cost).
- A lower cost doesn't mean a better model. You have to check if there is possibly overfitting. It happens when the training accuracy is a lot higher than the test accuracy.
- In deep learning, we usually recommend that you:
    - Choose the learning rate that better minimizes the cost function.
    - If your model overfits, use other techniques to reduce overfitting. (We'll talk about this in later videos.)
7 - Test with your own image (optional/ungraded exercise)
Congratulations on finishing this assignment. You can use your own image and see the output of your model. To do that:
    1. Click on "File" in the upper bar of this notebook, then click "Open" to go on your Coursera Hub.
    2. Add your image to this Jupyter Notebook's directory, in the "images" folder
    3. Change your image's name in the following code
    4. Run the code and check if the algorithm is right (1 = cat, 0 = non-cat)!
Can you write Python code for it?

## START CODE HERE ## (PUT YOUR IMAGE NAME)
my_image = "h1.jpg"   # change this to the name of your image file
## END CODE HERE ##

# We preprocess the image to fit your algorithm.
fname = "images/" + my_image
image = np.array(ndimage.imread(fname, flatten=False))

my_image = scipy.misc.imresize(image, size=(num_px,num_px)).reshape((1, num_px*num_px*3)).T
my_predicted_image = predict(d["w"], d["b"], my_image)

plt.imshow(image)
print("y = " + str(np.squeeze(my_predicted_image)) + ", your algorithm predicts a \"" + classes[int(np.squeeze(my_predicted_image)),].decode("utf-8") +  "\" picture.")

 ## START CODE HERE ## (PUT YOUR IMAGE NAME)
my_image = "h2.jpeg"   # change this to the name of your image file
## END CODE HERE ##

# We preprocess the image to fit your algorithm.
fname = "images/" + my_image
image = np.array(ndimage.imread(fname, flatten=False))

my_image = scipy.misc.imresize(image, size=(num_px,num_px)).reshape((1, num_px*num_px*3)).T
my_predicted_image = predict(d["w"], d["b"], my_image)

plt.imshow(image)
print("y = " + str(np.squeeze(my_predicted_image)) + ", your algorithm predicts a \"" + classes[int(np.squeeze(my_predicted_image)),].decode("utf-8") +  "\" picture.")


I am working on the file "Neural Networks and Deep Learning/Logistic_Regression_with_a_Neural_Network_mindset_v3.ipynb".
The first task is:
Let's also plot the cost function and the gradients.
Can you write Python code for it?

# Plot learning curve (with costs)
costs = np.squeeze(d['costs'])
plt.plot(costs)
plt.ylabel('cost')
plt.xlabel('iterations (per hundreds)')
plt.title("Learning rate =" + str(d["learning_rate"]))
plt.show()


I am working on the file "Neural Networks and Deep Learning/Logistic_Regression_with_a_Neural_Network_mindset_v3.ipynb".
The first task is:
Run the following cell to train your model.
Can you write Python code for it?

d = model(train_set_x, train_set_y, test_set_x, test_set_y, num_iterations = 2000, learning_rate = 0.005, print_cost = True)


I am working on the file "Neural Networks and Deep Learning/Logistic_Regression_with_a_Neural_Network_mindset_v3.ipynb".
The first task is:
We added "_orig" at the end of image datasets (train and test) because we are going to preprocess them. After preprocessing, we will end up with train_set_x and test_set_x (the labels train_set_y and test_set_y don't need any preprocessing).
Each line of your train_set_x_orig and test_set_x_orig is an array representing an image. You can visualize an example by running the following code. Feel free also to change the index value and re-run to see other images.
Can you write Python code for it?

# Example of a picture
index = 25
plt.imshow(train_set_x_orig[index])
print ("y = " + str(train_set_y[:, index]) + ", it's a '" + classes[np.squeeze(train_set_y[:, index])].decode("utf-8") +  "' picture.")


I am working on the file "Neural Networks and Deep Learning/Python_Basics_With_Numpy_v3.ipynb".
The first task is:
Any time you need more info on a numpy function, we encourage you to look at the official documentation.
You can also create a new cell in the notebook and write np.exp? (for example) to get quick access to the documentation.
Exercise: Implement the sigmoid function using numpy.
Instructions: x could now be either a real number, a vector, or a matrix. The data structures we use in numpy to represent these shapes (vectors, matrices...) are called numpy arrays. You don't need to know more for now.
$$ \text{For } x \in \mathbb{R}^n \text{,     } sigmoid(x) = sigmoid\begin{pmatrix}
    x_1  \
    x_2  \
    ...  \
    x_n  \
\end{pmatrix} = \begin{pmatrix}
    \frac{1}{1+e^{-x_1}}  \
    \frac{1}{1+e^{-x_2}}  \
    ...  \
    \frac{1}{1+e^{-x_n}}  \
\end{pmatrix}\tag{1} $$
Can you write Python code for it?

# GRADED FUNCTION: sigmoid

import numpy as np # this means you can access numpy functions by writing np.function() instead of numpy.function()

def sigmoid(x):
    """
    Compute the sigmoid of x

    Arguments:
    x -- A scalar or numpy array of any size

    Return:
    s -- sigmoid(x)
    """

    ### START CODE HERE ### (≈ 1 line of code)
    s = 1/(1+np.exp(-x))
    ### END CODE HERE ###

    return s

x = np.array([1, 2, 3])
sigmoid(x)


I am working on the file "Neural Networks and Deep Learning/Python_Basics_With_Numpy_v3.ipynb".
The first task is:
As you may have noticed, the vectorized implementation is much cleaner and more efficient. For bigger vectors/matrices, the differences in running time become even bigger.
Note that np.dot() performs a matrix-matrix or matrix-vector multiplication. This is different from np.multiply() and the * operator (which is equivalent to  .* in Matlab/Octave), which performs an element-wise multiplication.
2.1 Implement the L1 and L2 loss functions
Exercise: Implement the numpy vectorized version of the L1 loss. You may find the function abs(x) (absolute value of x) useful.
Reminder:
- The loss is used to evaluate the performance of your model. The bigger your loss is, the more different your predictions ($ \hat{y} $) are from the true values ($y$). In deep learning, you use optimization algorithms like Gradient Descent to train your model and to minimize the cost.
- L1 loss is defined as:
$$\begin{align} & L_1(\hat{y}, y) = \sum_{i=0}^m|y^{(i)} - \hat{y}^{(i)}| \end{align}\tag{6}$$
Can you write Python code for it?

# GRADED FUNCTION: L1

def L1(yhat, y):
    """
    Arguments:
    yhat -- vector of size m (predicted labels)
    y -- vector of size m (true labels)

    Returns:
    loss -- the value of the L1 loss function defined above
    """

    ### START CODE HERE ### (≈ 1 line of code)
    loss = np.sum(np.abs(y-yhat))
    ### END CODE HERE ###

    return loss

yhat = np.array([.9, 0.2, 0.1, .4, .9])
y = np.array([1, 0, 0, 1, 1])
print("L1 = " + str(L1(yhat,y)))


I am working on the file "Neural Networks and Deep Learning/Python_Basics_With_Numpy_v3.ipynb".
The first task is:
Expected Output:
<table style="width:20%">

     <tr>
       <td> **L1** </td>
       <td> 1.1 </td>
     </tr>
</table>

Exercise: Implement the numpy vectorized version of the L2 loss. There are several way of implementing the L2 loss but you may find the function np.dot() useful. As a reminder, if $x = [x_1, x_2, ..., x_n]$, then np.dot(x,x) = $\sum_{j=0}^n x_j^{2}$.

L2 loss is defined as $$\begin{align} & L_2(\hat{y},y) = \sum_{i=0}^m(y^{(i)} - \hat{y}^{(i)})^2 \end{align}\tag{7}$$
Can you write Python code for it?

# GRADED FUNCTION: L2

def L2(yhat, y):
    """
    Arguments:
    yhat -- vector of size m (predicted labels)
    y -- vector of size m (true labels)

    Returns:
    loss -- the value of the L2 loss function defined above
    """

    ### START CODE HERE ### (≈ 1 line of code)
    loss = np.sum((y-yhat)**2)
    ### END CODE HERE ###

    return loss

yhat = np.array([.9, 0.2, 0.1, .4, .9])
y = np.array([1, 0, 0, 1, 1])
print("L2 = " + str(L2(yhat,y)))


I am working on the file "Neural Networks and Deep Learning/Python_Basics_With_Numpy_v3.ipynb".
The first task is:
Expected Output:

        **softmax(x)**
        [[  9.80897665e-01   8.94462891e-04   1.79657674e-02   1.21052389e-04
    1.21052389e-04]
 [  8.78679856e-01   1.18916387e-01   8.01252314e-04   8.01252314e-04
    8.01252314e-04]]

Note:
- If you print the shapes of x_exp, x_sum and s above and rerun the assessment cell, you will see that x_sum is of shape (2,1) while x_exp and s are of shape (2,5). x_exp/x_sum works due to python broadcasting.
Congratulations! You now have a pretty good understanding of python numpy and have implemented a few useful functions that you will be using in deep learning.

What you need to remember:
- np.exp(x) works for any np.array x and applies the exponential function to every coordinate
- the sigmoid function and its gradient
- image2vector is commonly used in deep learning
- np.reshape is widely used. In the future, you'll see that keeping your matrix/vector dimensions straight will go toward eliminating a lot of bugs.
- numpy has efficient built-in functions
- broadcasting is extremely useful
2) Vectorization
In deep learning, you deal with very large datasets. Hence, a non-computationally-optimal function can become a huge bottleneck in your algorithm and can result in a model that takes ages to run. To make sure that your code is  computationally efficient, you will use vectorization. For example, try to tell the difference between the following implementations of the dot/outer/elementwise product.
Can you write Python code for it?

import time

x1 = [9, 2, 5, 0, 0, 7, 5, 0, 0, 0, 9, 2, 5, 0, 0]
x2 = [9, 2, 2, 9, 0, 9, 2, 5, 0, 0, 9, 2, 5, 0, 0]

### CLASSIC DOT PRODUCT OF VECTORS IMPLEMENTATION ###
tic = time.process_time()
dot = 0
for i in range(len(x1)):
    dot+= x1[i]*x2[i]
toc = time.process_time()
print ("dot = " + str(dot) + "\n ----- Computation time = " + str(1000*(toc - tic)) + "ms")

### CLASSIC OUTER PRODUCT IMPLEMENTATION ###
tic = time.process_time()
outer = np.zeros((len(x1),len(x2))) # we create a len(x1)*len(x2) matrix with only zeros
for i in range(len(x1)):
    for j in range(len(x2)):
        outer[i,j] = x1[i]*x2[j]
toc = time.process_time()
print ("outer = " + str(outer) + "\n ----- Computation time = " + str(1000*(toc - tic)) + "ms")

### CLASSIC ELEMENTWISE IMPLEMENTATION ###
tic = time.process_time()
mul = np.zeros(len(x1))
for i in range(len(x1)):
    mul[i] = x1[i]*x2[i]
toc = time.process_time()
print ("elementwise multiplication = " + str(mul) + "\n ----- Computation time = " + str(1000*(toc - tic)) + "ms")

### CLASSIC GENERAL DOT PRODUCT IMPLEMENTATION ###
W = np.random.rand(3,len(x1)) # Random 3*len(x1) numpy array
tic = time.process_time()
gdot = np.zeros(W.shape[0])
for i in range(W.shape[0]):
    for j in range(len(x1)):
        gdot[i] += W[i,j]*x1[j]
toc = time.process_time()
print ("gdot = " + str(gdot) + "\n ----- Computation time = " + str(1000*(toc - tic)) + "ms")

x1 = [9, 2, 5, 0, 0, 7, 5, 0, 0, 0, 9, 2, 5, 0, 0]
x2 = [9, 2, 2, 9, 0, 9, 2, 5, 0, 0, 9, 2, 5, 0, 0]

### VECTORIZED DOT PRODUCT OF VECTORS ###
tic = time.process_time()
dot = np.dot(x1,x2)
toc = time.process_time()
print ("dot = " + str(dot) + "\n ----- Computation time = " + str(1000*(toc - tic)) + "ms")

### VECTORIZED OUTER PRODUCT ###
tic = time.process_time()
outer = np.outer(x1,x2)
toc = time.process_time()
print ("outer = " + str(outer) + "\n ----- Computation time = " + str(1000*(toc - tic)) + "ms")

### VECTORIZED ELEMENTWISE MULTIPLICATION ###
tic = time.process_time()
mul = np.multiply(x1,x2)
toc =...


I am working on the file "Neural Networks and Deep Learning/Python_Basics_With_Numpy_v3.ipynb".
The first task is:
Expected Output:

    ** basic_sigmoid(3) **
        0.9525741268224334

Actually, we rarely use the "math" library in deep learning because the inputs of the functions are real numbers. In deep learning we mostly use matrices and vectors. This is why numpy is more useful.
Can you write Python code for it?

### One reason why we use "numpy" instead of "math" in Deep Learning ###
x = [1, 2, 3]
basic_sigmoid(x) # you will see this give an error when you run it, because x is a vector.


I am working on the file "Neural Networks and Deep Learning/Python_Basics_With_Numpy_v3.ipynb".
The first task is:
Expected Output:

        **normalizeRows(x)**
        [[ 0.          0.6         0.8       ]
 [ 0.13736056  0.82416338  0.54944226]]

Note:
In normalizeRows(), you can try to print the shapes of x_norm and x, and then rerun the assessment. You'll find out that they have different shapes. This is normal given that x_norm takes the norm of each row of x. So x_norm has the same number of rows but only 1 column. So how did it work when you divided x by x_norm? This is called broadcasting and we'll talk about it now!
1.5 - Broadcasting and the softmax function
A very important concept to understand in numpy is "broadcasting". It is very useful for performing mathematical operations between arrays of different shapes. For the full details on broadcasting, you can read the official broadcasting documentation.
Exercise: Implement a softmax function using numpy. You can think of softmax as a normalizing function used when your algorithm needs to classify two or more classes. You will learn more about softmax in the second course of this specialization.
Instructions:
- $ \text{for } x \in \mathbb{R}^{1\times n} \text{,     } softmax(x) = softmax(\begin{bmatrix}
    x_1  &&
    x_2 &&
    ...  &&
    x_n
\end{bmatrix}) = \begin{bmatrix}
     \frac{e^{x_1}}{\sum_{j}e^{x_j}}  &&
    \frac{e^{x_2}}{\sum_{j}e^{x_j}}  &&
    ...  &&
    \frac{e^{x_n}}{\sum_{j}e^{x_j}}
\end{bmatrix} $

$\text{for a matrix } x \in \mathbb{R}^{m \times n} \text{,  $x_{ij}$ maps to the element in the $i^{th}$ row and $j^{th}$ column of $x$, thus we have: }$  $$softmax(x) = softmax\begin{bmatrix}
    x_{11} & x_{12} & x_{13} & \dots  & x_{1n} \
    x_{21} & x_{22} & x_{23} & \dots  & x_{2n} \
    \vdots & \vdots & \vdots & \ddots & \vdots \
    x_{m1} & x_{m2} & x_{m3} & \dots  & x_{mn}
\end{bmatrix} = \begin{bmatrix}
    \frac{e^{x_{11}}}{\sum_{j}e^{x_{1j}}} & \frac{e^{x_{12}}}{\sum_{j}e^{x_{1j}}} & \frac{e^{x_{13}}}{\sum_{j}e^{x_{1j}}} & \dots  &...
Can you write Python code for it?

# GRADED FUNCTION: softmax

def softmax(x):
    """Calculates the softmax for each row of the input x.

    Your code should work for a row vector and also for matrices of shape (n, m).

    Argument:
    x -- A numpy matrix of shape (n,m)

    Returns:
    s -- A numpy matrix equal to the softmax of x, of shape (n,m)
    """

    ### START CODE HERE ### (≈ 3 lines of code)
    # Apply exp() element-wise to x. Use np.exp(...).
    x_exp = np.exp(x)

    # Create a vector x_sum that sums each row of x_exp. Use np.sum(..., axis = 1, keepdims = True).
    x_sum = np.sum(x_exp,axis=1,keepdims=True)

    # Compute softmax(x) by dividing x_exp by x_sum. It should automatically use numpy broadcasting.
    s = x_exp/x_sum

    ### END CODE HERE ###

    return s

x = np.array([
    [9, 2, 5, 0, 0],
    [7, 5, 0, 0 ,0]])
print("softmax(x) = " + str(softmax(x)))


I am working on the file "Neural Networks and Deep Learning/Python_Basics_With_Numpy_v3.ipynb".
The first task is:
Expected Output:

         **sigmoid([1,2,3])**
         array([ 0.73105858,  0.88079708,  0.95257413])

1.2 - Sigmoid gradient
As you've seen in lecture, you will need to compute gradients to optimize loss functions using backpropagation. Let's code your first gradient function.
Exercise: Implement the function sigmoid_grad() to compute the gradient of the sigmoid function with respect to its input x. The formula is: $$sigmoid_derivative(x) = \sigma'(x) = \sigma(x) (1 - \sigma(x))\tag{2}$$
You often code this function in two steps:
1. Set s to be the sigmoid of x. You might find your sigmoid(x) function useful.
2. Compute $\sigma'(x) = s(1-s)$
Can you write Python code for it?

# GRADED FUNCTION: sigmoid_derivative

def sigmoid_derivative(x):
    """
    Compute the gradient (also called the slope or derivative) of the sigmoid function with respect to its input x.
    You can store the output of the sigmoid function into variables and then use it to calculate the gradient.

    Arguments:
    x -- A scalar or numpy array

    Return:
    ds -- Your computed gradient.
    """

    ### START CODE HERE ### (≈ 2 lines of code)
    s = sigmoid(x)
    ds = s*(1-s)
    ### END CODE HERE ###

    return ds

x = np.array([1, 2, 3])
print ("sigmoid_derivative(x) = " + str(sigmoid_derivative(x)))


I am working on the file "Neural Networks and Deep Learning/Python_Basics_With_Numpy_v3.ipynb".
The first task is:
Expected Output:

         **sigmoid_derivative([1,2,3])**
         [ 0.19661193  0.10499359  0.04517666]

1.3 - Reshaping arrays
Two common numpy functions used in deep learning are np.shape and np.reshape().
- X.shape is used to get the shape (dimension) of a matrix/vector X.
- X.reshape(...) is used to reshape X into some other dimension.
For example, in computer science, an image is represented by a 3D array of shape $(length, height, depth = 3)$. However, when you read an image as the input of an algorithm you convert it to a vector of shape $(lengthheight3, 1)$. In other words, you "unroll", or reshape, the 3D array into a 1D vector.

Exercise: Implement image2vector() that takes an input of shape (length, height, 3) and returns a vector of shape (length*height*3, 1). For example, if you would like to reshape an array v of shape (a, b, c) into a vector of shape (a*b,c) you would do:
python
v = v.reshape((v.shape[0]*v.shape[1], v.shape[2])) # v.shape[0] = a ; v.shape[1] = b ; v.shape[2] = c
- Please don't hardcode the dimensions of image as a constant. Instead look up the quantities you need with image.shape[0], etc.
Can you write Python code for it?

# GRADED FUNCTION: image2vector
def image2vector(image):
    """
    Argument:
    image -- a numpy array of shape (length, height, depth)

    Returns:
    v -- a vector of shape (length*height*depth, 1)
    """

    ### START CODE HERE ### (≈ 1 line of code)
    v = image.reshape((image.shape[0]*image.shape[1]*image.shape[2],1))
    ### END CODE HERE ###

    return v

# This is a 3 by 3 by 2 array, typically images will be (num_px_x, num_px_y,3) where 3 represents the RGB values
image = np.array([[[ 0.67826139,  0.29380381],
        [ 0.90714982,  0.52835647],
        [ 0.4215251 ,  0.45017551]],

       [[ 0.92814219,  0.96677647],
        [ 0.85304703,  0.52351845],
        [ 0.19981397,  0.27417313]],

       [[ 0.60659855,  0.00533165],
        [ 0.10820313,  0.49978937],
        [ 0.34144279,  0.94630077]]])

print ("image2vector(image) = " + str(image2vector(image)))


I am working on the file "Neural Networks and Deep Learning/Python_Basics_With_Numpy_v3.ipynb".
The first task is:
Expected output:
test: Hello World

What you need to remember:
- Run your cells using SHIFT+ENTER (or "Run cell")
- Write code in the designated areas using Python 3 only
- Do not modify the code outside of the designated areas
1 - Building basic functions with numpy
Numpy is the main package for scientific computing in Python. It is maintained by a large community (www.numpy.org). In this exercise you will learn several key numpy functions such as np.exp, np.log, and np.reshape. You will need to know how to use these functions for future assignments.
1.1 - sigmoid function, np.exp()
Before using np.exp(), you will use math.exp() to implement the sigmoid function. You will then see why np.exp() is preferable to math.exp().
Exercise: Build a function that returns the sigmoid of a real number x. Use math.exp(x) for the exponential function.
Reminder:
$sigmoid(x) = \frac{1}{1+e^{-x}}$ is sometimes also known as the logistic function. It is a non-linear function used not only in Machine Learning (Logistic Regression), but also in Deep Learning.

To refer to a function belonging to a specific package you could call it using package_name.function(). Run the code below to see an example with math.exp().
Can you write Python code for it?

# GRADED FUNCTION: basic_sigmoid

import math

def basic_sigmoid(x):
    """
    Compute sigmoid of x.

    Arguments:
    x -- A scalar

    Return:
    s -- sigmoid(x)
    """

    ### START CODE HERE ### (≈ 1 line of code)
    s = 1/(1+math.exp(-x))
    ### END CODE HERE ###

    return s

basic_sigmoid(3)


I am working on the file "Neural Networks and Deep Learning/Python_Basics_With_Numpy_v3.ipynb".
The first task is:
Furthermore, if x is a vector, then a Python operation such as $s = x + 3$ or $s = \frac{1}{x}$ will output s as a vector of the same size as x.
Can you write Python code for it?

# example of vector operation
x = np.array([1, 2, 3])
print (x + 3)


I am working on the file "Neural Networks and Deep Learning/Python_Basics_With_Numpy_v3.ipynb".
The first task is:
In fact, if $ x = (x_1, x_2, ..., x_n)$ is a row vector then $np.exp(x)$ will apply the exponential function to every element of x. The output will thus be: $np.exp(x) = (e^{x_1}, e^{x_2}, ..., e^{x_n})$
Can you write Python code for it?

import numpy as np

# example of np.exp
x = np.array([1, 2, 3])
print(np.exp(x)) # result is (exp(1), exp(2), exp(3))


I am working on the file "Neural Networks.ipynb".
The first task is:
Feed Forward NN
Can you write Python code for it?

def sigmoid(t):
    return 1.0 / (1 + math.exp(-t))

def neuron_output(weights, inputs):
    return sigmoid(dot(weights, inputs))

# NN is then a list of lists of lists (where layers have neurons that have weights)

def feed_forward(neural_network, input_vector):
    """NN (list of lists of lists)
    returns output from forward prop"""

    outputs = []

    # for each layer add bias, compute output, pass on
    for layer in neural_network:
        input_with_bias = input_vector + [1]
        output = [neuron_output(neuron, input_with_bias)
                 for neuron in layer]
        outputs.append(output)

        input_vector = output

    return outputs


I am working on the file "Neural Networks.ipynb".
The first task is:
Perceptrons
Can you write Python code for it?

def step_function(x):
    return 1 if x >= 0 else 0

def perceptron_output(weights, bias, x):
    """returns 1 if perceptron fires, 0 if not"""
    calculation = dot(weights, x) + bias
    return step_function(calculation)


I am working on the file "NeuralEquationFinder/NeuralEquationFinder_Part_1.ipynb".
The first task is:
Can we get away not having to use normalized features and achieve good performance?
We have seen till now that normalizing features, mini-batches of size 1 and SGD optimizer gave us one of the fastest and best convergences. Wouldn't it be cool if we don't have to normalize features? After all, we are trying to converge to a simple y = 2*x equation.
The answer to why Adam and RMSprop optimizers work with unnormalized raw features is the factor 1/sqrt(norm(gradients)) which makes sure the gradients don't explode.
If we want to use unnormalized features then it's best to use Adam optimizer and full batches for training.
Can you write Python code for it?

RunLinearNNTraining(X=X_tensor, y=y_tensor, batch_size=None, epochs=25000, learning_rate=1e-3,
                    X_test=X_test_tsr, use_optimizer='Adam', adam_betas=(0.99, 0.999))


I am working on the file "NeuralEquationFinder/NeuralEquationFinder_Part_1.ipynb".
The first task is:
Full batch gradient descent with feature normalization
Converges more smoothly and better. Needs higher learning_rate (like 100x) than unnormalized features. What else? Even SGD converges and its better than Adam!
Can you write Python code for it?

# Using Adam Optimizer
RunLinearNNTraining(X=X_norm_tsr, y=y_tensor, batch_size=None, epochs=25000, learning_rate=1e-1,
                    X_test=X_test_norm_tsr, use_optimizer='Adam')

# Look SGD is faster than Adam now. It's curve is more steeper.
RunLinearNNTraining(X=X_norm_tsr, y=y_tensor, batch_size=None, epochs=25000, learning_rate=1e-1,
                    X_test=X_test_norm_tsr, use_optimizer='SGD')


I am working on the file "NeuralEquationFinder/NeuralEquationFinder_Part_1.ipynb".
The first task is:
Full batch gradient descent with unnormalized features
SGD, ASGD, Adadelta and Adagrad:
It doesn't converge to global optima and for some of them the learning rate has to be set very low if not the gradients explode. Ouch!
Adam and RMSprop:
It needs a lot of epochs to converge close to global optima. The solution and loss is not good enough.
Can you write Python code for it?

# use_optimizer can be Adam, RMSprop, Adadelta, ASGD, SGD, Adagrad
RunLinearNNTraining(X=X_tensor, y=y_tensor, batch_size=None, epochs=25000, learning_rate=1e-3,
                    X_test=X_test_tsr, use_optimizer='Adam')


I am working on the file "NeuralEquationFinder/NeuralEquationFinder_Part_1.ipynb".
The first task is:
Introduction
Let's say y = f(x_1, x_2, ..., x_n)
If f is a mathematical function of x_1, x_2, x_3, ..., x_n then we can wire up a neural network to try to find this function given some examples of (x_1, x_2, ...., x_n, y)
1st: Normal Equation Method
Can you write Python code for it?

# Let's try to find the equation y = 2 * x
# We have 6 examples:- (x,y) = (0.1,0.2), (1,2), (2, 4), (3, 6), (-4, -8), (25, 50)
# Let's assume y is a linear combination of the features x, x^2, x^3
# We know that Normal Equation gives us the exact solution so let's first use that

N = 6
x = np.array([0.1, 1, 2, 3, -4, 25])
y = np.array([0.2, 2, 4, 6, -8, 50])
x_2 = x**2
x_3 = x**3

X = np.ones((N, 4))
X[:,1] = x
X[:,2] = x_2
X[:,3] = x_3

_, D = np.shape(X)
regularization_strength = 0.0
XtX = (X.T).dot(X)
I = np.eye(D, dtype=float)
XtX_RI = XtX + regularization_strength*I
XtY = (X.T).dot(y)
w = np.linalg.solve(XtX_RI, XtY)
y_pred = X.dot(w)
loss = np.sqrt(np.mean((y_pred-y)**2))
# As expected w ~ [0 2 0 0]
print("W : ", w)
print("Predicted Y : ", y_pred)
print("RMS loss : ", loss)


I am working on the file "NeuralEquationFinder/NeuralEquationFinder_Part_1.ipynb".
The first task is:
Mini-Batch Gradient Descent with UNnormalized features and batch_size = 1
SGD doesn't converge and Adam performs bad. Well, worth a try.
Can you write Python code for it?

RunLinearNNTraining(X=X_tensor, y=y_tensor, batch_size=1, epochs=150, learning_rate=1e-3,
                    X_test=X_test_tsr, use_optimizer='Adam')


I am working on the file "NeuralEquationFinder/NeuralEquationFinder_Part_1.ipynb".
The first task is:
Mini-Batch Gradient Descent with normalized features
Cool! normalization of features is better for convergence.
It takes longer than Full-Batch Gradient Descent and results are comparable to Full-Batch Gradient Descent with normalized features.
Can you write Python code for it?

# SGD is better than Adam in this case so using SGD
RunLinearNNTraining(X=X_norm_tsr, y=y_tensor, batch_size=25, epochs=3000, learning_rate=1e-1,
                    X_test=X_test_norm_tsr, use_optimizer='SGD')


I am working on the file "NeuralEquationFinder/NeuralEquationFinder_Part_1.ipynb".
The first task is:
Mini-Batch Gradient Descent with normalized features and batch_size = 1
Its much faster and better than the previous approaches for this problem. Awesome!
Can you write Python code for it?

# SGD is better than Adam in this case so using SGD
RunLinearNNTraining(X=X_norm_tsr, y=y_tensor, batch_size=1, epochs=25, learning_rate=1e-1,
                    X_test=X_test_norm_tsr, use_optimizer='SGD')


I am working on the file "NeuralEquationFinder/NeuralEquationFinder_Part_1.ipynb".
The first task is:
Mini-Batch Gradient Descent with unnormalized features
Requires much lesser #epochs because in each epoch we are doing more gradient updates due to mini-batches.
It takes longer than Full-Batch Gradient Descent! Ouch! Results are comparable to Full-Batch Gradient Descent.
Can you write Python code for it?

# SGD doesn't work with unnormalized features so using Adam.
RunLinearNNTraining(X=X_tensor, y=y_tensor, batch_size=25, epochs=3000, learning_rate=1e-3,
                    X_test=X_test_tsr, use_optimizer='Adam')


I am working on the file "NeuralEquationFinder/NeuralEquationFinder_Part_2.ipynb".
The first task is:
Normalized features
After normalizing the features, SGD is not converging! what? and there was no performance advantage compared to unnormalized features.
Can you write Python code for it?

model = torch.nn.Sequential(OrderedDict([
            ("Pow123Net", Pow123Net()),
            ("FC", torch.nn.Linear(3, 1))]
        ))
RunV2NNTraining(X=X_norm_tsr.view(-1,1), y=y_tensor, model=model, batch_size=None, epochs=25000, learning_rate=1e-1,
                    X_test=X_test_norm_tsr.view(-1,1), use_optimizer='Adam')


I am working on the file "NeuralEquationFinder/NeuralEquationFinder_Part_2.ipynb".
The first task is:
Unnormalized features
Can you write Python code for it?

# use_optimizer can be Adam, RMSprop, Adadelta, ASGD, SGD, Adagrad
model = torch.nn.Sequential(OrderedDict([
            ("Pow123Net", Pow123Net()),
            ("FC", torch.nn.Linear(3, 1))]
        ))
RunV2NNTraining(X=X_tensor.view(-1,1), y=y_tensor, model=model, batch_size=None, epochs=25000, learning_rate=5e-3,
                    X_test=X_test_tsr.view(-1,1), use_optimizer='Adam')

# Now, how do we find the equation?
# One way to find is to see the effect of each activation on the loss
print("Final Loss: ", GetV2NNLoss(X=X_tensor.view(-1,1), y=y_tensor, model=model))

# mask f(x) = x
Pow123Net_Mask[0] = 0.0
print("Loss with x masked: ", GetV2NNLoss(X=X_tensor.view(-1,1), y=y_tensor, model=model))
Pow123Net_Mask[0] = 1.0

# mask f(x) = x^2
Pow123Net_Mask[1] = 0.0
print("Loss with x^2 masked: ", GetV2NNLoss(X=X_tensor.view(-1,1), y=y_tensor, model=model))
Pow123Net_Mask[1] = 1.0

# mask f(x) = x^3
Pow123Net_Mask[2] = 0.0
print("Loss with x^3 masked: ", GetV2NNLoss(X=X_tensor.view(-1,1), y=y_tensor, model=model))
Pow123Net_Mask[2] = 1.0

# Clearly activations X^2 and X^3 are not important
# Now what is the final equation?
p1_w = None
p1_b = None
fc1_w = None
fc_b = None
for name, param in model.named_parameters():
    if name == 'Pow123Net.p1.linear.weight':
        p1_w = param.data[0]
    if name == 'Pow123Net.p1.linear.bias':
        p1_b = param.data[0]
    if name == 'FC.weight':
        fc1_w = param.data[0,0]
    if name == 'FC.bias':
        fc_b = param.data[0]

coeff_x = p1_w*fc1_w
const = p1_b*fc1_w+fc_b
print("Finally the equation is y = ",coeff_x[0],"*x + ", const)
print("Pretty close to y = 2*x")


I am working on the file "NeuralNetwork - AtomicCharges.ipynb".
The first task is:
5. Putting it all together
After training NNs for each atomic species (MBTR), combine them into one program that predicts charges for all atoms in a molecule.

Compute local MBTR for the molecule below.
Compute all atomic charges with the NNs.
Is the total charge zero? If not, normalise it.

Note: Careful about the training: if the training data was transformed, the MBTR here should be as well.
Can you write Python code for it?

# atomic positions as matrix
molxyz = numpy.load("./data/molecule.coords.npy")
# atom types
moltyp = numpy.load("./data/molecule.types.npy")

atoms_sys = Atoms(positions=molxyz, numbers=moltyp)
view(atoms_sys)

# compute MBTR descriptor for the molecule
# ...

# compute all atomic charghes using previously trained NNs
# ...


I am working on the file "NeuralNetwork - AtomicCharges.ipynb".
The first task is:
Check the ANN quality with a regression plot, showing the mismatch between the exact and NN predicted outputs for the validation set.
Can you write Python code for it?

# evaluate the training and validation error
trainMLOut = nn.predict(trainIn)
validMLOut = nn.predict(validIn)

print ("Mean Abs Error (training)  : ", (numpy.abs(trainMLOut-trainOut)).mean())
print ("Mean Abs Error (validation): ", (numpy.abs(validMLOut-validOut)).mean())

plt.plot(validOut,validMLOut,'o')
plt.plot([Z-1,Z+1],[Z-1,Z+1]) # perfect fit line
plt.xlabel('correct output')
plt.ylabel('NN output')
plt.show()

# error histogram
plt.hist(validMLOut-validOut,50)
plt.xlabel("Error")
plt.ylabel("Occurrences")
plt.show()


I am working on the file "NeuralNetwork - AtomicCharges.ipynb".
The first task is:
Exercises
1. Compare descriptors
Test the accuracy of different descriptors with the same NN size.
Can you write Python code for it?


I am working on the file "NeuralNetwork - AtomicCharges.ipynb".
The first task is:
Let's pick a descriptor and an atomic type.
Available descriptors are:
* boba: bag of bonds (per atom)
* boba2: bag of bonds - advanced (per atom)
* acsf: atom centered symmetry functions - 40k atoms max for each type
* gnn: graph based fingerprint from NanoLayers
* soap: smooth overlap of atomic positions (per atom)
* mbtr: manybody tensor representation (per atom)
Possible atom types are:
* 1 = Hydrogen
* 6 = Carbon
* 7 = Nitrogen
* 8 = Oxygen
* 9 = Fluorine
Can you write Python code for it?

# Z is the atom type: allowed values are 1, 6, 7, 8, or 9
Z = 6

# TYPE is the descriptor type
TYPE = "mbtr"

#show descriptor details
print("\nDescriptor details")
desc = open("./data/descriptor."+TYPE+".txt","r").readlines()
for l in desc: print(l.strip())
print(" ")


I am working on the file "NeuralNetwork - AtomicCharges.ipynb".
The first task is:
Load the databases with the descriptors (input) and the correct charge densities (output). Databases are quite big, so we can decide how many samples to use for training.
Can you write Python code for it?

# load input/output data
trainIn = load_npz("./data/charge."+str(Z)+".input."+TYPE+".npz").toarray()
trainOut = numpy.load("./data/charge."+str(Z)+".output.npy")
trainOut = trainOut[0:trainIn.shape[0]]

# decide how many samples to take from the database
samples  = min(trainIn.shape[0], 9000) # change the number here if needed!

print("training samples:   "+str(samples))
print("validation samples: "+str(trainIn.shape[0]-samples))
print("number of features: {}".format(trainIn.shape[1]))

# 70-30 split between training and validation
validIn = trainIn[samples:]
validOut = trainOut[samples:]

trainIn  = trainIn[0:samples]
trainOut = trainOut[0:samples]

# shift and scale the inputs - OPTIONAL
train_mean = numpy.mean(trainIn, axis=0)
train_std = numpy.std(trainIn, axis=0)
train_std[train_std == 0] = 1
for a in range(trainIn.shape[1]):
    trainIn[:,a] -= train_mean[a]
    trainIn[:,a] /= train_std[a]

# also for validation set
for a in range(validIn.shape[1]):
    validIn[:,a] -= train_mean[a]
    validIn[:,a] /= train_std[a]

# show the first few descriptors
print("\nDescriptors for the first 5 atoms:")
print(trainIn[0:5])


I am working on the file "NeuralNetwork - AtomicCharges.ipynb".
The first task is:
Next we setup a multilayer perceptron of suitable size. Out package of choice is scikit-learn, but more efficient ones are available.

Check the scikit-learn documentation for a list of parameters.
Can you write Python code for it?

# setup the neural network
# alpha is a regularisation parameter, explained later

nn = MLPRegressor(hidden_layer_sizes=(10),  activation='tanh', solver='adam', alpha=0.01, learning_rate='adaptive')


I am working on the file "NeuralNetwork - AtomicCharges.ipynb".
The first task is:
Training
Now comes the tough part! The idea of training is to evaluate the ANN with the training inputs and measure its error (since we know the correct outputs). It is then possible to compute the derivative (gradient) of the error w.r.t. each parameter (connections and biases). By shifting the parameters in the opposite direction of the gradient, we obtain a better set of parameters, that should give smaller error.
This procedure can be repeated until the error is minimised.

It may take a while...
Can you write Python code for it?

# use this to change some parameters during training if needed
nn.set_params(solver='lbfgs')

nn.fit(trainIn, trainOut);


I am working on the file "NeuralNetwork - Intro.ipynb".
The first task is:
This is the model function that will approximate:

A small random noise is added to the function... to spice things up!
Now we use the function to create a database of reference points, divided into training and validation sets.
The training set provides the NN with a set of examples to learn from, while the validation set is later used to check the quality of the trained NN.
The following code creates a list of $\left(x,y\right)$ points on the real plane, with $x,y \in \left[-1,1 \right]$, to use as inputs.
By applying the Model function to each one, we get a list of corresponding output:
$z = \sin\left( 0.4x - 0.3y + 0.2xy \right) + \mathrm{noise}$
Can you write Python code for it?

# training points
nTrainPts = 100
trainIn = 2*numpy.random.rand(nTrainPts,2) -1 # (x,y) points beween -1 and 1

# validation points
nValidPts = 40
validIn = 2*numpy.random.rand(nValidPts,2)-1 # (x,y) points beween -1 and 1

# apply Model to the points to compute z for the two sets
trainOut = numpy.apply_along_axis(Model, 1, trainIn)
validOut = numpy.apply_along_axis(Model, 1, validIn)

# visualise the training set
plt.scatter(trainIn[:,0], trainIn[:,1], c=trainOut[:])
plt.colorbar()
plt.show()


I am working on the file "NeuralNetwork - Intro.ipynb".
The first task is:
Training
Now comes the tough part!
The idea of training is to evaluate the NN on the training inputs and measure the error between its output and the reference values:
$$ E = \sum_{i}^{\mathrm{samples}} \left( y_i - y_i^{\mathrm{ref}} \right)^2$$
It is then possible to compute the derivative (gradient) of the error w.r.t. each parameter (connections and biases):
$$ \mathbf{G} = \frac{\partial E}{\partial \mathbf{\omega}} $$
By shifting the parameters in the opposite direction of the gradient, we obtain a better set of parameters, that should give smaller error. This procedure can be repeated until $E$ is minimised.
Can you write Python code for it?

# some NN parameters can be changed here, without re-initialising the whole thing
nn.set_params(learning_rate='adaptive', solver='lbfgs')
nn.fit(trainIn, trainOut);

# evaluate the training and validation error
trainMLOut = nn.predict(trainIn)
validMLOut = nn.predict(validIn)

print ("Mean Abs Error (training)  : " + str ((numpy.abs(trainMLOut-trainOut)).mean()))
print ("Mean Abs Error (validation): " + str ((numpy.abs(validMLOut-validOut)).mean()))

# make a nice plot
plt.plot(validOut,validMLOut,'o')
plt.plot([-1,1],[-1,1]) # perfect fit line
plt.xlabel('correct output')
plt.ylabel('NN output')
plt.show()


I am working on the file "NeuralNetwork - TotalEnergy.ipynb".
The first task is:
Check the ANN quality with a regression plot, showing the mismatch between the exact and NN predicted outputs for the validation set.
Can you write Python code for it?

# evaluate the training and validation error
trainMLOut = nn.predict(trainIn)
validMLOut = nn.predict(validIn)

print ("Mean Abs Error (training)  : ", (numpy.abs(trainMLOut-trainOut)).mean())
print ("Mean Abs Error (validation): ", (numpy.abs(validMLOut-validOut)).mean())

plt.plot(validOut,validMLOut,'o')
plt.plot([min(validOut),max(validOut)],[min(validOut),max(validOut)]) # perfect fit line
plt.xlabel('correct output')
plt.ylabel('NN output')
plt.show()

# error histogram
plt.hist(validMLOut-validOut,50)
plt.xlabel("Error")
plt.ylabel("Occurrences")
plt.show()


I am working on the file "NeuralNetwork - TotalEnergy.ipynb".
The first task is:
Exercises
1. Compare descriptors
Keeping the size of the NN constant, test the accuracy of different descriptors with the same NN size, and find the best one for these systems.
Can you write Python code for it?


I am working on the file "NeuralNetwork - TotalEnergy.ipynb".
The first task is:
Training
Now comes the tough part! The idea of training is to evaluate the ANN with the training inputs and measure its error (since we know the correct outputs). It is then possible to compute the derivative (gradient) of the error w.r.t. each parameter (connections and biases). By shifting the parameters in the opposite direction of the gradient, we obtain a better set of parameters, that should give smaller error.
This procedure can be repeated until the error is minimised.
It may take a while...
Can you write Python code for it?

# use this to change some parameters during training if the NN gets stuck in a bad spot
nn.set_params(solver='lbfgs')

nn.fit(trainIn, trainOut);


I am working on the file "NeuralNetwork - TotalEnergy.ipynb".
The first task is:
and load the databases with the descriptors (input) and the correct charge densities (output). Databases are quite big, so we can decide how many samples to use for training.
Can you write Python code for it?

# load input/output data
trainIn = load_npz("./data/energy.input."+TYPE+".npz").toarray()
trainOut = numpy.load("./data/energy.output.npy")
trainIn = trainIn.astype(dtype=numpy.float64, casting='safe')

# decide how many samples to take from the database
samples  = min(trainIn.shape[0], 9000)
vsamples = min(trainIn.shape[0]-samples,1000)
print("training samples:   "+str(samples))
print("validation samples: "+str(vsamples))
print("number of features: {}".format(trainIn.shape[1]))

# split between training and validation
validIn = trainIn[samples:samples+vsamples]
validOut = trainOut[samples:samples+vsamples]

trainIn  = trainIn[0:samples]
trainOut = trainOut[0:samples]

# shift and scale the inputs
train_mean = numpy.mean(trainIn, axis=0)
train_std = numpy.std(trainIn, axis=0)
train_std[train_std==0] = 1
for a in range(trainIn.shape[1]):
    trainIn[:,a] -= train_mean[a]

for a in range(trainIn.shape[1]):
    trainIn[:,a] /= train_std[a]
# also for validation set
for a in range(validIn.shape[1]):
    validIn[:,a] -= train_mean[a]
for a in range(validIn.shape[1]):
    validIn[:,a] /= train_std[a]

# show the first few descriptors
print("\nDescriptors for the first 5 molecules:")
print(trainIn[0:5])


I am working on the file "NeuralNetwork/auto_encorder_and_rnn.ipynb".
The first task is:
NNのクラスはすでにNN.pyからimportしてある。
Can you write Python code for it?


I am working on the file "NeuralNetwork/auto_encorder_and_rnn.ipynb".
The first task is:
nn.supervised_function(f, idata)は教師データを作成する。引数は関数とサンプルデータをとる。
Can you write Python code for it?

idata = [1, 2]

nn.supervised_function(f, idata)


I am working on the file "NeuralNetwork/auto_encorder_and_rnn.ipynb".
The first task is:
の組み合わせである。
Can you write Python code for it?

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)

a = np.array([])
b = np.array([])
c = np.array([])

nn = NN()
nn.set_network()

for x in X:
    for y in Y:
        a = np.append(a, x)
        b = np.append(b, y)

for i in range(100):
        l = np.random.choice([i for i in range(len(a))])
        m = nn.main2(1, f, [a[l], b[l]], 0.5)

for x in X:
    for y in Y:
        idata = [x, y]
        c = np.append(c, nn.realize(f, idata))

a

b

c


I am working on the file "NeuralNetwork/auto_encorder_and_rnn.ipynb".
The first task is:
下の図より、最初は交互に相手にしっぺ返しをしているが、やがて両者が裏切り合うこと状態に収束する。
Can you write Python code for it?

p1 = []
p2 = []
for i in range(len(idata1)):
    p1.append(idata1[i][0])
for i in range(len(idata2)):
    p2.append(idata2[i][0])
p1 = plt.plot(p1, label="player1")
p2 = plt.plot(p2, label="player2")
plt.legend()


I am working on the file "NeuralNetwork/auto_encorder_and_rnn.ipynb".
The first task is:
例えば(0, 0)を入力すると0.52328635を返している（つまりa[0]とb[0]を入力して、c[0]の値を返している）。
ここでは交差検定は用いていない。
Can you write Python code for it?

fig = pl.figure()
ax = Axes3D(fig)

ax.scatter(a, b, c)

pl.show()


I am working on the file "NeuralNetwork/auto_encorder_and_rnn.ipynb".
The first task is:
例３：次に相手の行動を完全には観測できない場合を考える。t期の相手の行動をt+1期にノイズが加わって知る。例えば、1期目に相手が協調したことを、確率90%で2期目に正しく知れるが、10%で裏切りと誤って伝わる場合である。
ノイズは20%の確率で加わるものとする。その他の条件は例1と同じにした。
Can you write Python code for it?

nn1 = RNN()
nn1.set_network()
nn2 = RNN()
nn2.set_network()

idata1 = [[1, 0]]
idata2 = [[0, 1]]
sdata1 = [[0]]
sdata2 = [[1]]

for t in range(10):

    for i in range(10):
        nn1.main2(idata1, sdata2, 0.9)
        nn2.main2(idata2, sdata1, 0.9)

    idata1.append([sdata1[-1][0], np.random.choice([sdata2[-1][0], 1-sdata2[-1][0]], p=[0.8, 0.2])])
    idata2.append([sdata2[-1][0], np.random.choice([sdata1[-1][0], 1-sdata1[-1][0]], p=[0.8, 0.2])])

    n1r = nn1.realize(idata1)
    n2r = nn2.realize(idata1)

    prob1 = n1r
    prob2 = n2r

    sdata1.append([np.random.choice([1, 0], p=[prob1, 1-prob1])])

    sdata2.append([np.random.choice([1, 0], p=[prob2, 1-prob2])])

idata.append([sdata1[-1][0], sdata2[-1][0]])
print nn1.realize(idata1), nn2.realize(idata2), idata1

p1 = []
p2 = []
for i in range(len(idata1)):
    p1.append(idata1[i][0])
for i in range(len(idata2)):
    p2.append(idata2[i][0])
p1 = plt.plot(p1, label="player1")
p2 = plt.plot(p2, label="player2")
plt.legend()


I am working on the file "NeuralNetwork/auto_encorder_and_rnn.ipynb".
The first task is:
入力層、中間層、出力層を作る関数を実行する。引数には層の数を用いる。
Can you write Python code for it?

nn.set_input_layer(2)

nn.set_hidden_layer(2)

nn.set_output_layer(2)


I am working on the file "NeuralNetwork/auto_encorder_and_rnn.ipynb".
The first task is:
最初の行動はRNNで指定できないので、所与となる。この初期値と裏切りに対する感応度で収束の仕方が決まる。
協調を1、裏切りを0としている。RNNの予測値は整数値でないが、p=（RNNの出力値）で次回に協調を行う。
例１：1期目に、プレーヤー１が協力、プレーヤー２が裏切り。
Can you write Python code for it?

nn1 = RNN()
nn1.set_network()
nn2 = RNN()
nn2.set_network()

idata1 = [[1, 0]]
idata2 = [[0, 1]]
sdata1 = [[0]]
sdata2 = [[1]]

for t in range(10):

    for i in range(10):
        nn1.main2(idata1, sdata2, 0.9)
        nn2.main2(idata2, sdata1, 0.9)

    idata1.append([sdata1[-1][0], sdata2[-1][0]])
    idata2.append([idata1[-1][1], idata1[-1][0]])

    n1r = nn1.realize(idata1)
    n2r = nn2.realize(idata1)
    sdata1.append([np.random.choice([1, 0], p=[n1r, 1-n1r])])

    sdata2.append([np.random.choice([1, 0], p=[n2r, 1-n2r])])

idata.append([sdata1[-1][0], sdata2[-1][0]])
print nn1.realize(idata1), nn2.realize(idata2), idata1


I am working on the file "NeuralNetwork/auto_encorder_and_rnn.ipynb".
The first task is:
次に十分大きく(100回に)してみる。
Can you write Python code for it?

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)

a = np.array([])
b = np.array([])
c = np.array([])

nn = NN()
nn.set_network(h=7)

for x in X:
    for y in Y:
        a = np.append(a, x)
        b = np.append(b, y)

evl = np.array([])

for i in range(len(a)):
    nn = NN()
    nn.set_network()
    for j in range(100):
        l = np.random.choice([i for i in range(len(a))])
        if l != i:
            nn.main2(1, f, [a[l], b[l]], 0.5)
    idata = [a[i], b[i]]
    evl = np.append(evl, math.fabs(nn.realize(f, idata) - nn.supervised_data))

np.average(evl)


I am working on the file "NeuralNetwork/auto_encorder_and_rnn.ipynb".
The first task is:
確率的勾配降下法を100回繰り返したが見た感じから近づいている。回数を10000回に増やしてみる。
Can you write Python code for it?

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)

a = np.array([])
b = np.array([])
c = np.array([])

nn = NN()
nn.set_network()

for x in X:
    for y in Y:
        a = np.append(a, x)
        b = np.append(b, y)

for i in range(10000):
        l = np.random.choice([i for i in range(len(a))])
        m = nn.main2(1, f, [a[l], b[l]], 0.5)

for x in X:
    for y in Y:
        idata = [x, y]
        c = np.append(c, nn.realize(f, idata))

fig = pl.figure()
ax = Axes3D(fig)

ax.scatter(a, b, c)

pl.show()


I am working on the file "NeuralNetwork/auto_encorder_and_rnn.ipynb".
The first task is:
見た感じ随分近づいているように見える。
最後に交差検定を行う。
初めに学習回数が極めて少ないNNである。
Can you write Python code for it?

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)

a = np.array([])
b = np.array([])
c = np.array([])

for x in X:
    for y in Y:
        a = np.append(a, x)
        b = np.append(b, y)

evl = np.array([])

for i in range(len(a)):
    nn = NN()
    nn.set_network()
    for j in range(1):
        l = np.random.choice([i for i in range(len(a))])
        if l != i:
            nn.main2(1, f, [a[l], b[l]], 0.5)
    idata = [a[i], b[i]]
    est = nn.realize(f, idata)
    evl = np.append(evl, math.fabs(est - nn.supervised_data))

np.average(evl)


I am working on the file "NeuralNetwork/auto_encorder_and_rnn.ipynb".
The first task is:
誤差の平均であるので小さい方よい。
学習回数を増やした結果、精度が上がった。
最後にオートエンコーダを作成する。回数を増やした方がよいことが分かったため、10000回学習させてみる。
Can you write Python code for it?

nn = NN()
nn.set_network()

X = np.arange(0, 1, 0.05)
Y = np.arange(0, 1, 0.05)

a = np.array([])
b = np.array([])
c = np.array([])

for x in X:
    for y in Y:
        a = np.append(a, x)
        b = np.append(b, y)

evl = np.array([])

s = [i for i in range(len(a))]

for j in range(1000):
    l = np.random.choice(s)
    nn.main2(1, f, [a[l], b[l]], 0.5)

c = np.array([])

for i in range(len(a)):
    idata = [a[i], b[i]]
    c = np.append(c, nn.realize(f, idata))

fig = pl.figure()
ax = Axes3D(fig)

ax.scatter(a, b, c)

pl.show()


I am working on the file "NeuralNetwork/nn.ipynb".
The first task is:

課題の例で使われた関数は以下の通りである。

z = (1+np.sin(4*math.pi*x_1))*x_2*1/2

NNのクラスはすでにNN.pyからimportしてある。
Can you write Python code for it?


I am working on the file "NeuralNetwork/nn.ipynb".
The first task is:
nn.calculation()は学習せずに入力層から出力層の計算を行う。nn.simulate()内にも用いられている。
次に実際に学習を行う。サンプルデータは、
Can you write Python code for it?

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)
print X, Y


I am working on the file "NeuralNetwork/nn.ipynb".
The first task is:
nn.simulate(N, eta)は引数に更新回数と学習率をとる。普通はN=1で行うべきかもしれないが、工夫として作成してみた。N回学習した後に出力層を返す。
Can you write Python code for it?


I am working on the file "NeuralNetwork/nn.ipynb".
The first task is:
nn.supervised_function(f, idata)は教師データを作成する。引数は関数とサンプルデータをとる。
Can you write Python code for it?


I am working on the file "NeuralNetwork/nn.ipynb".
The first task is:
の組み合わせである。
Can you write Python code for it?

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)

a = np.array([])
b = np.array([])
c = np.array([])

nn = NN()
nn.set_network()

for x in X:
    for y in Y:
        a = np.append(a, x)
        b = np.append(b, y)

for i in range(100):
        l = np.random.choice([i for i in range(len(a))])
        m = nn.main(1, f, [a[l], b[l]], 0.5)

for x in X:
    for y in Y:
        idata = [x, y]
        c = np.append(c, nn.realize(f, idata))

a

b

c


I am working on the file "NeuralNetwork/nn.ipynb".
The first task is:
上手く近似できないので中間層の数を変えてみる。5層にしてみる。
Can you write Python code for it?

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)

a = np.array([])
b = np.array([])
c = np.array([])

nn = NN()
nn.set_network(h=5)

for x in X:
    for y in Y:
        a = np.append(a, x)
        b = np.append(b, y)

for i in range(1000):
        l = np.random.choice([i for i in range(len(a))])
        m = nn.main(1, f, [a[l], b[l]], 0.5)

for x in X:
    for y in Y:
        idata = [x, y]
        c = np.append(c, nn.realize(f, idata))

fig = pl.figure()
ax = Axes3D(fig)

ax.scatter(a, b, c)

pl.show()


I am working on the file "NeuralNetwork/nn.ipynb".
The first task is:
例えば(0, 0)を入力すると0.52328635を返している（つまりa[0]とb[0]を入力して、c[0]の値を返している）。
Can you write Python code for it?


I am working on the file "NeuralNetwork/nn.ipynb".
The first task is:
入力層、中間層、出力層を作る関数を実行する。引数には層の数を用いる。
Can you write Python code for it?


I am working on the file "NeuralNetwork/nn.ipynb".
The first task is:
同じ方法でコブ・ダグラス型生産関数を学習させた様子をアニメーションにしてみた。この方法が何の意味を持つかは分からないが学習はまあまよくできていた。
mp4ファイルで添付した。
[考察]ここにはコードを書けなかったが、サンプル数が多くなった時は、ミニバッチ式の学習を試すべきだと思った。
最後に交差検定を行う。
初めに学習回数が極めて少ないNNである。
Can you write Python code for it?

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)

a = np.array([])
b = np.array([])
c = np.array([])

for x in X:
    for y in Y:
        a = np.append(a, x)
        b = np.append(b, y)

evl = np.array([])

for i in range(len(a)):
    nn = NN()
    nn.set_network()
    for j in range(1):
        l = np.random.choice([i for i in range(len(a))])
        if l != i:
            m = nn.main(1, f, [a[l], b[l]], 0.5)
    idata = [a[i], b[i]]
    est = nn.realize(f, idata)
    evl = np.append(evl, math.fabs(est - nn.supervised_data))

np.average(evl)


I am working on the file "NeuralNetwork/nn.ipynb".
The first task is:
同様のことを課題の例で使われた関数でも試してみる。
Can you write Python code for it?

x_1 = Symbol('x_1')
x_2 = Symbol('x_2')
f = (1+sin(4*math.pi*x_1))*x_2*1/2

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)

a = np.array([])
b = np.array([])
c = np.array([])

nn = NN()
nn.set_network()

for x in X:
    for y in Y:
        a = np.append(a, x)
        b = np.append(b, y)

for i in range(1000):
        l = np.random.choice([i for i in range(len(a))])
        m = nn.main(1, f, [a[l], b[l]], 0.5)

for x in X:
    for y in Y:
        idata = [x, y]
        c = np.append(c, nn.realize(f, idata))

fig = pl.figure()
ax = Axes3D(fig)

ax.scatter(a, b, c)

pl.show()


I am working on the file "NeuralNetwork/nn.ipynb".
The first task is:
本来ならば下のような形になるべきであるので上手くいっているとは言い難い。
Can you write Python code for it?

def example2(x_1, x_2):
    z = (1+np.sin(4*math.pi*x_1))*x_2*1/2
    return z

fig = pl.figure()
ax = Axes3D(fig)

X = np.arange(-1, 1, 0.1)
Y = np.arange(-1, 1, 0.1)

X, Y = np.meshgrid(X, Y)
Z = example2(X, Y)

ax.plot_surface(X, Y, Z, rstride=1, cstride=1)
ax.set_zlim(-1, 1)

pl.show()


I am working on the file "NeuralNetwork/nn.ipynb".
The first task is:
次に十分大きく(1000回に)してみる。
Can you write Python code for it?

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)

a = np.array([])
b = np.array([])
c = np.array([])

nn = NN()
nn.set_network(h=7)

for x in X:
    for y in Y:
        a = np.append(a, x)
        b = np.append(b, y)

evl = np.array([])

for i in range(len(a)):
    for j in range(10000):
        nn = NN()
        nn.set_network()
        l = np.random.choice([i for i in range(len(a))])
        if l != i:
            m = nn.main(1, f, [a[l], b[l]], 0.5)
    idata = [a[i], b[i]]
    evl = np.append(evl, math.fabs(nn.realize(f, idata) - nn.supervised_data))

evl

np.average(evl)


I am working on the file "NeuralNetwork/nn.ipynb".
The first task is:
目標と比べると大きく異なる。
サンプル数を、
Can you write Python code for it?

X = np.arange(-1, 1, 0.1)
Y = np.arange(-1, 1, 0.1)
print X, Y


I am working on the file "NeuralNetwork/nn.ipynb".
The first task is:
誤差の平均であるので小さい方よい。
学習回数を大幅に増やした結果、精度が上がった。
次に回数を100回にして、中間層の数を2と5で精度を比較する。
Can you write Python code for it?

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)

a = np.array([])
b = np.array([])
c = np.array([])

for x in X:
    for y in Y:
        a = np.append(a, x)
        b = np.append(b, y)

evl = np.array([])

for i in range(len(a)):
    for j in range(100):
        nn = NN()
        nn.set_network()
        l = np.random.choice([i for i in range(len(a))])
        if l != i:
            m = nn.main(1, f, [a[l], b[l]], 0.5)
    idata = [a[i], b[i]]
    est = nn.realize(f, idata)
    evl = np.append(evl, math.fabs(est - nn.supervised_data))

np.average(evl)

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)

a = np.array([])
b = np.array([])
c = np.array([])

for x in X:
    for y in Y:
        a = np.append(a, x)
        b = np.append(b, y)

evl = np.array([])

for i in range(len(a)):
    for j in range(100):
        nn = NN()
        nn.set_network(h=5)
        l = np.random.choice([i for i in range(len(a))])
        if l != i:
            m = nn.main(1, f, [a[l], b[l]], 0.5)
    idata = [a[i], b[i]]
    est = nn.realize(f, idata)
    evl = np.append(evl, math.fabs(est - nn.supervised_data))

np.average(evl)


I am working on the file "NeuralNetworks.ipynb".
The first task is:
Let's try to use the multi-layer perceptron classifier on the digits data set. We will use a single hidden layer to keep the training time reasonable.
Can you write Python code for it?

%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
from sklearn import datasets, cross_validation, neural_network, svm, metrics
from sklearn.neural_network import MLPClassifier

digits = datasets.load_digits()
images_and_labels = list(zip(digits.images, digits.target))

for index, (image, label) in enumerate(images_and_labels[:4]):
    plt.subplot(2, 4, index + 1)
    plt.axis('off')
    plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
    plt.title('Training: %i' % label)
plt.show()

# To apply a classifier on this data, we need to flatten the image, to
# turn the data in a (samples, feature) matrix:
n_samples = len(digits.images)
data = digits.images.reshape((n_samples, -1))

# Create a classifier: a support vector classifier
classifier = MLPClassifier(solver='lbfgs', alpha=1e-5, random_state=0, hidden_layer_sizes=(15,) )

# We learn the digits on the first half of the digits
classifier.fit(data[:n_samples / 2], digits.target[:n_samples / 2])
print("Training set score: %f" % classifier.score(data[n_samples / 2:], digits.target[n_samples / 2:]))

# Now predict the value of the digit on the second half:
expected = digits.target[n_samples / 2:]
predicted = classifier.predict(data[n_samples / 2:])

print("Classification report for classifier %s:\n%s\n"
      % (classifier, metrics.classification_report(expected, predicted)))
print("Confusion matrix:\n%s" % metrics.confusion_matrix(expected, predicted))


I am working on the file "NeuralNetworks.ipynb".
The first task is:
Our basic perceptron can do a pretty good job recognizing handwritten digits, assuming the digits are all centered in an 8x8 image. What happens if we embed the digit images at random locations within a 32x32 image? Try increasing the size of the hidden layer and see if we can improve the performance.
Can you write Python code for it?

%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
from sklearn import datasets, cross_validation, neural_network, svm, metrics
from sklearn.neural_network import MLPClassifier

digits = datasets.load_digits()
resize = 32 #Size of larger image to embed the digits
images_ex = np.zeros((digits.target.size,resize,resize))
for index, image in enumerate(digits.images):
    offrow = np.random.randint(low=0,high=resize-8,size=1)
    offcol = np.random.randint(low=0,high=resize-8,size=1)
    images_ex[index,offrow:offrow+8,offcol:offcol+8] = digits.images[index,:,:]

for jj in range(1,4):
    fig = plt.figure()
    ax1 = fig.add_subplot(1,2,2)
    ax1.imshow(images_ex[jj,:,:],aspect='auto',origin='lower',cmap=plt.cm.gray_r, interpolation='nearest')
    ax2 = fig.add_subplot(1,2,1)
    ax2.imshow(digits.images[jj,:,:],aspect='auto',origin='lower',cmap=plt.cm.gray_r, interpolation='nearest')
    plt.title(digits.target[jj])
plt.show()

# To apply a classifier on this data, we need to flatten the image, to
# turn the data in a (samples, feature) matrix:
n_samples = len(digits.images)
data_ex = images_ex.reshape((n_samples,-1))

# Create a classifier: Multi-layer perceptron
classifier = MLPClassifier(solver='lbfgs', alpha=1e-5, random_state=0, hidden_layer_sizes=(64,) )

classifier.fit(data_ex[:n_samples / 2], digits.target[:n_samples / 2])

# Now predict the value of the digit on the second half:
expected = digits.target[n_samples / 2:]
predicted = classifier.predict(data_ex[n_samples / 2:])

print("Classification report for classifier %s:\n%s\n"
      % (classifier, metrics.classification_report(expected, predicted)))
print("Confusion matrix:\n%s" % metrics.confusion_matrix(expected, predicted))