I am working on the file "CourseraDeepLearningSpecialization/2.HyperparameterTrainingRegularizationAndOptimization/Week2/Exercises/Optimization+methods.ipynb".
The first task is:
Expected Output:

     **W1**
            [[ 1.62544598 -0.61290114 -0.52907334]
 [-1.07347112  0.86450677 -2.30085497]]

     **b1**
            [[ 1.74493465]
 [-0.76027113]]

     **W2**
            [[ 0.31930698 -0.24990073  1.4627996 ]
 [-2.05974396 -0.32173003 -0.38320915]
 [ 1.13444069 -1.0998786  -0.1713109 ]]

     **b2**
            [[-0.87809283]
 [ 0.04055394]
 [ 0.58207317]]

     **v["dW1"]**
            [[-0.11006192  0.11447237  0.09015907]
 [ 0.05024943  0.09008559 -0.06837279]]

     **v["db1"]**
            [[-0.01228902]
 [-0.09357694]]

     **v["dW2"]**
            [[-0.02678881  0.05303555 -0.06916608]
 [-0.03967535 -0.06871727 -0.08452056]
 [-0.06712461 -0.00126646 -0.11173103]]

     **v["db2"]**
            [[ 0.02344157]
 [ 0.16598022]
 [ 0.07420442]]

Note that:
- The velocity is initialized with zeros. So the algorithm will take a few iterations to "build up" velocity and start to take bigger steps.
- If $\beta = 0$, then this just becomes standard gradient descent without momentum.
How do you choose $\beta$?

The larger the momentum $\beta$ is, the smoother the update because the more we take the past gradients into account. But if $\beta$ is too big, it could also smooth out the updates too much.
Common values for $\beta$ range from 0.8 to 0.999. If you don't feel inclined to tune this, $\beta = 0.9$ is often a reasonable default.
Tuning the optimal $\beta$ for your model might need trying several values to see what works best in term of reducing the value of the cost function $J$.

What you should remember:
- Momentum takes past gradients into account to smooth out the steps...
Can you write Python code for it?

# GRADED FUNCTION: initialize_adam

def initialize_adam(parameters) :
    """
    Initializes v and s as two python dictionaries with:
                - keys: "dW1", "db1", ..., "dWL", "dbL"
                - values: numpy arrays of zeros of the same shape as the corresponding gradients/parameters.

    Arguments:
    parameters -- python dictionary containing your parameters.
                    parameters["W" + str(l)] = Wl
                    parameters["b" + str(l)] = bl

    Returns:
    v -- python dictionary that will contain the exponentially weighted average of the gradient.
                    v["dW" + str(l)] = ...
                    v["db" + str(l)] = ...
    s -- python dictionary that will contain the exponentially weighted average of the squared gradient.
                    s["dW" + str(l)] = ...
                    s["db" + str(l)] = ...

    """

    L = len(parameters) // 2 # number of layers in the neural networks
    v = {}
    s = {}

    # Initialize v, s. Input: "parameters". Outputs: "v, s".
    for l in range(L):
    ### START CODE HERE ### (approx. 4 lines)
        v["dW" + str(l+1)] = None
        v["db" + str(l+1)] = None
        s["dW" + str(l+1)] = None
        s["db" + str(l+1)] = None
    ### END CODE HERE ###

    return v, s

parameters = initialize_adam_test_case()

v, s = initialize_adam(parameters)
print("v[\"dW1\"] = " + str(v["dW1"]))
print("v[\"db1\"] = " + str(v["db1"]))
print("v[\"dW2\"] = " + str(v["dW2"]))
print("v[\"db2\"] = " + str(v["db2"]))
print("s[\"dW1\"] = " + str(s["dW1"]))
print("s[\"db1\"] = " + str(s["db1"]))
print("s[\"dW2\"] = " + str(s["dW2"]))
print("s[\"db2\"] = " + str(s["db2"]))


I am working on the file "CourseraDeepLearningSpecialization/2.HyperparameterTrainingRegularizationAndOptimization/Week2/Exercises/Optimization+methods.ipynb".
The first task is:
Expected Output:

     **W1**
            [[ 1.63535156 -0.62320365 -0.53718766]
 [-1.07799357  0.85639907 -2.29470142]]

     **b1**
            [[ 1.74604067]
 [-0.75184921]]

     **W2**
            [[ 0.32171798 -0.25467393  1.46902454]
 [-2.05617317 -0.31554548 -0.3756023 ]
 [ 1.1404819  -1.09976462 -0.1612551 ]]

     **b2**
            [[-0.88020257]
 [ 0.02561572]
 [ 0.57539477]]

A variant of this is Stochastic Gradient Descent (SGD), which is equivalent to mini-batch gradient descent where each mini-batch has just 1 example. The update rule that you have just implemented does not change. What changes is that you would be computing gradients on just one training example at a time, rather than on the whole training set. The code examples below illustrate the difference between stochastic gradient descent and (batch) gradient descent.

(Batch) Gradient Descent:

``` python
X = data_input
Y = labels
parameters = initialize_parameters(layers_dims)
for i in range(0, num_iterations):
    # Forward propagation
    a, caches = forward_propagation(X, parameters)
    # Compute cost.
    cost = compute_cost(a, Y)
    # Backward propagation.
    grads = backward_propagation(a, caches, parameters)
    # Update parameters.
    parameters = update_parameters(parameters, grads)
```

Stochastic Gradient Descent:

python
X = data_input
Y = labels
parameters = initialize_parameters(layers_dims)
for i in range(0, num_iterations):
    for j in range(0, m):
        # Forward propagation
        a, caches = forward_propagation(X[:,j], parameters)
        # Compute cost
        cost = compute_cost(a, Y[:,j])
        # Backward propagation
        grads = backward_propagation(a, caches, parameters)
        # Update parameters.
        parameters = update_parameters(parameters, grads)
In Stochastic Gradient Descent, you use only 1...
Can you write Python code for it?

# GRADED FUNCTION: random_mini_batches

def random_mini_batches(X, Y, mini_batch_size = 64, seed = 0):
    """
    Creates a list of random minibatches from (X, Y)

    Arguments:
    X -- input data, of shape (input size, number of examples)
    Y -- true "label" vector (1 for blue dot / 0 for red dot), of shape (1, number of examples)
    mini_batch_size -- size of the mini-batches, integer

    Returns:
    mini_batches -- list of synchronous (mini_batch_X, mini_batch_Y)
    """

    np.random.seed(seed)            # To make your "random" minibatches the same as ours
    m = X.shape[1]                  # number of training examples
    mini_batches = []

    # Step 1: Shuffle (X, Y)
    permutation = list(np.random.permutation(m))
    shuffled_X = X[:, permutation]
    shuffled_Y = Y[:, permutation].reshape((1,m))

    # Step 2: Partition (shuffled_X, shuffled_Y). Minus the end case.
    num_complete_minibatches = math.floor(m/mini_batch_size) # number of mini batches of size mini_batch_size in your partitionning
    for k in range(0, num_complete_minibatches):
        ### START CODE HERE ### (approx. 2 lines)
        mini_batch_X = None
        mini_batch_Y = None
        ### END CODE HERE ###
        mini_batch = (mini_batch_X, mini_batch_Y)
        mini_batches.append(mini_batch)

    # Handling the end case (last mini-batch < mini_batch_size)
    if m % mini_batch_size != 0:
        ### START CODE HERE ### (approx. 2 lines)
        mini_batch_X = None
        mini_batch_Y = None
        ### END CODE HERE ###
        mini_batch = (mini_batch_X, mini_batch_Y)
        mini_batches.append(mini_batch)

    return mini_batches

X_assess, Y_assess, mini_batch_size = random_mini_batches_test_case()
mini_batches = random_mini_batches(X_assess, Y_assess, mini_batch_size)

print ("shape of the 1st mini_batch_X: " + str(mini_batches[0][0].shape))
print ("shape of the 2nd mini_batch_X: " + str(mini_batches[1][0].shape))
print ("shape of the 3rd mini_batch_X: " +...


I am working on the file "CourseraDeepLearningSpecialization/2.HyperparameterTrainingRegularizationAndOptimization/Week2/Exercises/Optimization+methods.ipynb".
The first task is:
We have already implemented a 3-layer neural network. You will train it with:
- Mini-batch Gradient Descent: it will call your function:
    - update_parameters_with_gd()
- Mini-batch Momentum: it will call your functions:
    - initialize_velocity() and update_parameters_with_momentum()
- Mini-batch Adam: it will call your functions:
    - initialize_adam() and update_parameters_with_adam()
Can you write Python code for it?

def model(X, Y, layers_dims, optimizer, learning_rate = 0.0007, mini_batch_size = 64, beta = 0.9,
          beta1 = 0.9, beta2 = 0.999,  epsilon = 1e-8, num_epochs = 10000, print_cost = True):
    """
    3-layer neural network model which can be run in different optimizer modes.

    Arguments:
    X -- input data, of shape (2, number of examples)
    Y -- true "label" vector (1 for blue dot / 0 for red dot), of shape (1, number of examples)
    layers_dims -- python list, containing the size of each layer
    learning_rate -- the learning rate, scalar.
    mini_batch_size -- the size of a mini batch
    beta -- Momentum hyperparameter
    beta1 -- Exponential decay hyperparameter for the past gradients estimates
    beta2 -- Exponential decay hyperparameter for the past squared gradients estimates
    epsilon -- hyperparameter preventing division by zero in Adam updates
    num_epochs -- number of epochs
    print_cost -- True to print the cost every 1000 epochs

    Returns:
    parameters -- python dictionary containing your updated parameters
    """

    L = len(layers_dims)             # number of layers in the neural networks
    costs = []                       # to keep track of the cost
    t = 0                            # initializing the counter required for Adam update
    seed = 10                        # For grading purposes, so that your "random" minibatches are the same as ours

    # Initialize parameters
    parameters = initialize_parameters(layers_dims)

    # Initialize the optimizer
    if optimizer == "gd":
        pass # no initialization required for gradient descent
    elif optimizer == "momentum":
        v = initialize_velocity(parameters)
    elif optimizer == "adam":
        v, s = initialize_adam(parameters)

    # Optimization loop
    for i in range(num_epochs):

        # Define the random minibatches. We increment the seed to reshuffle differently the dataset after each epoch
        seed = seed + 1
        minibatches = random_mini_batches(X, Y,...


I am working on the file "CourseraDeepLearningSpecialization/2.HyperparameterTrainingRegularizationAndOptimization/Week3/Exercises/Tensorflow+Tutorial.ipynb".
The first task is:
As usual you flatten the image dataset, then normalize it by dividing by 255. On top of that, you will convert each label to a one-hot vector as shown in Figure 1. Run the cell below to do so.
Can you write Python code for it?

# Flatten the training and test images
X_train_flatten = X_train_orig.reshape(X_train_orig.shape[0], -1).T
X_test_flatten = X_test_orig.reshape(X_test_orig.shape[0], -1).T
# Normalize image vectors
X_train = X_train_flatten/255.
X_test = X_test_flatten/255.
# Convert training and test labels to one hot matrices
Y_train = convert_to_one_hot(Y_train_orig, 6)
Y_test = convert_to_one_hot(Y_test_orig, 6)

print ("number of training examples = " + str(X_train.shape[1]))
print ("number of test examples = " + str(X_test.shape[1]))
print ("X_train shape: " + str(X_train.shape))
print ("Y_train shape: " + str(Y_train.shape))
print ("X_test shape: " + str(X_test.shape))
print ("Y_test shape: " + str(Y_test.shape))


I am working on the file "CourseraDeepLearningSpecialization/2.HyperparameterTrainingRegularizationAndOptimization/Week3/Exercises/Tensorflow+Tutorial.ipynb".
The first task is:
Change the index below and run the cell to visualize some examples in the dataset.
Can you write Python code for it?

# Example of a picture
index = 0
plt.imshow(X_train_orig[index])
print ("y = " + str(np.squeeze(Y_train_orig[:, index])))


I am working on the file "CourseraDeepLearningSpecialization/2.HyperparameterTrainingRegularizationAndOptimization/Week3/Exercises/Tensorflow+Tutorial.ipynb".
The first task is:
Expected Output :

**result**

[[-2.15657382]
 [ 2.95891446]
 [-1.08926781]
 [-0.84538042]]

1.2 - Computing the sigmoid
Great! You just implemented a linear function. Tensorflow offers a variety of commonly used neural network functions like tf.sigmoid and tf.softmax. For this exercise lets compute the sigmoid function of an input.
You will do this exercise using a placeholder variable x. When running the session, you should use the feed dictionary to pass in the input z. In this exercise, you will have to (i) create a placeholder x, (ii) define the operations needed to compute the sigmoid using tf.sigmoid, and then (iii) run the session.
 Exercise : Implement the sigmoid function below. You should use the following:

tf.placeholder(tf.float32, name = "...")
tf.sigmoid(...)
sess.run(..., feed_dict = {x: z})

Note that there are two typical ways to create and use sessions in tensorflow:
Method 1:
```python
sess = tf.Session()
Run the variables initialization (if needed), run the operations
result = sess.run(..., feed_dict = {...})
sess.close() # Close the session
**Method 2:**python
with tf.Session() as sess:
    # run the variables initialization (if needed), run the operations
    result = sess.run(..., feed_dict = {...})
    # This takes care of closing the session for you :)
```
Can you write Python code for it?

# GRADED FUNCTION: sigmoid

def sigmoid(z):
    """
    Computes the sigmoid of z

    Arguments:
    z -- input value, scalar or vector

    Returns:
    results -- the sigmoid of z
    """

    ### START CODE HERE ### ( approx. 4 lines of code)
    # Create a placeholder for x. Name it 'x'.
    x = None

    # compute sigmoid(x)
    sigmoid = None

    # Create a session, and run it. Please use the method 2 explained above.
    # You should use a feed_dict to pass z's value to x.
    None
        # Run session and call the output "result"
        result = None

    ### END CODE HERE ###

    return result

print ("sigmoid(0) = " + str(sigmoid(0)))
print ("sigmoid(12) = " + str(sigmoid(12)))


I am working on the file "CourseraDeepLearningSpecialization/2.HyperparameterTrainingRegularizationAndOptimization/Week3/Exercises/Tensorflow+Tutorial.ipynb".
The first task is:
Expected Output:

            **Train Accuracy**

        0.999074

            **Test Accuracy**

        0.716667

Amazing, your algorithm can recognize a sign representing a figure between 0 and 5 with 71.7% accuracy.
Insights:
- Your model seems big enough to fit the training set well. However, given the difference between train and test accuracy, you could try to add L2 or dropout regularization to reduce overfitting.
- Think about the session as a block of code to train the model. Each time you run the session on a minibatch, it trains the parameters. In total you have run the session a large number of times (1500 epochs) until you obtained well trained parameters.
2.7 - Test with your own image (optional / ungraded exercise)
Congratulations on finishing this assignment. You can now take a picture of your hand and see the output of your model. To do that:
    1. Click on "File" in the upper bar of this notebook, then click "Open" to go on your Coursera Hub.
    2. Add your image to this Jupyter Notebook's directory, in the "images" folder
    3. Write your image's name in the following code
    4. Run the code and check if the algorithm is right!
Can you write Python code for it?

import scipy
from PIL import Image
from scipy import ndimage

## START CODE HERE ## (PUT YOUR IMAGE NAME)
my_image = "thumbs_up.jpg"
## END CODE HERE ##

# We preprocess your image to fit your algorithm.
fname = "images/" + my_image
image = np.array(ndimage.imread(fname, flatten=False))
my_image = scipy.misc.imresize(image, size=(64,64)).reshape((1, 64*64*3)).T
my_image_prediction = predict(my_image, parameters)

plt.imshow(image)
print("Your algorithm predicts: y = " + str(np.squeeze(my_image_prediction)))


I am working on the file "CourseraDeepLearningSpecialization/2.HyperparameterTrainingRegularizationAndOptimization/Week3/Exercises/Tensorflow+Tutorial.ipynb".
The first task is:
Expected Output:

            **Z3**

        Tensor("Add_2:0", shape=(6, ?), dtype=float32)

You may have noticed that the forward propagation doesn't output any cache. You will understand why below, when we get to brackpropagation.
2.4 Compute cost
As seen before, it is very easy to compute the cost using:
python
tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits = ..., labels = ...))
Question: Implement the cost function below.
- It is important to know that the "logits" and "labels" inputs of tf.nn.softmax_cross_entropy_with_logits are expected to be of shape (number of examples, num_classes). We have thus transposed Z3 and Y for you.
- Besides, tf.reduce_mean basically does the summation over the examples.
Can you write Python code for it?

# GRADED FUNCTION: compute_cost

def compute_cost(Z3, Y):
    """
    Computes the cost

    Arguments:
    Z3 -- output of forward propagation (output of the last LINEAR unit), of shape (6, number of examples)
    Y -- "true" labels vector placeholder, same shape as Z3

    Returns:
    cost - Tensor of the cost function
    """

    # to fit the tensorflow requirement for tf.nn.softmax_cross_entropy_with_logits(...,...)
    logits = tf.transpose(Z3)
    labels = tf.transpose(Y)

    ### START CODE HERE ### (1 line of code)
    cost = None
    ### END CODE HERE ###

    return cost

tf.reset_default_graph()

with tf.Session() as sess:
    X, Y = create_placeholders(12288, 6)
    parameters = initialize_parameters()
    Z3 = forward_propagation(X, parameters)
    cost = compute_cost(Z3, Y)
    print("cost = " + str(cost))


I am working on the file "CourseraDeepLearningSpecialization/2.HyperparameterTrainingRegularizationAndOptimization/Week3/Exercises/Tensorflow+Tutorial.ipynb".
The first task is:
Expected Output:

            **one_hot**

        [[ 0.  0.  0.  1.  0.  0.]
 [ 1.  0.  0.  0.  0.  1.]
 [ 0.  1.  0.  0.  1.  0.]
 [ 0.  0.  1.  0.  0.  0.]]

1.5 - Initialize with zeros and ones
Now you will learn how to initialize a vector of zeros and ones. The function you will be calling is tf.ones(). To initialize with zeros you could use tf.zeros() instead. These functions take in a shape and return an array of dimension shape full of zeros and ones respectively.
Exercise: Implement the function below to take in a shape and to return an array (of the shape's dimension of ones).

tf.ones(shape)
Can you write Python code for it?

# GRADED FUNCTION: ones

def ones(shape):
    """
    Creates an array of ones of dimension shape

    Arguments:
    shape -- shape of the array you want to create

    Returns:
    ones -- array containing only ones
    """

    ### START CODE HERE ###

    # Create "ones" tensor using tf.ones(...). (approx. 1 line)
    ones = None

    # Create the session (approx. 1 line)
    sess = None

    # Run the session to compute 'ones' (approx. 1 line)
    ones = None

    # Close the session (approx. 1 line). See method 1 above.
    None

    ### END CODE HERE ###
    return ones

print ("ones = " + str(ones([3])))


I am working on the file "CourseraDeepLearningSpecialization/2.HyperparameterTrainingRegularizationAndOptimization/Week3/Exercises/Tensorflow+Tutorial.ipynb".
The first task is:
Note that 12288 comes from $64 \times 64 \times 3$. Each image is square, 64 by 64 pixels, and 3 is for the RGB colors. Please make sure all these shapes make sense to you before continuing.
Your goal is to build an algorithm capable of recognizing a sign with high accuracy. To do so, you are going to build a tensorflow model that is almost the same as one you have previously built in numpy for cat recognition (but now using a softmax output). It is a great occasion to compare your numpy implementation to the tensorflow one.
The model is LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SOFTMAX. The SIGMOID output layer has been converted to a SOFTMAX. A SOFTMAX layer generalizes SIGMOID to when there are more than two classes.
2.1 - Create placeholders
Your first task is to create placeholders for X and Y. This will allow you to later pass your training data in when you run your session.
Exercise: Implement the function below to create the placeholders in tensorflow.
Can you write Python code for it?

# GRADED FUNCTION: create_placeholders

def create_placeholders(n_x, n_y):
    """
    Creates the placeholders for the tensorflow session.

    Arguments:
    n_x -- scalar, size of an image vector (num_px * num_px = 64 * 64 * 3 = 12288)
    n_y -- scalar, number of classes (from 0 to 5, so -> 6)

    Returns:
    X -- placeholder for the data input, of shape [n_x, None] and dtype "float"
    Y -- placeholder for the input labels, of shape [n_y, None] and dtype "float"

    Tips:
    - You will use None because it let's us be flexible on the number of examples you will for the placeholders.
      In fact, the number of examples during test/train is different.
    """

    ### START CODE HERE ### (approx. 2 lines)
    X = None
    Y = None
    ### END CODE HERE ###

    return X, Y

X, Y = create_placeholders(12288, 6)
print ("X = " + str(X))
print ("Y = " + str(Y))


I am working on the file "CourseraDeepLearningSpecialization/2.HyperparameterTrainingRegularizationAndOptimization/Week3/Exercises/Tensorflow+Tutorial.ipynb".
The first task is:
Now that you have imported the library, we will walk you through its different applications. You will start with an example, where we compute for you the loss of one training example.
$$loss = \mathcal{L}(\hat{y}, y) = (\hat y^{(i)} - y^{(i)})^2 \tag{1}$$
Can you write Python code for it?

y_hat = tf.constant(36, name='y_hat')            # Define y_hat constant. Set to 36.
y = tf.constant(39, name='y')                    # Define y. Set to 39

loss = tf.Variable((y - y_hat)**2, name='loss')  # Create a variable for the loss

init = tf.global_variables_initializer()         # When init is run later (session.run(init)),
                                                 # the loss variable will be initialized and ready to be computed
with tf.Session() as session:                    # Create a session and print the output
    session.run(init)                            # Initializes the variables
    print(session.run(loss))                     # Prints the loss


I am working on the file "CourseraDeepLearningSpecialization/2.HyperparameterTrainingRegularizationAndOptimization/Week3/Exercises/Tensorflow+Tutorial.ipynb".
The first task is:
Run the following cell to train your model! On our machine it takes about 5 minutes. Your "Cost after epoch 100" should be 1.016458. If it's not, don't waste time; interrupt the training by clicking on the square (⬛) in the upper bar of the notebook, and try to correct your code. If it is the correct cost, take a break and come back in 5 minutes!
Can you write Python code for it?

parameters = model(X_train, Y_train, X_test, Y_test)


I am working on the file "CourseraDeepLearningSpecialization/2.HyperparameterTrainingRegularizationAndOptimization/Week3/Exercises/Tensorflow+Tutorial.ipynb".
The first task is:
Writing and running programs in TensorFlow has the following steps:

Create Tensors (variables) that are not yet executed/evaluated.
Write operations between those Tensors.
Initialize your Tensors.
Create a Session.
Run the Session. This will run the operations you'd written above.

Therefore, when we created a variable for the loss, we simply defined the loss as a function of other quantities, but did not evaluate its value. To evaluate it, we had to run init=tf.global_variables_initializer(). That initialized the loss variable, and in the last line we were finally able to evaluate the value of loss and print its value.
Now let us look at an easy example. Run the cell below:
Can you write Python code for it?

a = tf.constant(2)
b = tf.constant(10)
c = tf.multiply(a,b)
print(c)


I am working on the file "Coursera_DL/Building+your+Deep+Neural+Network+-+Step+by+Step+v5.ipynb".
The first task is:
2 - Outline of the Assignment
To build your neural network, you will be implementing several "helper functions". These helper functions will be used in the next assignment to build a two-layer neural network and an L-layer neural network. Each small helper function you will implement will have detailed instructions that will walk you through the necessary steps. Here is an outline of this assignment, you will:

Initialize the parameters for a two-layer network and for an $L$-layer neural network.
Implement the forward propagation module (shown in purple in the figure below).
Complete the LINEAR part of a layer's forward propagation step (resulting in $Z^{[l]}$).
We give you the ACTIVATION function (relu/sigmoid).
Combine the previous two steps into a new [LINEAR->ACTIVATION] forward function.
Stack the [LINEAR->RELU] forward function L-1 time (for layers 1 through L-1) and add a [LINEAR->SIGMOID] at the end (for the final layer $L$). This gives you a new L_model_forward function.

Compute the loss.
Implement the backward propagation module (denoted in red in the figure below).
Complete the LINEAR part of a layer's backward propagation step.
We give you the gradient of the ACTIVATE function (relu_backward/sigmoid_backward)
Combine the previous two steps into a new [LINEAR->ACTIVATION] backward function.
Stack [LINEAR->RELU] backward L-1 times and add [LINEAR->SIGMOID] backward in a new L_model_backward function

Finally update the parameters.

 Figure 1

Note that for every forward function, there is a corresponding backward function. That is why at every step of your forward module you will be storing some values in a cache. The cached values are useful for computing gradients. In the backpropagation module you will then use the cache to calculate the gradients. This assignment will show you exactly how to carry out each of these steps.
3 - Initialization
You will write two helper functions that...
Can you write Python code for it?

# GRADED FUNCTION: initialize_parameters

def initialize_parameters(n_x, n_h, n_y):
    """
    Argument:
    n_x -- size of the input layer
    n_h -- size of the hidden layer
    n_y -- size of the output layer

    Returns:
    parameters -- python dictionary containing your parameters:
                    W1 -- weight matrix of shape (n_h, n_x)
                    b1 -- bias vector of shape (n_h, 1)
                    W2 -- weight matrix of shape (n_y, n_h)
                    b2 -- bias vector of shape (n_y, 1)
    """

    np.random.seed(1)

    ### START CODE HERE ### (≈ 4 lines of code)
    W1 = np.random.randn(n_h, n_x) * 0.01
    b1 = np.zeros(shape=(n_h, 1))
    W2 = np.random.randn(n_y, n_h) * 0.01
    b2 = np.zeros(shape=(n_y, 1))
    ### END CODE HERE ###

    assert(W1.shape == (n_h, n_x))
    assert(b1.shape == (n_h, 1))
    assert(W2.shape == (n_y, n_h))
    assert(b2.shape == (n_y, 1))

    parameters = {"W1": W1,
                  "b1": b1,
                  "W2": W2,
                  "b2": b2}

    return parameters

parameters = initialize_parameters(3,2,1)
print("W1 = " + str(parameters["W1"]))
print("b1 = " + str(parameters["b1"]))
print("W2 = " + str(parameters["W2"]))
print("b2 = " + str(parameters["b2"]))


I am working on the file "Coursera_DL/Building+your+Deep+Neural+Network+-+Step+by+Step+v5.ipynb".
The first task is:
Expected Output:

    **cost**
     0.41493159961539694

6 - Backward propagation module
Just like with forward propagation, you will implement helper functions for backpropagation. Remember that back propagation is used to calculate the gradient of the loss function with respect to the parameters.
Reminder:

 Figure 3 : Forward and Backward propagation for LINEAR->RELU->LINEAR->SIGMOID
 The purple blocks represent the forward propagation, and the red blocks represent the backward propagation.

Now, similar to forward propagation, you are going to build the backward propagation in three steps:
- LINEAR backward
- LINEAR -> ACTIVATION backward where ACTIVATION computes the derivative of either the ReLU or sigmoid activation
- [LINEAR -> RELU] $\times$ (L-1) -> LINEAR -> SIGMOID backward (whole...
Can you write Python code for it?

# GRADED FUNCTION: linear_backward

def linear_backward(dZ, cache):
    """
    Implement the linear portion of backward propagation for a single layer (layer l)

    Arguments:
    dZ -- Gradient of the cost with respect to the linear output (of current layer l)
    cache -- tuple of values (A_prev, W, b) coming from the forward propagation in the current layer

    Returns:
    dA_prev -- Gradient of the cost with respect to the activation (of the previous layer l-1), same shape as A_prev
    dW -- Gradient of the cost with respect to W (current layer l), same shape as W
    db -- Gradient of the cost with respect to b (current layer l), same shape as b
    """
    A_prev, W, b = cache
    m = A_prev.shape[1]

    ### START CODE HERE ### (≈ 3 lines of code)
    dW = 1 / m * np.dot(dZ, A_prev.T)
    db = 1 / m * np.sum(dZ, axis=1, keepdims=True)
    dA_prev = np.dot(W.T, dZ)
    ### END CODE HERE ###

    assert (dA_prev.shape == A_prev.shape)
    assert (dW.shape == W.shape)
    assert (db.shape == b.shape)

    return dA_prev, dW, db

# Set up some test inputs
dZ, linear_cache = linear_backward_test_case()

dA_prev, dW, db = linear_backward(dZ, linear_cache)
print ("dA_prev = "+ str(dA_prev))
print ("dW = " + str(dW))
print ("db = " + str(db))


I am working on the file "Coursera_DL/Building+your+Deep+Neural+Network+-+Step+by+Step+v5.ipynb".
The first task is:
Expected Output:

     **dA_prev**
     [[ 0.51822968 -0.19517421]
 [-0.40506361  0.15255393]
 [ 2.37496825 -0.89445391]]

         **dW**
         [[-0.10076895  1.40685096  1.64992505]]

         **db**
         [[ 0.50629448]]

6.2 - Linear-Activation backward
Next, you will create a function that merges the two helper functions: linear_backward and the backward step for the activation linear_activation_backward.
To help you implement linear_activation_backward, we provided two backward functions:
- sigmoid_backward: Implements the backward propagation for SIGMOID unit. You can call it as follows:
python
dZ = sigmoid_backward(dA, activation_cache)

relu_backward: Implements the backward propagation for RELU unit. You can call it as follows:

python
dZ = relu_backward(dA, activation_cache)
If $g(.)$ is the activation function,
sigmoid_backward and relu_backward compute $$dZ^{[l]} = dA^{[l]} * g'(Z^{[l]}) \tag{11}$$.
Exercise: Implement the backpropagation for the LINEAR->ACTIVATION layer.
Can you write Python code for it?

# GRADED FUNCTION: linear_activation_backward

def linear_activation_backward(dA, cache, activation):
    """
    Implement the backward propagation for the LINEAR->ACTIVATION layer.

    Arguments:
    dA -- post-activation gradient for current layer l
    cache -- tuple of values (linear_cache, activation_cache) we store for computing backward propagation efficiently
    activation -- the activation to be used in this layer, stored as a text string: "sigmoid" or "relu"

    Returns:
    dA_prev -- Gradient of the cost with respect to the activation (of the previous layer l-1), same shape as A_prev
    dW -- Gradient of the cost with respect to W (current layer l), same shape as W
    db -- Gradient of the cost with respect to b (current layer l), same shape as b
    """
    linear_cache, activation_cache = cache

    if activation == "relu":
        ### START CODE HERE ### (≈ 2 lines of code)
        dZ = relu_backward(dA, activation_cache)
        ### END CODE HERE ###

    elif activation == "sigmoid":
        ### START CODE HERE ### (≈ 2 lines of code)
        dZ = sigmoid_backward(dA, activation_cache)
        ### END CODE HERE ###

    # Shorten the code
    dA_prev, dW, db = linear_backward(dZ, linear_cache)

    return dA_prev, dW, db

AL, linear_activation_cache = linear_activation_backward_test_case()

dA_prev, dW, db = linear_activation_backward(AL, linear_activation_cache, activation = "sigmoid")
print ("sigmoid:")
print ("dA_prev = "+ str(dA_prev))
print ("dW = " + str(dW))
print ("db = " + str(db) + "\n")

dA_prev, dW, db = linear_activation_backward(AL, linear_activation_cache, activation = "relu")
print ("relu:")
print ("dA_prev = "+ str(dA_prev))
print ("dW = " + str(dW))
print ("db = " + str(db))


I am working on the file "Coursera_DL/Building+your+Deep+Neural+Network+-+Step+by+Step+v5.ipynb".
The first task is:
Expected output with sigmoid:

     dA_prev
           [[ 0.11017994  0.01105339]
 [ 0.09466817  0.00949723]
 [-0.05743092 -0.00576154]]

     dW
            [[ 0.10266786  0.09778551 -0.01968084]]

     db
            [[-0.05729622]]

Expected output with relu:

     dA_prev
            [[ 0.44090989  0.        ]
 [ 0.37883606  0.        ]
 [-0.2298228   0.        ]]

     dW
            [[ 0.44513824  0.37371418 -0.10478989]]

     db
            [[-0.20837892]]

6.3 - L-Model Backward
Now you will implement the backward function for the whole network. Recall that when you implemented the L_model_forward function, at each iteration, you stored a cache which contains (X,W,b, and z). In the back propagation module, you will use those variables to compute the gradients. Therefore, in the L_model_backward function, you will iterate through all the hidden layers backward, starting from layer $L$. On each step, you will use the cached values for layer $l$ to backpropagate through layer $l$. Figure 5 below shows the backward pass.

  Figure 5 : Backward pass

 Initializing backpropagation:
To backpropagate through this network, we know that the output is,
$A^{[L]} = \sigma(Z^{[L]})$. Your code thus needs to compute dAL $= \frac{\partial \mathcal{L}}{\partial A^{[L]}}$.
To do so, use this formula (derived using calculus which you don't need in-depth knowledge of):
python
dAL = - (np.divide(Y, AL) - np.divide(1 - Y, 1 - AL)) # derivative of cost with respect to AL
You can then use this post-activation gradient dAL to keep going backward. As seen in Figure 5, you can now feed in dAL into the...
Can you write Python code for it?

# GRADED FUNCTION: L_model_backward

def L_model_backward(AL, Y, caches):
    """
    Implement the backward propagation for the [LINEAR->RELU] * (L-1) -> LINEAR -> SIGMOID group

    Arguments:
    AL -- probability vector, output of the forward propagation (L_model_forward())
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat)
    caches -- list of caches containing:
                every cache of linear_activation_forward() with "relu" (it's caches[l], for l in range(L-1) i.e l = 0...L-2)
                the cache of linear_activation_forward() with "sigmoid" (it's caches[L-1])

    Returns:
    grads -- A dictionary with the gradients
             grads["dA" + str(l)] = ...
             grads["dW" + str(l)] = ...
             grads["db" + str(l)] = ...
    """
    grads = {}
    L = len(caches) # the number of layers
    m = AL.shape[1]
    Y = Y.reshape(AL.shape) # after this line, Y is the same shape as AL

    # Initializing the backpropagation
    ### START CODE HERE ### (1 line of code)
    dAL = - (np.divide(Y, AL) - np.divide(1 - Y, 1 - AL))
    ### END CODE HERE ###

    # Lth layer (SIGMOID -> LINEAR) gradients. Inputs: "AL, Y, caches". Outputs: "grads["dAL"], grads["dWL"], grads["dbL"]
    ### START CODE HERE ### (approx. 2 lines)
    current_cache = caches[-1]
    grads["dA" + str(L)], grads["dW" + str(L)], grads["db" + str(L)] = linear_activation_backward(dAL,
                                                                                current_cache,
                                                                                "sigmoid")
    ### END CODE HERE ###

    for l in reversed(range(L-1)):
        # lth layer: (RELU -> LINEAR) gradients.
        # Inputs: "grads["dA" + str(l + 2)], caches". Outputs: "grads["dA" + str(l + 1)] , grads["dW" + str(l + 1)] , grads["db" + str(l + 1)]
        ### START CODE HERE ### (approx. 5 lines)
        current_cache = caches[l]
        dA_prev_temp, dW_temp, db_temp =...


I am working on the file "Coursera_DL/Building+your+Deep+Neural+Network+-+Step+by+Step+v5.ipynb".
The first task is:
Expected output:

     **With sigmoid: A **
     [[ 0.96890023  0.11013289]]

     **With ReLU: A **
     [[ 3.43896131  0.        ]]

Note: In deep learning, the "[LINEAR->ACTIVATION]" computation is counted as a single layer in the neural network, not two layers.
d) L-Layer Model
For even more convenience when implementing the $L$-layer Neural Net, you will need a function that replicates the previous one (linear_activation_forward with RELU) $L-1$ times, then follows that with one linear_activation_forward with SIGMOID.

 Figure 2 : [LINEAR -> RELU] $\times$ (L-1) -> LINEAR -> SIGMOID model

Exercise: Implement the forward propagation of the above model.
Instruction: In the code below, the variable AL will denote $A^{[L]} = \sigma(Z^{[L]}) = \sigma(W^{[L]} A^{[L-1]} + b^{[L]})$. (This is sometimes also called Yhat, i.e., this is $\hat{Y}$.)
Tips:
- Use the functions you had previously written
- Use a for loop to replicate [LINEAR->RELU] (L-1) times
- Don't forget to keep track of the caches in the "caches" list. To add a new value c to a list, you can use list.append(c).
Can you write Python code for it?

# GRADED FUNCTION: L_model_forward

def L_model_forward(X, parameters):
    """
    Implement forward propagation for the [LINEAR->RELU]*(L-1)->LINEAR->SIGMOID computation

    Arguments:
    X -- data, numpy array of shape (input size, number of examples)
    parameters -- output of initialize_parameters_deep()

    Returns:
    AL -- last post-activation value
    caches -- list of caches containing:
                every cache of linear_relu_forward() (there are L-1 of them, indexed from 0 to L-2)
                the cache of linear_sigmoid_forward() (there is one, indexed L-1)
    """

    caches = []
    A = X
    L = len(parameters) // 2                  # number of layers in the neural network

    # Implement [LINEAR -> RELU]*(L-1). Add "cache" to the "caches" list.
    for l in range(1, L):
        A_prev = A
        ### START CODE HERE ### (≈ 2 lines of code)
        A, cache = linear_activation_forward(A_prev,
                                             parameters['W{}'.format(l)],
                                             parameters['b{}'.format(l)],
                                             'relu')
        caches.append(cache)
        ### END CODE HERE ###

    # Implement LINEAR -> SIGMOID. Add "cache" to the "caches" list.
    ### START CODE HERE ### (≈ 2 lines of code)
    AL, cache = linear_activation_forward(A,
                                          parameters['W{}'.format(L)],
                                          parameters['b{}'.format(L)],
                                          'sigmoid')
    caches.append(cache)
    ### END CODE HERE ###

    assert(AL.shape == (1,X.shape[1]))

    return AL, caches

X, parameters = L_model_forward_test_case_2hidden()
AL, caches = L_model_forward(X, parameters)
print("AL = " + str(AL))
print("Length of caches list = " + str(len(caches)))


I am working on the file "Coursera_DL/Building+your+Deep+Neural+Network+-+Step+by+Step+v5.ipynb".
The first task is:
Expected output:

     **W1**
     [[ 0.01624345 -0.00611756 -0.00528172]
 [-0.01072969  0.00865408 -0.02301539]]

     **b1**
    [[ 0.]
 [ 0.]]

    **W2**
     [[ 0.01744812 -0.00761207]]

     **b2**
     [[ 0.]]

3.2 - L-layer Neural Network
The initialization for a deeper L-layer neural network is more complicated because there are many more weight matrices and bias vectors. When completing the initialize_parameters_deep, you should make sure that your dimensions match between each layer. Recall that $n^{[l]}$ is the number of units in layer $l$. Thus for example if the size of our input $X$ is $(12288, 209)$ (with $m=209$ examples) then:

         **Shape of W**
         **Shape of b**
         **Activation**
         **Shape of Activation**

         **Layer 1**
         $(n^{[1]},12288)$
         $(n^{[1]},1)$
         $Z^{[1]} = W^{[1]}  X + b^{[1]} $

         $(n^{[1]},209)$

         **Layer 2**
         $(n^{[2]}, n^{[1]})$
         $(n^{[2]},1)$
        $Z^{[2]} = W^{[2]} A^{[1]} + b^{[2]}$
         $(n^{[2]}, 209)$

         $\vdots$
         $\vdots$
         $\vdots$
         $\vdots$
         $\vdots$

         **Layer L-1**
         $(n^{[L-1]}, n^{[L-2]})$
         $(n^{[L-1]}, 1)$
        $Z^{[L-1]} =  W^{[L-1]} A^{[L-2]} + b^{[L-1]}$
         $(n^{[L-1]}, 209)$

         **Layer L**
         $(n^{[L]}, n^{[L-1]})$
         $(n^{[L]}, 1)$
         $Z^{[L]} =  W^{[L]}...
Can you write Python code for it?

# GRADED FUNCTION: initialize_parameters_deep

def initialize_parameters_deep(layer_dims):
    """
    Arguments:
    layer_dims -- python array (list) containing the dimensions of each layer in our network

    Returns:
    parameters -- python dictionary containing your parameters "W1", "b1", ..., "WL", "bL":
                    Wl -- weight matrix of shape (layer_dims[l], layer_dims[l-1])
                    bl -- bias vector of shape (layer_dims[l], 1)
    """

    np.random.seed(3)
    parameters = {}
    L = len(layer_dims)            # number of layers in the network

    for l in range(1, L):
        ### START CODE HERE ### (≈ 2 lines of code)
        parameters['W' + str(l)] = np.random.randn(layer_dims[l], layer_dims[l - 1]) * 0.01
        parameters['b' + str(l)] = np.zeros((layer_dims[l], 1))
        ### END CODE HERE ###

        assert(parameters['W' + str(l)].shape == (layer_dims[l], layer_dims[l - 1]))
        assert(parameters['b' + str(l)].shape == (layer_dims[l], 1))

    return parameters

parameters = initialize_parameters_deep([5,4,3])
print("W1 = " + str(parameters["W1"]))
print("b1 = " + str(parameters["b1"]))
print("W2 = " + str(parameters["W2"]))
print("b2 = " + str(parameters["b2"]))


I am working on the file "Coursera_DL/Building+your+Deep+Neural+Network+-+Step+by+Step+v5.ipynb".
The first task is:
Expected output:

     **W1**
    [[ 0.01788628  0.0043651   0.00096497 -0.01863493 -0.00277388]
 [-0.00354759 -0.00082741 -0.00627001 -0.00043818 -0.00477218]
 [-0.01313865  0.00884622  0.00881318  0.01709573  0.00050034]
 [-0.00404677 -0.0054536  -0.01546477  0.00982367 -0.01101068]]

    **b1**
    [[ 0.]
 [ 0.]
 [ 0.]
 [ 0.]]

    **W2**
    [[-0.01185047 -0.0020565   0.01486148  0.00236716]
 [-0.01023785 -0.00712993  0.00625245 -0.00160513]
 [-0.00768836 -0.00230031  0.00745056  0.01976111]]

    **b2**
    [[ 0.]
 [ 0.]
 [ 0.]]

4 - Forward propagation module
4.1 - Linear Forward
Now that you have initialized your parameters, you will do the forward propagation module. You will start by implementing some basic functions that you will use later when implementing the model. You will complete three functions in this order:

LINEAR
LINEAR -> ACTIVATION where ACTIVATION will be either ReLU or Sigmoid.
[LINEAR -> RELU] $\times$ (L-1) -> LINEAR -> SIGMOID (whole model)

The linear forward module (vectorized over all the examples) computes the following equations:
$$Z^{[l]} = W^{[l]}A^{[l-1]} +b^{[l]}\tag{4}$$
where $A^{[0]} = X$.
Exercise: Build the linear part of forward propagation.
Reminder:
The mathematical representation of this unit is $Z^{[l]} = W^{[l]}A^{[l-1]} +b^{[l]}$. You may also find np.dot() useful. If your dimensions don't match, printing W.shape may help.
Can you write Python code for it?

# GRADED FUNCTION: linear_forward

def linear_forward(A, W, b):
    """
    Implement the linear part of a layer's forward propagation.

    Arguments:
    A -- activations from previous layer (or input data): (size of previous layer, number of examples)
    W -- weights matrix: numpy array of shape (size of current layer, size of previous layer)
    b -- bias vector, numpy array of shape (size of the current layer, 1)

    Returns:
    Z -- the input of the activation function, also called pre-activation parameter
    cache -- a python dictionary containing "A", "W" and "b" ; stored for computing the backward pass efficiently
    """

    ### START CODE HERE ### (≈ 1 line of code)
    Z = np.dot(W, A) + b
    ### END CODE HERE ###

    assert(Z.shape == (W.shape[0], A.shape[1]))
    cache = (A, W, b)

    return Z, cache

A, W, b = linear_forward_test_case()

Z, linear_cache = linear_forward(A, W, b)
print("Z = " + str(Z))


I am working on the file "Coursera_DL/Logistic+Regression+with+a+Neural+Network+mindset+v4.ipynb".
The first task is:
2 - Overview of the Problem set
Problem Statement: You are given a dataset ("data.h5") containing:
    - a training set of m_train images labeled as cat (y=1) or non-cat (y=0)
    - a test set of m_test images labeled as cat or non-cat
    - each image is of shape (num_px, num_px, 3) where 3 is for the 3 channels (RGB). Thus, each image is square (height = num_px) and (width = num_px).
You will build a simple image-recognition algorithm that can correctly classify pictures as cat or non-cat.
Let's get more familiar with the dataset. Load the data by running the following code.
Can you write Python code for it?

# Loading the data (cat/non-cat)
train_set_x_orig, train_set_y, test_set_x_orig, test_set_y, classes = load_dataset()


I am working on the file "Coursera_DL/Logistic+Regression+with+a+Neural+Network+mindset+v4.ipynb".
The first task is:

What you need to remember:
Common steps for pre-processing a new dataset are:
- Figure out the dimensions and shapes of the problem (m_train, m_test, num_px, ...)
- Reshape the datasets such that each example is now a vector of size (num_px * num_px * 3, 1)
- "Standardize" the data
3 - General Architecture of the learning algorithm
It's time to design a simple algorithm to distinguish cat images from non-cat images.
You will build a Logistic Regression, using a Neural Network mindset. The following Figure explains why Logistic Regression is actually a very simple Neural Network!

Mathematical expression of the algorithm:
For one example $x^{(i)}$:
$$z^{(i)} = w^T x^{(i)} + b \tag{1}$$
$$\hat{y}^{(i)} = a^{(i)} = sigmoid(z^{(i)})\tag{2}$$
$$ \mathcal{L}(a^{(i)}, y^{(i)}) =  - y^{(i)}  \log(a^{(i)}) - (1-y^{(i)} )  \log(1-a^{(i)})\tag{3}$$
The cost is then computed by summing over all training examples:
$$ J = \frac{1}{m} \sum_{i=1}^m \mathcal{L}(a^{(i)}, y^{(i)})\tag{6}$$
Key steps:
In this exercise, you will carry out the following steps:
    - Initialize the parameters of the model
    - Learn the parameters for the model by minimizing the cost
    - Use the learned parameters to make predictions (on the test set)
    - Analyse the results and conclude
4 - Building the parts of our algorithm ##
The main steps for building a Neural Network are:
1. Define the model structure (such as number of input features)
2. Initialize the model's parameters
3. Loop:
    - Calculate current loss (forward propagation)
    - Calculate current gradient (backward propagation)
    - Update parameters (gradient descent)
You often build 1-3 separately and integrate them into one function we call model().
4.1 - Helper functions
Exercise: Using your code from "Python Basics", implement sigmoid(). As you've seen in the figure above, you need to compute $sigmoid( w^T x + b) = \frac{1}{1 + e^{-(w^T x + b)}}$ to make predictions. Use np.exp().
Can you write Python code for it?

# GRADED FUNCTION: sigmoid

def sigmoid(z):
    """
    Compute the sigmoid of z

    Arguments:
    z -- A scalar or numpy array of any size.

    Return:
    s -- sigmoid(z)
    """

    ### START CODE HERE ### (≈ 1 line of code)
    s = 1 / (1 + np.exp(-z))
    ### END CODE HERE ###

    return s

print ("sigmoid([0, 2]) = " + str(sigmoid(np.array([0,2]))))


I am working on the file "Coursera_DL/Logistic+Regression+with+a+Neural+Network+mindset+v4.ipynb".
The first task is:
Expected Output:

          ** dw **
       [[ 0.99845601]
     [ 2.39507239]]

          ** db **
         0.00145557813678

          ** cost **
         5.801545319394553

d) Optimization

You have initialized your parameters.
You are also able to compute a cost function and its gradient.
Now, you want to update the parameters using gradient descent.

Exercise: Write down the optimization function. The goal is to learn $w$ and $b$ by minimizing the cost function $J$. For a parameter $\theta$, the update rule is $ \theta = \theta - \alpha \text{ } d\theta$, where $\alpha$ is the learning rate.
Can you write Python code for it?

# GRADED FUNCTION: optimize

def optimize(w, b, X, Y, num_iterations, learning_rate, print_cost = False):
    """
    This function optimizes w and b by running a gradient descent algorithm

    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of shape (num_px * num_px * 3, number of examples)
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat), of shape (1, number of examples)
    num_iterations -- number of iterations of the optimization loop
    learning_rate -- learning rate of the gradient descent update rule
    print_cost -- True to print the loss every 100 steps

    Returns:
    params -- dictionary containing the weights w and bias b
    grads -- dictionary containing the gradients of the weights and bias with respect to the cost function
    costs -- list of all the costs computed during the optimization, this will be used to plot the learning curve.

    Tips:
    You basically need to write down two steps and iterate through them:
        1) Calculate the cost and the gradient for the current parameters. Use propagate().
        2) Update the parameters using gradient descent rule for w and b.
    """

    costs = []

    for i in range(num_iterations):

        # Cost and gradient calculation (≈ 1-4 lines of code)
        ### START CODE HERE ###
        grads, cost = propagate(w, b, X, Y)
        ### END CODE HERE ###

        # Retrieve derivatives from grads
        dw = grads["dw"]
        db = grads["db"]

        # update rule (≈ 2 lines of code)
        ### START CODE HERE ###
        w = w - learning_rate * dw
        b = b - learning_rate * db
        ### END CODE HERE ###

        # Record the costs
        if i % 100 == 0:
            costs.append(cost)

        # Print the cost every 100 training examples
        if print_cost and i % 100 == 0:
            print ("Cost after iteration %i: %f" %(i, cost))

    params = {"w": w,
        ...


I am working on the file "Coursera_DL/Logistic+Regression+with+a+Neural+Network+mindset+v4.ipynb".
The first task is:
Expected Output:

          ** w **
         [[ 0.]
 [ 0.]]

          ** b **
         0

For image inputs, w will be of shape (num_px $\times$ num_px $\times$ 3, 1).
4.3 - Forward and Backward propagation
Now that your parameters are initialized, you can do the "forward" and "backward" propagation steps for learning the parameters.
Exercise: Implement a function propagate() that computes the cost function and its gradient.
Hints:
Forward Propagation:
- You get X
- You compute $A = \sigma(w^T X + b) = (a^{(0)}, a^{(1)}, ..., a^{(m-1)}, a^{(m)})$
- You calculate the cost function: $J = -\frac{1}{m}\sum_{i=1}^{m}y^{(i)}\log(a^{(i)})+(1-y^{(i)})\log(1-a^{(i)})$
Here are the two formulas you will be using:
$$ \frac{\partial J}{\partial w} = \frac{1}{m}X(A-Y)^T\tag{7}$$
$$ \frac{\partial J}{\partial b} = \frac{1}{m} \sum_{i=1}^m (a^{(i)}-y^{(i)})\tag{8}$$
Can you write Python code for it?

# GRADED FUNCTION: propagate

def propagate(w, b, X, Y):
    """
    Implement the cost function and its gradient for the propagation explained above

    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of size (num_px * num_px * 3, number of examples)
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat) of size (1, number of examples)

    Return:
    cost -- negative log-likelihood cost for logistic regression
    dw -- gradient of the loss with respect to w, thus same shape as w
    db -- gradient of the loss with respect to b, thus same shape as b

    Tips:
    - Write your code step by step for the propagation. np.log(), np.dot()
    """

    m = X.shape[1]

    # FORWARD PROPAGATION (FROM X TO COST)
    ### START CODE HERE ### (≈ 2 lines of code)
    A = sigmoid(np.dot(w.T, X)+b)                                    # compute activation
    cost = (- 1 / m) * np.sum(Y * np.log(A) + (1 - Y) * (np.log(1 - A)))# compute cost
    ### END CODE HERE ###

    # BACKWARD PROPAGATION (TO FIND GRAD)
    ### START CODE HERE ### (≈ 2 lines of code)
    dw = (1 / m) * np.dot(X, (A - Y).T)
    db = (1 / m) * np.sum(A - Y)
    ### END CODE HERE ###

    assert(dw.shape == w.shape)
    assert(db.dtype == float)
    cost = np.squeeze(cost)
    assert(cost.shape == ())

    grads = {"dw": dw,
             "db": db}

    return grads, cost

w, b, X, Y = np.array([[1.],[2.]]), 2., np.array([[1.,2.,-1.],[3.,4.,-3.2]]), np.array([[1,0,1]])
grads, cost = propagate(w, b, X, Y)
print ("dw = " + str(grads["dw"]))
print ("db = " + str(grads["db"]))
print ("cost = " + str(cost))


I am working on the file "Coursera_DL/Logistic+Regression+with+a+Neural+Network+mindset+v4.ipynb".
The first task is:
Expected Output:

             **predictions**

            [[ 1.  1.  0.]]

What to remember:
You've implemented several functions that:
- Initialize (w,b)
- Optimize the loss iteratively to learn parameters (w,b):
    - computing the cost and its gradient
    - updating the parameters using gradient descent
- Use the learned (w,b) to predict the labels for a given set of examples
5 - Merge all functions into a model
You will now see how the overall model is structured by putting together all the building blocks (functions implemented in the previous parts) together, in the right order.
Exercise: Implement the model function. Use the following notation:
    - Y_prediction for your predictions on the test set
    - Y_prediction_train for your predictions on the train set
    - w, costs, grads for the outputs of optimize()
Can you write Python code for it?

# GRADED FUNCTION: model

def model(X_train, Y_train, X_test, Y_test, num_iterations = 2000, learning_rate = 0.5, print_cost = False):
    """
    Builds the logistic regression model by calling the function you've implemented previously

    Arguments:
    X_train -- training set represented by a numpy array of shape (num_px * num_px * 3, m_train)
    Y_train -- training labels represented by a numpy array (vector) of shape (1, m_train)
    X_test -- test set represented by a numpy array of shape (num_px * num_px * 3, m_test)
    Y_test -- test labels represented by a numpy array (vector) of shape (1, m_test)
    num_iterations -- hyperparameter representing the number of iterations to optimize the parameters
    learning_rate -- hyperparameter representing the learning rate used in the update rule of optimize()
    print_cost -- Set to true to print the cost every 100 iterations

    Returns:
    d -- dictionary containing information about the model.
    """

    ### START CODE HERE ###

    # initialize parameters with zeros (≈ 1 line of code)
    w, b = initialize_with_zeros((X_train.shape[0]))

    # Gradient descent (≈ 1 line of code)
    parameters, grads, costs = optimize(w, b, X_train, Y_train, num_iterations, learning_rate, print_cost)

    # Retrieve parameters w and b from dictionary "parameters"
    w = parameters["w"]
    b = parameters["b"]

    # Predict test/train set examples (≈ 2 lines of code)
    Y_prediction_test = predict(w, b, X_test)
    Y_prediction_train = predict(w, b, X_train)

    ### END CODE HERE ###

    # Print train/test Errors
    print("train accuracy: {} %".format(100 - np.mean(np.abs(Y_prediction_train - Y_train)) * 100))
    print("test accuracy: {} %".format(100 - np.mean(np.abs(Y_prediction_test - Y_test)) * 100))

    d = {"costs": costs,
         "Y_prediction_test": Y_prediction_test,
         "Y_prediction_train" : Y_prediction_train,
         "w" : w,
         "b" : b,
         "learning_rate" : learning_rate,
        ...


I am working on the file "Coursera_DL/Logistic+Regression+with+a+Neural+Network+mindset+v4.ipynb".
The first task is:
Expected Output:

    **train_set_x_flatten shape**
     (12288, 209)

    **train_set_y shape**
    (1, 209)

    **test_set_x_flatten shape**
    (12288, 50)

    **test_set_y shape**
    (1, 50)

  **sanity check after reshaping**
  [17 31 56 22 33]

To represent color images, the red, green and blue channels (RGB) must be specified for each pixel, and so the pixel value is actually a vector of three numbers ranging from 0 to 255.
One common preprocessing step in machine learning is to center and standardize your dataset, meaning that you substract the mean of the whole numpy array from each example, and then divide each example by the standard deviation of the whole numpy array. But for picture datasets, it is simpler and more convenient and works almost as well to just divide every row of the dataset by 255 (the maximum value of a pixel channel).

Let's standardize our dataset.
Can you write Python code for it?

train_set_x = train_set_x_flatten/255.
test_set_x = test_set_x_flatten/255.


I am working on the file "Coursera_DL/Logistic+Regression+with+a+Neural+Network+mindset+v4.ipynb".
The first task is:
Expected Output:

        **w**
       [[ 0.19033591]
 [ 0.12259159]]

        **b**
        1.92535983008

        **dw**
        [[ 0.67752042]
 [ 1.41625495]]

        **db**
        0.219194504541

Exercise: The previous function will output the learned w and b. We are able to use w and b to predict the labels for a dataset X. Implement the predict() function. There is two steps to computing predictions:

Calculate $\hat{Y} = A = \sigma(w^T X + b)$

Convert the entries of a into 0 (if activation <= 0.5) or 1 (if activation > 0.5), stores the predictions in a vector Y_prediction. If you wish, you can use an if/else statement in a for loop (though there is also a way to vectorize this).
Can you write Python code for it?

# GRADED FUNCTION: predict

def predict(w, b, X):
    '''
    Predict whether the label is 0 or 1 using learned logistic regression parameters (w, b)

    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of size (num_px * num_px * 3, number of examples)

    Returns:
    Y_prediction -- a numpy array (vector) containing all predictions (0/1) for the examples in X
    '''

    m = X.shape[1]
    Y_prediction = np.zeros((1,m))
    w = w.reshape(X.shape[0], 1)

    # Compute vector "A" predicting the probabilities of a cat being present in the picture
    ### START CODE HERE ### (≈ 1 line of code)
    A = sigmoid(np.dot(w.T,X)+ b)
    ### END CODE HERE ###

    for i in range(A.shape[1]):

        # Convert probabilities A[0,i] to actual predictions p[0,i]
        ### START CODE HERE ### (≈ 4 lines of code)
        Y_prediction[0, i] = 1 if A[0, i] > 0.5 else 0
        ### END CODE HERE ###

    assert(Y_prediction.shape == (1, m))

    return Y_prediction

w = np.array([[0.1124579],[0.23106775]])
b = -0.3
X = np.array([[1.,-1.1,-3.2],[1.2,2.,0.1]])
print ("predictions = " + str(predict(w, b, X)))


I am working on the file "Coursera_DL/Logistic+Regression+with+a+Neural+Network+mindset+v4.ipynb".
The first task is:
Expected Output:

    **sigmoid([0, 2])**
     [ 0.5         0.88079708]

4.2 - Initializing parameters
Exercise: Implement parameter initialization in the cell below. You have to initialize w as a vector of zeros. If you don't know what numpy function to use, look up np.zeros() in the Numpy library's documentation.
Can you write Python code for it?

# GRADED FUNCTION: initialize_with_zeros

def initialize_with_zeros(dim):
    """
    This function creates a vector of zeros of shape (dim, 1) for w and initializes b to 0.

    Argument:
    dim -- size of the w vector we want (or number of parameters in this case)

    Returns:
    w -- initialized vector of shape (dim, 1)
    b -- initialized scalar (corresponds to the bias)
    """

    ### START CODE HERE ### (≈ 1 line of code)
    w = np.zeros(shape=(dim, 1))
    b = 0
    ### END CODE HERE ###

    assert(w.shape == (dim, 1))
    assert(isinstance(b, float) or isinstance(b, int))

    return w, b

dim = 2
w, b = initialize_with_zeros(dim)
print ("w = " + str(w))
print ("b = " + str(b))


I am working on the file "Coursera_DL/Logistic+Regression+with+a+Neural+Network+mindset+v4.ipynb".
The first task is:
Interpretation:
You can see the cost decreasing. It shows that the parameters are being learned. However, you see that you could train the model even more on the training set. Try to increase the number of iterations in the cell above and rerun the cells. You might see that the training set accuracy goes up, but the test set accuracy goes down. This is called overfitting.
6 - Further analysis (optional/ungraded exercise)
Congratulations on building your first image classification model. Let's analyze it further, and examine possible choices for the learning rate $\alpha$.
Choice of learning rate
Reminder:
In order for Gradient Descent to work you must choose the learning rate wisely. The learning rate $\alpha$  determines how rapidly we update the parameters. If the learning rate is too large we may "overshoot" the optimal value. Similarly, if it is too small we will need too many iterations to converge to the best values. That's why it is crucial to use a well-tuned learning rate.
Let's compare the learning curve of our model with several choices of learning rates. Run the cell below. This should take about 1 minute. Feel free also to try different values than the three we have initialized the learning_rates variable to contain, and see what happens.
Can you write Python code for it?

learning_rates = [0.01, 0.001, 0.0001]
models = {}
for i in learning_rates:
    print ("learning rate is: " + str(i))
    models[str(i)] = model(train_set_x, train_set_y, test_set_x, test_set_y, num_iterations = 1500, learning_rate = i, print_cost = False)
    print ('\n' + "-------------------------------------------------------" + '\n')

for i in learning_rates:
    plt.plot(np.squeeze(models[str(i)]["costs"]), label= str(models[str(i)]["learning_rate"]))

plt.ylabel('cost')
plt.xlabel('iterations')

legend = plt.legend(loc='upper center', shadow=True)
frame = legend.get_frame()
frame.set_facecolor('0.90')
plt.show()


I am working on the file "Coursera_DL/Logistic+Regression+with+a+Neural+Network+mindset+v4.ipynb".
The first task is:
Interpretation:
- Different learning rates give different costs and thus different predictions results.
- If the learning rate is too large (0.01), the cost may oscillate up and down. It may even diverge (though in this example, using 0.01 still eventually ends up at a good value for the cost).
- A lower cost doesn't mean a better model. You have to check if there is possibly overfitting. It happens when the training accuracy is a lot higher than the test accuracy.
- In deep learning, we usually recommend that you:
    - Choose the learning rate that better minimizes the cost function.
    - If your model overfits, use other techniques to reduce overfitting. (We'll talk about this in later videos.)
7 - Test with your own image (optional/ungraded exercise)
Congratulations on finishing this assignment. You can use your own image and see the output of your model. To do that:
    1. Click on "File" in the upper bar of this notebook, then click "Open" to go on your Coursera Hub.
    2. Add your image to this Jupyter Notebook's directory, in the "images" folder
    3. Change your image's name in the following code
    4. Run the code and check if the algorithm is right (1 = cat, 0 = non-cat)!
Can you write Python code for it?

## START CODE HERE ## (PUT YOUR IMAGE NAME)
my_image = "my_image.jpg"   # change this to the name of your image file
## END CODE HERE ##

# We preprocess the image to fit your algorithm.
fname = "images/" + my_image
image = np.array(ndimage.imread(fname, flatten=False))
my_image = scipy.misc.imresize(image, size=(num_px,num_px)).reshape((1, num_px*num_px*3)).T
my_predicted_image = predict(d["w"], d["b"], my_image)

plt.imshow(image)
print("y = " + str(np.squeeze(my_predicted_image)) + ", your algorithm predicts a \"" + classes[int(np.squeeze(my_predicted_image)),].decode("utf-8") +  "\" picture.")


I am working on the file "Coursera_DL/Logistic+Regression+with+a+Neural+Network+mindset+v4.ipynb".
The first task is:
Let's also plot the cost function and the gradients.
Can you write Python code for it?

# Plot learning curve (with costs)
costs = np.squeeze(d['costs'])
plt.plot(costs)
plt.ylabel('cost')
plt.xlabel('iterations (per hundreds)')
plt.title("Learning rate =" + str(d["learning_rate"]))
plt.show()


I am working on the file "Coursera_DL/Logistic+Regression+with+a+Neural+Network+mindset+v4.ipynb".
The first task is:
Many software bugs in deep learning come from having matrix/vector dimensions that don't fit. If you can keep your matrix/vector dimensions straight you will go a long way toward eliminating many bugs.
Exercise: Find the values for:
    - m_train (number of training examples)
    - m_test (number of test examples)
    - num_px (= height = width of a training image)
Remember that train_set_x_orig is a numpy-array of shape (m_train, num_px, num_px, 3). For instance, you can access m_train by writing train_set_x_orig.shape[0].
Can you write Python code for it?

### START CODE HERE ### (≈ 3 lines of code)
m_train = len(train_set_x_orig)
m_test = len(test_set_x_orig)
num_px = np.shape(train_set_x_orig[0])[0]
### END CODE HERE ###

print ("Number of training examples: m_train = " + str(m_train))
print ("Number of testing examples: m_test = " + str(m_test))
print ("Height/Width of each image: num_px = " + str(num_px))
print ("Each image is of size: (" + str(num_px) + ", " + str(num_px) + ", 3)")
print ("train_set_x shape: " + str(train_set_x_orig.shape))
print ("train_set_y shape: " + str(train_set_y.shape))
print ("test_set_x shape: " + str(test_set_x_orig.shape))
print ("test_set_y shape: " + str(test_set_y.shape))


I am working on the file "Coursera_DL/Logistic+Regression+with+a+Neural+Network+mindset+v4.ipynb".
The first task is:
Run the following cell to train your model.
Can you write Python code for it?

d = model(train_set_x, train_set_y, test_set_x, test_set_y, num_iterations = 2000, learning_rate = 0.005, print_cost = True)


I am working on the file "Coursera_DL/Logistic+Regression+with+a+Neural+Network+mindset+v4.ipynb".
The first task is:
We added "_orig" at the end of image datasets (train and test) because we are going to preprocess them. After preprocessing, we will end up with train_set_x and test_set_x (the labels train_set_y and test_set_y don't need any preprocessing).
Each line of your train_set_x_orig and test_set_x_orig is an array representing an image. You can visualize an example by running the following code. Feel free also to change the index value and re-run to see other images.
Can you write Python code for it?

# Example of a picture
index = 3
plt.imshow(train_set_x_orig[index])
print ("y = " + str(train_set_y[:, index]) + ", it's a '" + classes[np.squeeze(train_set_y[:, index])].decode("utf-8") +  "' picture.")


I am working on the file "Coursera_DL/Python+Basics+With+Numpy+v3.ipynb".
The first task is:
Any time you need more info on a numpy function, we encourage you to look at the official documentation.
You can also create a new cell in the notebook and write np.exp? (for example) to get quick access to the documentation.
Exercise: Implement the sigmoid function using numpy.
Instructions: x could now be either a real number, a vector, or a matrix. The data structures we use in numpy to represent these shapes (vectors, matrices...) are called numpy arrays. You don't need to know more for now.
$$ \text{For } x \in \mathbb{R}^n \text{,     } sigmoid(x) = sigmoid\begin{pmatrix}
    x_1  \
    x_2  \
    ...  \
    x_n  \
\end{pmatrix} = \begin{pmatrix}
    \frac{1}{1+e^{-x_1}}  \
    \frac{1}{1+e^{-x_2}}  \
    ...  \
    \frac{1}{1+e^{-x_n}}  \
\end{pmatrix}\tag{1} $$
Can you write Python code for it?

# GRADED FUNCTION: sigmoid

import numpy as np # this means you can access numpy functions by writing np.function() instead of numpy.function()

def sigmoid(x):
    """
    Compute the sigmoid of x

    Arguments:
    x -- A scalar or numpy array of any size

    Return:
    s -- sigmoid(x)
    """

    ### START CODE HERE ### (≈ 1 line of code)
    s = 1/(1+np.exp(np.asarray(-x)))
    ### END CODE HERE ###

    return s

x = np.array([1, 2, 3])
sigmoid(x)


I am working on the file "Coursera_DL/Python+Basics+With+Numpy+v3.ipynb".
The first task is:
As you may have noticed, the vectorized implementation is much cleaner and more efficient. For bigger vectors/matrices, the differences in running time become even bigger.
Note that np.dot() performs a matrix-matrix or matrix-vector multiplication. This is different from np.multiply() and the * operator (which is equivalent to  .* in Matlab/Octave), which performs an element-wise multiplication.
2.1 Implement the L1 and L2 loss functions
Exercise: Implement the numpy vectorized version of the L1 loss. You may find the function abs(x) (absolute value of x) useful.
Reminder:
- The loss is used to evaluate the performance of your model. The bigger your loss is, the more different your predictions ($ \hat{y} $) are from the true values ($y$). In deep learning, you use optimization algorithms like Gradient Descent to train your model and to minimize the cost.
- L1 loss is defined as:
$$\begin{align} & L_1(\hat{y}, y) = \sum_{i=0}^m|y^{(i)} - \hat{y}^{(i)}| \end{align}\tag{6}$$
Can you write Python code for it?

# GRADED FUNCTION: L1

def L1(yhat, y):
    """
    Arguments:
    yhat -- vector of size m (predicted labels)
    y -- vector of size m (true labels)

    Returns:
    loss -- the value of the L1 loss function defined above
    """

    ### START CODE HERE ### (≈ 1 line of code)
    loss = np.sum(np.abs(y - yhat) ,keepdims=True)
    ### END CODE HERE ###

    return loss

yhat = np.array([.9, 0.2, 0.1, .4, .9])
y = np.array([1, 0, 0, 1, 1])
print("L1 = " + str(L1(yhat,y)))


I am working on the file "Coursera_DL/Python+Basics+With+Numpy+v3.ipynb".
The first task is:
Expected Output:

    ** basic_sigmoid(3) **
        0.9525741268224334

Actually, we rarely use the "math" library in deep learning because the inputs of the functions are real numbers. In deep learning we mostly use matrices and vectors. This is why numpy is more useful.
Can you write Python code for it?

### One reason why we use "numpy" instead of "math" in Deep Learning ###
x = [1, 2, 3]
basic_sigmoid(np.asarray(x)) # you will see this give an error when you run it, because x is a vector.


I am working on the file "Coursera_DL/Python+Basics+With+Numpy+v3.ipynb".
The first task is:
Expected Output:

        **normalizeRows(x)**
        [[ 0.          0.6         0.8       ]
 [ 0.13736056  0.82416338  0.54944226]]

Note:
In normalizeRows(), you can try to print the shapes of x_norm and x, and then rerun the assessment. You'll find out that they have different shapes. This is normal given that x_norm takes the norm of each row of x. So x_norm has the same number of rows but only 1 column. So how did it work when you divided x by x_norm? This is called broadcasting and we'll talk about it now!
1.5 - Broadcasting and the softmax function
A very important concept to understand in numpy is "broadcasting". It is very useful for performing mathematical operations between arrays of different shapes. For the full details on broadcasting, you can read the official broadcasting documentation.
Exercise: Implement a softmax function using numpy. You can think of softmax as a normalizing function used when your algorithm needs to classify two or more classes. You will learn more about softmax in the second course of this specialization.
Instructions:
- $ \text{for } x \in \mathbb{R}^{1\times n} \text{,     } softmax(x) = softmax(\begin{bmatrix}
    x_1  &&
    x_2 &&
    ...  &&
    x_n
\end{bmatrix}) = \begin{bmatrix}
     \frac{e^{x_1}}{\sum_{j}e^{x_j}}  &&
    \frac{e^{x_2}}{\sum_{j}e^{x_j}}  &&
    ...  &&
    \frac{e^{x_n}}{\sum_{j}e^{x_j}}
\end{bmatrix} $

$\text{for a matrix } x \in \mathbb{R}^{m \times n} \text{,  $x_{ij}$ maps to the element in the $i^{th}$ row and $j^{th}$ column of $x$, thus we have: }$  $$softmax(x) = softmax\begin{bmatrix}
    x_{11} & x_{12} & x_{13} & \dots  & x_{1n} \
    x_{21} & x_{22} & x_{23} & \dots  & x_{2n} \
    \vdots & \vdots & \vdots & \ddots & \vdots \
    x_{m1} & x_{m2} & x_{m3} & \dots  & x_{mn}
\end{bmatrix} = \begin{bmatrix}
    \frac{e^{x_{11}}}{\sum_{j}e^{x_{1j}}} & \frac{e^{x_{12}}}{\sum_{j}e^{x_{1j}}} & \frac{e^{x_{13}}}{\sum_{j}e^{x_{1j}}} & \dots  &...
Can you write Python code for it?

# GRADED FUNCTION: softmax

def softmax(x):
    """Calculates the softmax for each row of the input x.

    Your code should work for a row vector and also for matrices of shape (n, m).

    Argument:
    x -- A numpy matrix of shape (n,m)

    Returns:
    s -- A numpy matrix equal to the softmax of x, of shape (n,m)
    """

    ### START CODE HERE ### (≈ 3 lines of code)
    # Apply exp() element-wise to x. Use np.exp(...).
    x_exp = np.exp(x)

    # Create a vector x_sum that sums each row of x_exp. Use np.sum(..., axis = 1, keepdims = True).
    x_sum = np.sum(x_exp, axis=1, keepdims= True)

    # Compute softmax(x) by dividing x_exp by x_sum. It should automatically use numpy broadcasting.
    s = x_exp/x_sum

    ### END CODE HERE ###

    return s

x = np.array([
    [9, 2, 5, 0, 0],
    [7, 5, 0, 0 ,0]])
print("softmax(x) = " + str(softmax(x)))


I am working on the file "Coursera_DL/Python+Basics+With+Numpy+v3.ipynb".
The first task is:
Expected Output:

         **sigmoid([1,2,3])**
         array([ 0.73105858,  0.88079708,  0.95257413])

1.2 - Sigmoid gradient
As you've seen in lecture, you will need to compute gradients to optimize loss functions using backpropagation. Let's code your first gradient function.
Exercise: Implement the function sigmoid_grad() to compute the gradient of the sigmoid function with respect to its input x. The formula is: $$sigmoid_derivative(x) = \sigma'(x) = \sigma(x) (1 - \sigma(x))\tag{2}$$
You often code this function in two steps:
1. Set s to be the sigmoid of x. You might find your sigmoid(x) function useful.
2. Compute $\sigma'(x) = s(1-s)$
Can you write Python code for it?

# GRADED FUNCTION: sigmoid_derivative

def sigmoid_derivative(x):
    """
    Compute the gradient (also called the slope or derivative) of the sigmoid function with respect to its input x.
    You can store the output of the sigmoid function into variables and then use it to calculate the gradient.

    Arguments:
    x -- A scalar or numpy array

    Return:
    ds -- Your computed gradient.
    """

    ### START CODE HERE ### (≈ 2 lines of code)
    s = 1/(1+np.exp((-x)))
    ds = s*(1-s)
    ### END CODE HERE ###

    return ds

x = np.array([1, 2, 3])
print ("sigmoid_derivative(x) = " + str(sigmoid_derivative(x)))


I am working on the file "Coursera_DL/Python+Basics+With+Numpy+v3.ipynb".
The first task is:
Expected Output:

         **sigmoid_derivative([1,2,3])**
         [ 0.19661193  0.10499359  0.04517666]

1.3 - Reshaping arrays
Two common numpy functions used in deep learning are np.shape and np.reshape().
- X.shape is used to get the shape (dimension) of a matrix/vector X.
- X.reshape(...) is used to reshape X into some other dimension.
For example, in computer science, an image is represented by a 3D array of shape $(length, height, depth = 3)$. However, when you read an image as the input of an algorithm you convert it to a vector of shape $(lengthheight3, 1)$. In other words, you "unroll", or reshape, the 3D array into a 1D vector.

Exercise: Implement image2vector() that takes an input of shape (length, height, 3) and returns a vector of shape (length*height*3, 1). For example, if you would like to reshape an array v of shape (a, b, c) into a vector of shape (a*b,c) you would do:
python
v = v.reshape((v.shape[0]*v.shape[1], v.shape[2])) # v.shape[0] = a ; v.shape[1] = b ; v.shape[2] = c
- Please don't hardcode the dimensions of image as a constant. Instead look up the quantities you need with image.shape[0], etc.
Can you write Python code for it?

# GRADED FUNCTION: image2vector
def image2vector(image):
    """
    Argument:
    image -- a numpy array of shape (length, height, depth)

    Returns:
    v -- a vector of shape (length*height*depth, 1)
    """

    ### START CODE HERE ### (≈ 1 line of code)
    v = image.reshape((image.shape[0]*image.shape[1]*image.shape[2], 1))
    ### END CODE HERE ###

    return v

# This is a 3 by 3 by 2 array, typically images will be (num_px_x, num_px_y,3) where 3 represents the RGB values
image = np.array([[[ 0.67826139,  0.29380381],
        [ 0.90714982,  0.52835647],
        [ 0.4215251 ,  0.45017551]],

       [[ 0.92814219,  0.96677647],
        [ 0.85304703,  0.52351845],
        [ 0.19981397,  0.27417313]],

       [[ 0.60659855,  0.00533165],
        [ 0.10820313,  0.49978937],
        [ 0.34144279,  0.94630077]]])

print ("image2vector(image) = " + str(image2vector(image)))


I am working on the file "Coursera_DL/Python+Basics+With+Numpy+v3.ipynb".
The first task is:
Expected output:
test: Hello World

What you need to remember:
- Run your cells using SHIFT+ENTER (or "Run cell")
- Write code in the designated areas using Python 3 only
- Do not modify the code outside of the designated areas
1 - Building basic functions with numpy
Numpy is the main package for scientific computing in Python. It is maintained by a large community (www.numpy.org). In this exercise you will learn several key numpy functions such as np.exp, np.log, and np.reshape. You will need to know how to use these functions for future assignments.
1.1 - sigmoid function, np.exp()
Before using np.exp(), you will use math.exp() to implement the sigmoid function. You will then see why np.exp() is preferable to math.exp().
Exercise: Build a function that returns the sigmoid of a real number x. Use math.exp(x) for the exponential function.
Reminder:
$sigmoid(x) = \frac{1}{1+e^{-x}}$ is sometimes also known as the logistic function. It is a non-linear function used not only in Machine Learning (Logistic Regression), but also in Deep Learning.

To refer to a function belonging to a specific package you could call it using package_name.function(). Run the code below to see an example with math.exp().
Can you write Python code for it?

# GRADED FUNCTION: basic_sigmoid

import math
import numpy as np
def basic_sigmoid(x):
    """
    Compute sigmoid of x.

    Arguments:
    x -- A scalar

    Return:
    s -- sigmoid(x)
    """

    ### START CODE HERE ### (≈ 1 line of code)
    s = 1/(1 + np.exp(-x))
    ### END CODE HERE ###

    return s

basic_sigmoid(3)


I am working on the file "Coursera_DL/Python+Basics+With+Numpy+v3.ipynb".
The first task is:
In fact, if $ x = (x_1, x_2, ..., x_n)$ is a row vector then $np.exp(x)$ will apply the exponential function to every element of x. The output will thus be: $np.exp(x) = (e^{x_1}, e^{x_2}, ..., e^{x_n})$
Can you write Python code for it?

import numpy as np

# example of np.exp
x = np.array([1, 2, 3])
print(np.exp(x)) # result is (exp(1), exp(2), exp(3))


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
%quickref
%magic
%debug
%hist
%pdb
%paste
%cpaste
%reset
%page OBJECT %run script.py %prun statement %time statement %timeit statement
%who, %who_ls, %whos
%xdel variable
Can you write Python code for it?

# May be useful later...
%matplotlib inline


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
50 years of Data Science, David Donoho (Optional)
Link: http://courses.csail.mit.edu/18.337/2015/docs/50YearsDataScience.pdf
Python Functions
Video: https://www.coursera.org/learn/python-data-analysis/lecture/v7PKy/python-functions
High-level, interpretted language. Often used interactively, especially in Jupyter. Dynamically-typed, meaning variables are not declared as a specific type.
Jupyter is stateful, meaning it retains values run in previous cells.
A function returns None if not specified.
Examples Demo'd
Can you write Python code for it?

x = 1
y = 2
x + y

x

# a function, with optional 3rd parameter
def add_numbers(x, y, z=None):
  if (z == None):
    return x + y
  else:
    return x + y + z

print(add_numbers(2, 4, 2))
print(add_numbers(2, 4))


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
Advanced Python Lambda and List Comprehensions
Video: https://www.coursera.org/learn/python-data-analysis/lecture/AVjRT/advanced-python-lambda-and-list-comprehensions
Lambdas are used to create a temporary function.
Can you write Python code for it?

my_function = lambda a, b, c : (a + b) ** c
my_function(1, 2, 3)


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
Aggregation
Can you write Python code for it?

# re-read and prep data
df = pd.read_csv(file)
df.reset_index()

df.groupby("cancellation_policy").agg({"review_scores_value":np.average})

# Note that this doesn't work because of NaN's. See next cell for solution.

# the previous example didn't work because of NaN's. Let's rewrite using np.nanmean

df.groupby("cancellation_policy").agg({"review_scores_value":np.nanmean})

# more than one grouped columns to print
# note the tuple with review_scores_value

df.groupby("cancellation_policy").agg({"review_scores_value":(np.nanmean, np.nanstd),
                                       "reviews_per_month":np.nanmean})


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
Anchors
Can you write Python code for it?

# Anchors, ^ = at start, $ = at end

print(re.search('^Amy', text)) # returns a re object


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
DataFrame Data Structure
Video: https://www.coursera.org/learn/python-data-analysis/lecture/J8MzB/dataframe-data-structure
Dataframes are the heart of the panda library. A 2-dimensional series, with an index and multiple columns of data.
Can you write Python code for it?

import pandas as pd

# let's set up some data to use (multiple dictionaries)

record1 = pd.Series({'Name': 'Alice',
                     'Class': 'Physics',
                     'Score': 85})
record2 = pd.Series({'Name': 'Jack',
                     'Class': 'Chemistry',
                     'Score': 82})
record3 = pd.Series({'Name': 'Helen',
                     'Class': 'Biology',
                     'Score': 90})

df = pd.DataFrame([record1, record2, record3],
                  index = ['school1', 'school2', 'school3'])

df.head()

# Another way to enter the data (list of dictionaries)

students = [{'Name': 'Alice',
                     'Class': 'Physics',
                     'Score': 85},
            {'Name': 'Jack',
             'Class': 'Chemistry',
                     'Score': 82},
            {'Name': 'Helen',
             'Class': 'Biology',
             'Score': 90}]

df = pd.DataFrame(students, index = ['school1', 'school2', 'school1']) # two school 1's
df.head()

# accessing by index

df.loc['school2']

# what type is the data?

print(type(df.loc['school2'])) # one record, so it's a Series
print(type(df.loc['school1'])) # two records, do it's a DataFrame

df.loc['school1']

# only return specific data, not entire record

df.loc['school1', 'Name']

# how to transpose the data

df.T

# What if we want all the contents of a column? Transpose first....

df.T.loc['Name']

# but this does the same...! (why transpose then????)
# note: .loc() not used to get the columnar data

df['Name']

# Columnar selection returns a Series, not a DataFrame

type(df['Name'])

# Just use type() to check what something returns

# .loc() can use slicing

df.loc[:,['Name', 'Score']]

# data can be dropped

df.drop('school2')

# ...but the dropped data is still there!
# (returns a copy)

df

# Use inpalce=True to really drop the data!
# axis=0 is a row (default), axis=1 is a column
# Using a copy of the data here

copy_df = df.copy()
copy_df.drop('Name', inplace=True, axis=1)
copy_df

# A 2nd way to drop a column
# This...


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
DataFrame Indexing and Loading
Video: https://www.coursera.org/learn/python-data-analysis/lecture/BLY4L/dataframe-indexing-and-loading
Basic process:

read data in, usually from external file (CSV, Excel)
clean and manipulate the dataset for analysis
Can you write Python code for it?

!cat /content/Coursera-Introduction-to-Data-Science-with-Python/resources/week-2/datasets/Admission_Predict.csv | head

# loading a dataset from CSV

import pandas as pd

file = '/content/Coursera-Introduction-to-Data-Science-with-Python/resources/week-2/datasets/Admission_Predict.csv'
# assign an index when reading in the CSV. Can be indexed or numeric
df = pd.read_csv(file, index_col = 'Serial No.') # index_col = 0 also works

df.head()

# Let's create a new DF and do a cleanup
new_df = df.rename(columns = {'GRE Score': 'GRE Score',
                              'TOEFL Score': 'TOEFL Score',
                              'University Rating': 'University Rating',
                              'SOP': 'Statement of Purpose',
                              'LOR': 'Letter of Recommendation',
                              'CGPA': 'CGPA',
                              'Research': 'Research',
                              'Chance of Admit': 'Chance of Admit'})
new_df.head()

# Some did not rename. Why not? Note the extra space in some column titles

new_df.columns

# Fixing the column names should work

new_df = df.rename(columns = {'GRE Score': 'GRE Score',
                              'TOEFL Score': 'TOEFL Score',
                              'University Rating': 'University Rating',
                              'SOP': 'Statement of Purpose',
                              'LOR ': 'Letter of Recommendation',
                              'CGPA': 'CGPA',
                              'Research': 'Research',
                              'Chance of Admit ': 'Chance of Admit'})
new_df.head()

# We could have also used strip() before renaming

new_df = new_df.rename(mapper=str.strip, axis='columns')
new_df.head()

df.columns

# Another way to clean up the column names

cols = list(df.columns)
cols = [x.strip() for x in cols]
df.columns = cols
df.columns # no spaces now


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
Dictionaries
Can you write Python code for it?

# example from https://www.w3schools.com/python/python_dictionaries.asp
thisdict = {
  "brand": "Ford",
  "model": "Mustang",
  "year": 1964
}
for spec, data in thisdict.items():
  print(spec, data)

for data in thisdict.values():
  print(data)

for key in thisdict.keys():
  print(key)


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
Example: Manipulating DataFrame
Video: https://www.coursera.org/learn/python-data-analysis/lecture/BvAgY/example-manipulating-dataframe
Can you write Python code for it?

import pandas as pd

!git clone https://github.com/pbeens/Coursera-Introduction-to-Data-Science-with-Python.git

file = '/content/Coursera-Introduction-to-Data-Science-with-Python/resources/week-2/datasets/presidents.csv'

df = pd.read_csv(file)
df.head()

# create first name field

# one technique: copy name field and do cleanup - not efficient

df['First'] = df['President']
df['First'] = df['First'].replace('[ ].*', '', regex=True)
df.head()

# Get rid of ['First'] so we can try another technique

del(df['First'])

# Another way, using .apply()

def split_name(row):
  row['First'] = row['President'].split(' ')[0]
  row['Last'] = row['President'].split(' ')[-1]
  return row

df = df.apply(split_name, axis='columns')
df.head()

# cleanup to try again

del(df['First'])
del(df['Last'])

# using a pattern

pattern='(^[\w]*)(?:.* )([\w]*$)'
df['President'].str.extract(pattern).head()

# Using the pattern again, but this time naming the fields with ?P<nameOfField>

pattern='(?P<First>^[\w]*)(?:.* )(?P<Last>[\w]*$)'
names = df['President'].str.extract(pattern).head()
names

df['First'], df['Last'] = names['First'], names['Last']
df.head()

# Getting the date of birth using the useful str module

df['Born'] = df['Born'].str.extract('([\w]{3} [\w]{1,2}, [\w]{4})')
df['Born'].head()

# Let's re-do this so it's a date format, not object

df['Born'] = pd.to_datetime(df['Born'])
df['Born'].head()


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
Example: New York Times and Hashtags
Can you write Python code for it?

# Example: New York Times and Hashtags

file = '/content/Coursera-Introduction-to-Data-Science-with-Python/resources/week-1/datasets/nytimeshealth.txt'
with open(file) as f:
  health = f.read()

print('=== health[:500] ===')
print(health[:500])

# let's grab hashtags

pattern = '#[\w\d]*(?=\s)'

print(re.findall(pattern, health))
print(set(re.findall(pattern, health))) # get rid of dupes


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
Filtering
Can you write Python code for it?

 # filtering with mean 'review_scores_value' results > 9.2

 df.groupby('cancellation_policy').filter(lambda x: np.nanmean(x['review_scores_value'])>9.2)
 df.head()


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
Groups
Can you write Python code for it?

# Groups

# Let's improve upone the previous example by creating groups (in parentheses)
# item.groups() returns a tuple of each group item
# group(0) is the entire item, group(1) is first, etc.

print('=== Old way ===')
print(re.findall('[\w ]*\[edit\]', wiki))

print('=== item.groups() ===')
for item in re.finditer('([\w ]*)(\[edit\])', wiki):
  print(item.groups())

print('=== item.group(1) ===')
for item in re.finditer('([\w ]*)(\[edit\])', wiki):
  print(item.group(1))

# groups can be named with ?P<name> and use of groupdict()[]

print('=== named with ?P<name> ===')
for item in re.finditer('(?P<title>[\w ]*)(?P<edit_link>\[edit\])', wiki):
  print(item.groupdict()['title'])

print('=== groupdict() ===')
for item in re.finditer('(?P<title>[\w ]*)(?P<edit_link>\[edit\])', wiki):
  print(item.groupdict())

# . = single character
# /d = digit
# /s = white space


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
Indexing Dataframes
Video: https://www.coursera.org/learn/python-data-analysis/lecture/w8ZNA/indexing-dataframes
Indexes can be set automatically or set manually with set_index().
Can you write Python code for it?

import pandas as pd

!git clone https://github.com/pbeens/Coursera-Introduction-to-Data-Science-with-Python.git

file = '/content/Coursera-Introduction-to-Data-Science-with-Python/resources/week-2/datasets/Admission_Predict.csv'
df = pd.read_csv(file, index_col=0)
# clean up the empty spaces
df.columns = [x.strip() for x in df.columns]
df.head()

df['Serial No.'] = df.index # but it's already Serial No so why do we do this???
df.head()

df = df.set_index('Chance of Admit') # set index to another column
df.head()

# to get rid of index

df = df.reset_index()
df.head()

# multi-column index, using a different data set

file = '/content/Coursera-Introduction-to-Data-Science-with-Python/resources/week-2/datasets/census.csv'
df = pd.read_csv(file)
df.head()

df.columns

# Checking the unqiue values in the SUMLEV column

df['SUMLEV'].unique()

# We only want a subset, where SUMLEV = 50

df = df[df['SUMLEV'] == 50]
df.head()

# Further reducing the subset

columns_to_keep = ['STNAME','CTYNAME','BIRTHS2010','BIRTHS2011','BIRTHS2012','BIRTHS2013',
                   'BIRTHS2014','BIRTHS2015','POPESTIMATE2010','POPESTIMATE2011',
                   'POPESTIMATE2012','POPESTIMATE2013','POPESTIMATE2014','POPESTIMATE2015']
df = df[columns_to_keep]
df.head()

# Let's try two columns for the index

df = df.set_index(['STNAME', 'CTYNAME'])
df.head()

# Let's do a complex query using the index
# Watch the order!

df.loc['Michigan', 'Washtenaw County']

# looking at two counties
# the index is a list of tuples

df.loc[[('Michigan', 'Washtenaw County'),
        ('Michigan', 'Wayne County')]]

# Transposed with .T

df.loc[[('Michigan', 'Washtenaw County'),
        ('Michigan', 'Wayne County')]].T


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
List comprehensions are an abbreviated method to created an iterable list. Are created on one line. LCs can include conditonal statements.
Can you write Python code for it?

# conventional way
my_list = []
for num in range(20):
  if num % 2 == 0:
    my_list.append(num)
print(my_list)

# list comprehension way
print([n for n in range(20) if n % 2 == 0])


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
Look-ahead and Look-behind (stuff to ignore)
Can you write Python code for it?

# Look-ahead and Look-behind with ?=
# This example ignores [edit]

for item in re.finditer('(?P<title>[\w ]+)(?=\[edit\])', wiki):
  print(item)


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
Manipulating Text with Regular Expression
Video: https://www.coursera.org/learn/python-data-analysis/lecture/aSuH7/manipulating-text-with-regular-expression
AKA Regex. Pattern matching. Returns chunks of text. Why use? To check if pattern exists, or return instances of text, or clean up data.
Can you write Python code for it?


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
Missing Values
Video: https://www.coursera.org/learn/python-data-analysis/lecture/Sg47m/missing-values
Can you write Python code for it?

import pandas as pd

!git clone https://github.com/pbeens/Coursera-Introduction-to-Data-Science-with-Python.git

file = '/content/Coursera-Introduction-to-Data-Science-with-Python/resources/week-2/datasets/class_grades.csv'

df = pd.read_csv(file)
df.head()

# How a mask works

mask=df.isnull()
mask.head()

# You can get rid of NaN data entirely

df.dropna().head(10)

# Replace NaN with...
# Note use of inplace to overwrite the dataframe

df.fillna(0, inplace=True)
df.head()

file = '/content/Coursera-Introduction-to-Data-Science-with-Python/resources/week-2/datasets/log.csv'

df = pd.read_csv(file)
df.head(10)

# Note: time in epoch format

# bfill() and ffill() (backward- and forward-fill)

df = df.set_index('time')
df = df.sort_index()
df.head(10)

# not all time indices are unique so let's use multi-column index

df = df.reset_index()
df = df.set_index(['time', 'user'])
df.head(10)

# let's fix missing data

df = df.fillna(method='ffill')
df.head(10)

# Playing with some new data

df = pd.DataFrame({'A': [1,1,2,3,4],
                   'B': [3,6,3,8,9],
                   'C': ['a', 'b', 'c', 'd', 'e']})
df

 # replacing the 1's with 100

 df.replace(1,100)

# multiple replacements

df.replace([1,3], [100,300])

# regex replacement (setup)

file = '/content/Coursera-Introduction-to-Data-Science-with-Python/resources/week-2/datasets/log.csv'

df = pd.read_csv(file)
df.head(10)

# regex replacement

df = df.replace(to_replace='.*.html$', value='webpage', regex=True)
df.head()


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
Multiple assignments can be done at once.
Can you write Python code for it?

a, b, c = 1, "abc", [4, 5]
print(a)
print(b)
print(c)


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
Numerical Python Library (NumPy)
Video: https://www.coursera.org/learn/python-data-analysis/lecture/kAQc1/numerical-python-library-numpy
Can you write Python code for it?

import numpy as np
import math

a = np.array([1, 2, 3])
print(a)
print(a.ndim) # number of dimensions

b = np.array([[1,2,3], [4,5,6]])
print(b)

print(b.shape) # length of each dimension

print(b.dtype) # what types of variables

# also floats
c = np.array([2.2, 5, 1.1])
print(c.dtype.name) # same o/p as w/o .name?

# create according to shape
d = np.zeros((2,3)) # must pass a tuple
print(d)
e = np.ones((2,3))
print(e)
f = np.random.rand(2,3) # not a tuple!
print(f)
print(type(f))

# range of integers
f = np.arange(10, 20, 2) # 2nd parm is exclusive
print(f)
print(type(f))

# a range of floats
np.linspace(0, 2, 15) # 15 numbers from 0 to 2

# Array Operations

a = np.array([10,20,30,40])
b = np.array([1,2,3,4])

c = a + b
print(c)
d = a * b
print(d)

deg_f = np.array([0,-10,-5,-15,0])
deg_c = (deg_f -32) * (5/9)
%precision 2
print(deg_c)

# Boolean expressions on NP arrays
deg_c > -20

# Matrix Manipulation
A = np.array([[1,1],[0,1]])
B = np.array([[2,0],[3,4]])
print(A * B)
# Matrix Product - Note the '@'
print(A@B)

# Video completed to 8:07 only


I am working on the file "Coursera_Introduction_to_Data_Science_with_Python.ipynb".
The first task is:
Patterns and Character Classes
Can you write Python code for it?

# Patterns and Character Classes

grades = 'ACAAAABCBCBAA'

print(re.findall('B', grades)) # ['B', 'B', 'B']

# [] are used for options (or)

print(re.findall('[BC]', grades)) # ['C', 'B', 'C', 'B', 'C', 'B']

print(re.findall('[A][BC]', grades)) # ['AC', 'AB']
print(re.findall('[A][B-C]', grades)) # [B-C] is the range from B-C

# | = OR

print(re.findall('BC|BA', grades)) # ['BC', 'BC', 'BA']

# A carat can be used to negate inside of a set operator

print(re.findall('[^A]', grades)) # ['C', 'B', 'C', 'B', 'C', 'B']
print(re.findall('^[^A]', grades)) # produces [] empty list (starts with A)