I am working on the file "Convolutional_Neural_Network/Convolution+model+-+Application+-+v1 (1)_regularization.ipynb".
The first task is:
Run the following cell to train your model for 100 epochs. Check if your cost after epoch 0 and 5 matches our output. If not, stop the cell and go back to your code!
Can you write Python code for it?

_, _, parameters = model(X_train, Y_train, X_test, Y_test)


I am working on the file "Convolutional_Neural_Network/Convolution+model+-+Application+-+v1 (1)_regularization.ipynb".
The first task is:
Run the next cell to load the "SIGNS" dataset you are going to use.
Can you write Python code for it?

# Loading the data (signs)
X_train_orig, Y_train_orig, X_test_orig, Y_test_orig, classes = load_dataset()


I am working on the file "Convolutional_Neural_Network/Convolution+model+-+Application+-+v1.ipynb".
The first task is:
As a reminder, the SIGNS dataset is a collection of 6 signs representing numbers from 0 to 5.
<img src="images/SIGNS.png" style="width:800px;height:300px;">
The next cell will show you an example of a labelled image in the dataset. Feel free to change the value of index below and re-run to see different examples.
Can you write Python code for it?

# Example of a picture
index = 6
plt.imshow(X_train_orig[index])
print ("y = " + str(np.squeeze(Y_train_orig[:, index])))


I am working on the file "Convolutional_Neural_Network/Convolution+model+-+Application+-+v1.ipynb".
The first task is:
Expected Output

    X = Tensor("Placeholder:0", shape=(?, 64, 64, 3), dtype=float32)

    Y = Tensor("Placeholder_1:0", shape=(?, 6), dtype=float32)

1.2 - Initialize parameters
You will initialize weights/filters $W1$ and $W2$ using tf.contrib.layers.xavier_initializer(seed = 0). You don't need to worry about bias variables as you will soon see that TensorFlow functions take care of the bias. Note also that you will only initialize the weights/filters for the conv2d functions. TensorFlow initializes the layers for the fully connected part automatically. We will talk more about that later in this assignment.
Exercise: Implement initialize_parameters(). The dimensions for each group of filters are provided below. Reminder - to initialize a parameter $W$ of shape [1,2,3,4] in Tensorflow, use:
python
W = tf.get_variable("W", [1,2,3,4], initializer = ...)
More Info.
Can you write Python code for it?

# GRADED FUNCTION: initialize_parameters

def initialize_parameters():
    """
    Initializes weight parameters to build a neural network with tensorflow. The shapes are:
                        W1 : [4, 4, 3, 8]
                        W2 : [2, 2, 8, 16]
    Returns:
    parameters -- a dictionary of tensors containing W1, W2
    """

    tf.set_random_seed(1)                              # so that your "random" numbers match ours

    ### START CODE HERE ### (approx. 2 lines of code)
    W1 = tf.get_variable("W1", [4,4,3,8], initializer = tf.contrib.layers.xavier_initializer(seed = 0))
    W2 = tf.get_variable("W2", [2,2,8,16], initializer = tf.contrib.layers.xavier_initializer(seed = 0))
    ### END CODE HERE ###

    parameters = {"W1": W1,
                  "W2": W2}

    return parameters

tf.reset_default_graph()
with tf.Session() as sess_test:
    parameters = initialize_parameters()
    init = tf.global_variables_initializer()
    sess_test.run(init)
    print("W1 = " + str(parameters["W1"].eval()[1,1,1]))
    print("W2 = " + str(parameters["W2"].eval()[1,1,1]))


I am working on the file "Convolutional_Neural_Network/Convolution+model+-+Application+-+v1.ipynb".
The first task is:
Expected Output:

    Z3 =

    [[-0.44670227 -1.57208765 -1.53049231 -2.31013036 -1.29104376  0.46852064]

 [-0.17601591 -1.57972014 -1.4737016  -2.61672091 -1.00810647  0.5747785 ]]

1.3 - Compute cost
Implement the compute cost function below. You might find these two functions helpful:

tf.nn.softmax_cross_entropy_with_logits(logits = Z3, labels = Y): computes the softmax entropy loss. This function both computes the softmax activation function as well as the resulting loss. You can check the full documentation  here.
tf.reduce_mean: computes the mean of elements across dimensions of a tensor. Use this to sum the losses over all the examples to get the overall cost. You can check the full documentation here.

 Exercise: Compute the cost below using the function above.
Can you write Python code for it?

# GRADED FUNCTION: compute_cost

def compute_cost(Z3, Y):
    """
    Computes the cost

    Arguments:
    Z3 -- output of forward propagation (output of the last LINEAR unit), of shape (6, number of examples)
    Y -- "true" labels vector placeholder, same shape as Z3

    Returns:
    cost - Tensor of the cost function
    """

    ### START CODE HERE ### (1 line of code)
    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits = Z3, labels = Y))
    ### END CODE HERE ###

    return cost

tf.reset_default_graph()

with tf.Session() as sess:
    np.random.seed(1)
    X, Y = create_placeholders(64, 64, 3, 6)
    parameters = initialize_parameters()
    Z3 = forward_propagation(X, parameters)
    cost = compute_cost(Z3, Y)
    init = tf.global_variables_initializer()
    sess.run(init)
    a = sess.run(cost, {X: np.random.randn(4,64,64,3), Y: np.random.randn(4,6)})
    print("cost = " + str(a))


I am working on the file "Convolutional_Neural_Network/Convolution+model+-+Application+-+v1.ipynb".
The first task is:
Expected Output:
<table>
    <td>
    cost =
    </td>

    <td>
    2.91034
    </td>
</table>

1.4 Model
Finally you will merge the helper functions you implemented above to build a model. You will train it on the SIGNS dataset.
You have implemented random_mini_batches() in the Optimization programming assignment of course 2. Remember that this function returns a list of mini-batches.
Exercise: Complete the function below.
The model below should:

create placeholders
initialize parameters
forward propagate
compute the cost
create an optimizer

Finally you will create a session and run a for loop  for num_epochs, get the mini-batches, and then for each mini-batch you will optimize the function. Hint for initializing the variables
Can you write Python code for it?

# GRADED FUNCTION: model

def model(X_train, Y_train, X_test, Y_test, learning_rate = 0.009,
          num_epochs = 100, minibatch_size = 64, print_cost = True):
    """
    Implements a three-layer ConvNet in Tensorflow:
    CONV2D -> RELU -> MAXPOOL -> CONV2D -> RELU -> MAXPOOL -> FLATTEN -> FULLYCONNECTED

    Arguments:
    X_train -- training set, of shape (None, 64, 64, 3)
    Y_train -- test set, of shape (None, n_y = 6)
    X_test -- training set, of shape (None, 64, 64, 3)
    Y_test -- test set, of shape (None, n_y = 6)
    learning_rate -- learning rate of the optimization
    num_epochs -- number of epochs of the optimization loop
    minibatch_size -- size of a minibatch
    print_cost -- True to print the cost every 100 epochs

    Returns:
    train_accuracy -- real number, accuracy on the train set (X_train)
    test_accuracy -- real number, testing accuracy on the test set (X_test)
    parameters -- parameters learnt by the model. They can then be used to predict.
    """

    ops.reset_default_graph()                         # to be able to rerun the model without overwriting tf variables
    tf.set_random_seed(1)                             # to keep results consistent (tensorflow seed)
    seed = 3                                          # to keep results consistent (numpy seed)
    (m, n_H0, n_W0, n_C0) = X_train.shape
    n_y = Y_train.shape[1]
    costs = []                                        # To keep track of the cost

    # Create Placeholders of the correct shape
    ### START CODE HERE ### (1 line)
    X, Y = create_placeholders(n_H0, n_W0, n_C0, n_y)
    ### END CODE HERE ###

    # Initialize parameters
    ### START CODE HERE ### (1 line)
    parameters = initialize_parameters()
    ### END CODE HERE ###

    # Forward propagation: Build the forward propagation in the tensorflow graph
    ### START CODE HERE ### (1 line)
    Z3 = forward_propagation(X, parameters)
    ### END CODE HERE ###

    # Cost...


I am working on the file "Convolutional_Neural_Network/Convolution+model+-+Application+-+v1.ipynb".
The first task is:
Expected output: although it may not match perfectly, your expected output should be close to ours and your cost value should decrease.

    **Cost after epoch 0 =**

      1.917929

    **Cost after epoch 5 =**

      1.506757

    **Train Accuracy   =**

      0.940741

    **Test Accuracy   =**

      0.783333

Congratulations! You have finised the assignment and built a model that recognizes SIGN language with almost 80% accuracy on the test set. If you wish, feel free to play around with this dataset further. You can actually improve its accuracy by spending more time tuning the hyperparameters, or using regularization (as this model clearly has a high variance).
Once again, here's a thumbs up for your work!
Can you write Python code for it?

fname = "images/thumbs_up.jpg"
image = np.array(ndimage.imread(fname, flatten=False))
my_image = scipy.misc.imresize(image, size=(64,64))
plt.imshow(my_image)


I am working on the file "Convolutional_Neural_Network/Convolution+model+-+Application+-+v1.ipynb".
The first task is:
In Course 2, you had built a fully-connected network for this dataset. But since this is an image dataset, it is more natural to apply a ConvNet to it.
To get started, let's examine the shapes of your data.
Can you write Python code for it?

X_train = X_train_orig/255.
X_test = X_test_orig/255.
Y_train = convert_to_one_hot(Y_train_orig, 6).T
Y_test = convert_to_one_hot(Y_test_orig, 6).T
print ("number of training examples = " + str(X_train.shape[0]))
print ("number of test examples = " + str(X_test.shape[0]))
print ("X_train shape: " + str(X_train.shape))
print ("Y_train shape: " + str(Y_train.shape))
print ("X_test shape: " + str(X_test.shape))
print ("Y_test shape: " + str(Y_test.shape))
conv_layers = {}


I am working on the file "Convolutional_Neural_Network/Convolution+model+-+Application+-+v1.ipynb".
The first task is:
Run the following cell to train your model for 100 epochs. Check if your cost after epoch 0 and 5 matches our output. If not, stop the cell and go back to your code!
Can you write Python code for it?


I am working on the file "Convolutional_Neural_Network/Convolution+model+-+Application+-+v1.ipynb".
The first task is:
Run the next cell to load the "SIGNS" dataset you are going to use.
Can you write Python code for it?


I am working on the file "Convolutional_Neural_Network/Face+Recognition+for+the+Happy+House+-+v3.ipynb".
The first task is:
0 - Naive Face Verification
In Face Verification, you're given two images and you have to tell if they are of the same person. The simplest way to do this is to compare the two images pixel-by-pixel. If the distance between the raw images are less than a chosen threshold, it may be the same person!

   Figure 1

Of course, this algorithm performs really poorly, since the pixel values change dramatically due to variations in lighting, orientation of the person's face, even minor changes in head position, and so on.
You'll see that rather than using the raw image, you can learn an encoding $f(img)$ so that element-wise comparisons of this encoding gives more accurate judgements as to whether two pictures are of the same person.
1 - Encoding face images into a 128-dimensional vector
1.1 - Using an ConvNet  to compute encodings
The FaceNet model takes a lot of data and a long time to train. So following common practice in applied deep learning settings, let's just load weights that someone else has already trained. The network architecture follows the Inception model from Szegedy et al.. We have provided an inception network implementation. You can look in the file inception_blocks.py to see how it is implemented (do so by going to "File->Open..." at the top of the Jupyter notebook).
The key things you need to know are:

This network uses 96x96 dimensional RGB images as its input. Specifically, inputs a face image (or batch of $m$ face images) as a tensor of shape $(m, n_C, n_H, n_W) = (m, 3, 96, 96)$
It outputs a matrix of shape $(m, 128)$ that encodes each input face image into a 128-dimensional vector

Run the cell below to create the model for face images.
Can you write Python code for it?

FRmodel = faceRecoModel(input_shape=(3, 96, 96))

print("Total Params:", FRmodel.count_params())


I am working on the file "Convolutional_Neural_Network/Face+Recognition+for+the+Happy+House+-+v3.ipynb".
The first task is:
Expected Output

Total Params: 3743280

By using a 128-neuron fully connected layer as its last layer, the model ensures that the output is an encoding vector of size 128. You then use the encodings the compare two face images as follows:

   Figure 2:
   By computing a distance between two encodings and thresholding, you can determine if the two pictures represent the same person

So, an encoding is a good one if:
- The encodings of two images of the same person are quite similar to each other
- The encodings of two images of different persons are very different
The triplet loss function formalizes this, and tries to "push" the encodings of two images of the same person (Anchor and Positive) closer together, while "pulling" the encodings of two images of different persons (Anchor, Negative) further apart.

   Figure 3:
   In the next part, we will call the pictures from left to right: Anchor (A), Positive (P), Negative (N)

1.2 - The Triplet Loss
For an image $x$, we denote its encoding $f(x)$, where $f$ is the function computed by the neural network.

Training will use triplets of images $(A, P, N)$:

A is an "Anchor" image--a picture of a person.
P is a "Positive" image--a picture of the same person as the Anchor image.
N is a "Negative" image--a picture of a different person than the Anchor image.

These triplets are picked from our training dataset. We will write $(A^{(i)}, P^{(i)}, N^{(i)})$ to denote the $i$-th...
Can you write Python code for it?

# GRADED FUNCTION: triplet_loss

def triplet_loss(y_true, y_pred, alpha = 0.2):
    """
    Implementation of the triplet loss as defined by formula (3)

    Arguments:
    y_true -- true labels, required when you define a loss in Keras, you don't need it in this function.
    y_pred -- python list containing three objects:
            anchor -- the encodings for the anchor images, of shape (None, 128)
            positive -- the encodings for the positive images, of shape (None, 128)
            negative -- the encodings for the negative images, of shape (None, 128)

    Returns:
    loss -- real number, value of the loss
    """

    anchor, positive, negative = y_pred[0], y_pred[1], y_pred[2]

    ### START CODE HERE ### (≈ 4 lines)
    # Step 1: Compute the (encoding) distance between the anchor and the positive, you will need to sum over axis=-1
    pos_dist = tf.square(tf.subtract(anchor, positive))
    # Step 2: Compute the (encoding) distance between the anchor and the negative, you will need to sum over axis=-1
    neg_dist = tf.square(tf.subtract(anchor, negative))
    # Step 3: subtract the two previous distances and add alpha.
    basic_loss = tf.add(tf.reduce_sum(tf.subtract(pos_dist, neg_dist), axis=None), alpha)
    # Step 4: Take the maximum of basic_loss and 0.0. Sum over the training examples.
    loss = tf.reduce_sum(tf.maximum(basic_loss, 0))
    ### END CODE HERE ###

    return loss

with tf.Session() as test:
    tf.set_random_seed(1)
    y_true = (None, None, None)
    y_pred = (tf.random_normal([3, 128], mean=6, stddev=0.1, seed = 1),
              tf.random_normal([3, 128], mean=1, stddev=1, seed = 1),
              tf.random_normal([3, 128], mean=3, stddev=4, seed = 1))
    loss = triplet_loss(y_true, y_pred)

    print("loss = " + str(loss.eval()))


I am working on the file "Convolutional_Neural_Network/Face+Recognition+for+the+Happy+House+-+v3.ipynb".
The first task is:
Expected Output:

            **It's not kian, please go away**

           (0.86224014, False)

3.2 - Face Recognition
Your face verification system is mostly working well. But since Kian got his ID card stolen, when he came back to the house that evening he couldn't get in!
To reduce such shenanigans, you'd like to change your face verification system to a face recognition system. This way, no one has to carry an ID card anymore. An authorized person can just walk up to the house, and the front door will unlock for them!
You'll implement a face recognition system that takes as input an image, and figures out if it is one of the authorized persons (and if so, who). Unlike the previous face verification system, we will no longer get a person's name as another input.
Exercise: Implement who_is_it(). You will have to go through the following steps:
1. Compute the target encoding of the image from image_path
2. Find the encoding from the database that has smallest distance with the target encoding.
    - Initialize the min_dist variable to a large enough number (100). It will help you keep track of what is the closest encoding to the input's encoding.
    - Loop over the database dictionary's names and encodings. To loop use for (name, db_enc) in database.items().
        - Compute L2 distance between the target "encoding" and the current "encoding" from the database.
        - If this distance is less than the min_dist, then set min_dist to dist, and identity to name.
Can you write Python code for it?

# GRADED FUNCTION: who_is_it

def who_is_it(image_path, database, model):
    """
    Implements face recognition for the happy house by finding who is the person on the image_path image.

    Arguments:
    image_path -- path to an image
    database -- database containing image encodings along with the name of the person on the image
    model -- your Inception model instance in Keras

    Returns:
    min_dist -- the minimum distance between image_path encoding and the encodings from the database
    identity -- string, the name prediction for the person on image_path
    """

    ### START CODE HERE ###

    ## Step 1: Compute the target "encoding" for the image. Use img_to_encoding() see example above. ## (≈ 1 line)
    encoding = img_to_encoding(image_path, model)

    ## Step 2: Find the closest encoding ##

    # Initialize "min_dist" to a large value, say 100 (≈1 line)
    min_dist = 100

    # Loop over the database dictionary's names and encodings.
    for (name, db_enc) in database.items():

        # Compute L2 distance between the target "encoding" and the current "emb" from the database. (≈ 1 line)
        dist = np.linalg.norm(encoding - database[name])

        # If this distance is less than the min_dist, then set min_dist to dist, and identity to name. (≈ 3 lines)
        if dist<min_dist:
            min_dist = dist
            identity = name

    ### END CODE HERE ###

    if min_dist > 0.7:
        print("Not in the database.")
    else:
        print ("it's " + str(identity) + ", the distance is " + str(min_dist))

    return min_dist, identity


I am working on the file "Convolutional_Neural_Network/Face+Recognition+for+the+Happy+House+-+v3.ipynb".
The first task is:
Expected Output:

            **loss**

           528.143

2 - Loading the trained model
FaceNet is trained by minimizing the triplet loss. But since training requires a lot of data and a lot of computation, we won't train it from scratch here. Instead, we load a previously trained model. Load a model using the following cell; this might take a couple of minutes to run.
Can you write Python code for it?

FRmodel.compile(optimizer = 'adam', loss = triplet_loss, metrics = ['accuracy'])
load_weights_from_FaceNet(FRmodel)


I am working on the file "Convolutional_Neural_Network/Face+Recognition+for+the+Happy+House+-+v3.ipynb".
The first task is:
Now, when someone shows up at your front door and swipes their ID card (thus giving you their name), you can look up their encoding in the database, and use it to check if the person standing at the front door matches the name on the ID.
Exercise: Implement the verify() function which checks if the front-door camera picture (image_path) is actually the person called "identity". You will have to go through the following steps:
1. Compute the encoding of the image from image_path
2. Compute the distance about this encoding and the encoding of the identity image stored in the database
3. Open the door if the distance is less than 0.7, else do not open.
As presented above, you should use the L2 distance (np.linalg.norm). (Note: In this implementation, compare the L2 distance, not the square of the L2 distance, to the threshold 0.7.)
Can you write Python code for it?

# GRADED FUNCTION: verify

def verify(image_path, identity, database, model):
    """
    Function that verifies if the person on the "image_path" image is "identity".

    Arguments:
    image_path -- path to an image
    identity -- string, name of the person you'd like to verify the identity. Has to be a resident of the Happy house.
    database -- python dictionary mapping names of allowed people's names (strings) to their encodings (vectors).
    model -- your Inception model instance in Keras

    Returns:
    dist -- distance between the image_path and the image of "identity" in the database.
    door_open -- True, if the door should open. False otherwise.
    """

    ### START CODE HERE ###

    # Step 1: Compute the encoding for the image. Use img_to_encoding() see example above. (≈ 1 line)
    encoding = img_to_encoding(image_path, model)

    # Step 2: Compute distance with identity's image (≈ 1 line)
    dist = np.linalg.norm(encoding - database[identity])

    # Step 3: Open the door if dist < 0.7, else don't open (≈ 3 lines)
    if dist < 0.7 :
        print("It's " + str(identity) + ", welcome home!")
        door_open = True
    else:
        print("It's not " + str(identity) + ", please go away")
        door_open = False

    ### END CODE HERE ###
    print(encoding.shape, database[identity].shape)
    return dist, door_open


I am working on the file "Convolutional_Neural_Network/Face+Recognition+for+the+Happy+House+-+v3.ipynb".
The first task is:
Younes is at the front-door and the camera takes a picture of him ("images/camera_0.jpg"). Let's see if your who_it_is() algorithm identifies Younes.
Can you write Python code for it?


I am working on the file "Convolutional_Neural_Network/Keras+-+Tutorial+-+Happy+House+v2.ipynb".
The first task is:
Exercise: Implement step 2, i.e. compile the model to configure the learning process. Choose the 3 arguments of compile() wisely. Hint: the Happy Challenge is a binary classification problem.
Can you write Python code for it?

### START CODE HERE ### (1 line)
happyModel.compile(optimizer = "Adam",loss='binary_crossentropy', metrics = ["accuracy"])
### END CODE HERE ###


I am working on the file "Convolutional_Neural_Network/Keras+-+Tutorial+-+Happy+House+v2.ipynb".
The first task is:
If your happyModel() function worked, you should have observed much better than random-guessing (50%) accuracy on the train and test sets.
To give you a point of comparison, our model gets around 95% test accuracy in 40 epochs (and 99% train accuracy) with a mini batch size of 16 and "adam" optimizer. But our model gets decent accuracy after just 2-5 epochs, so if you're comparing different models you can also train a variety of models on just a few epochs and see how they compare.
If you have not yet achieved a very good accuracy (let's say more than 80%), here're some things you can play around with to try to achieve it:

Try using blocks of CONV->BATCHNORM->RELU such as:
python
X = Conv2D(32, (3, 3), strides = (1, 1), name = 'conv0')(X)
X = BatchNormalization(axis = 3, name = 'bn0')(X)
X = Activation('relu')(X)
until your height and width dimensions are quite low and your number of channels quite large (≈32 for example). You are encoding useful information in a volume with a lot of channels. You can then flatten the volume and use a fully-connected layer.
You can use MAXPOOL after such blocks. It will help you lower the dimension in height and width.
Change your optimizer. We find Adam works well.
If the model is struggling to run and you get memory issues, lower your batch_size (12 is usually a good compromise)
Run on more epochs, until you see the train accuracy plateauing.

Even if you have achieved a good accuracy, please feel free to keep playing with your model to try to get even better results.
Note: If you perform hyperparameter tuning on your model, the test set actually becomes a dev set, and your model might end up overfitting to the test (dev) set. But just for the purpose of this assignment, we won't worry about that here.
3 - Conclusion
Congratulations, you have solved the Happy House challenge!
Now, you just need to link this model to the front-door camera of your house. We unfortunately won't go into the details of how to do that here.
<font color='blue'>
What we would like you to...
Can you write Python code for it?

### START CODE HERE ###
img_path = 'images/image111.jpg.jpg'
### END CODE HERE ###
img = image.load_img(img_path, target_size=(64, 64))
imshow(img)

x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

print(happyModel.predict(x))


I am working on the file "Convolutional_Neural_Network/Keras+-+Tutorial+-+Happy+House+v2.ipynb".
The first task is:
Note that if you run fit() again, the model will continue to train with the parameters it has already learnt instead of reinitializing them.
Exercise: Implement step 4, i.e. test/evaluate the model.
Can you write Python code for it?

### START CODE HERE ### (1 line)
preds = happyModel.evaluate(X_test,Y_test, batch_size=32, verbose=0)
### END CODE HERE ###
print()
print ("Loss = " + str(preds[0]))
print ("Test Accuracy = " + str(preds[1]))


I am working on the file "Convolutional_Neural_Network/Keras+-+Tutorial+-+Happy+House+v2.ipynb".
The first task is:
Note: As you can see, we've imported a lot of functions from Keras. You can use them easily just by calling them directly in the notebook. Ex: X = Input(...) or X = ZeroPadding2D(...).
1 - The Happy House
For your next vacation, you decided to spend a week with five of your friends from school. It is a very convenient house with many things to do nearby. But the most important benefit is that everybody has commited to be happy when they are in the house. So anyone wanting to enter the house must prove their current state of happiness.

   Figure 1   : the Happy House

As a deep learning expert, to make sure the "Happy" rule is strictly applied, you are going to build an algorithm which that uses pictures from the front door camera to check if the person is happy or not. The door should open only if the person is happy.
You have gathered pictures of your friends and yourself, taken by the front-door camera. The dataset is labbeled.

Run the following code to normalize the dataset and learn about its shapes.
Can you write Python code for it?

X_train_orig, Y_train_orig, X_test_orig, Y_test_orig, classes = load_dataset()

# Normalize image vectors
X_train = X_train_orig/255.
X_test = X_test_orig/255.

# Reshape
Y_train = Y_train_orig.T
Y_test = Y_test_orig.T

print ("number of training examples = " + str(X_train.shape[0]))
print ("number of test examples = " + str(X_test.shape[0]))
print ("X_train shape: " + str(X_train.shape))
print ("Y_train shape: " + str(Y_train.shape))
print ("X_test shape: " + str(X_test.shape))
print ("Y_test shape: " + str(Y_test.shape))


I am working on the file "Convolutional_Neural_Network/Keras+-+Tutorial+-+Happy+House+v2.ipynb".
The first task is:
You have now built a function to describe your model. To train and test this model, there are four steps in Keras:
1. Create the model by calling the function above
2. Compile the model by calling model.compile(optimizer = "...", loss = "...", metrics = ["accuracy"])
3. Train the model on train data by calling model.fit(x = ..., y = ..., epochs = ..., batch_size = ...)
4. Test the model on test data by calling model.evaluate(x = ..., y = ...)
If you want to know more about model.compile(), model.fit(), model.evaluate() and their arguments, refer to the official Keras documentation.
Exercise: Implement step 1, i.e. create the model.
Can you write Python code for it?

### START CODE HERE ### (1 line)
happyModel = HappyModel(X_train.shape[1:])
### END CODE HERE ###


I am working on the file "Convolutional_Neural_Network/Residual+Networks+-+v2.ipynb".
The first task is:
1 - The problem of very deep neural networks
Last week, you built your first convolutional neural network. In recent years, neural networks have become deeper, with state-of-the-art networks going from just a few layers (e.g., AlexNet) to over a hundred layers.
The main benefit of a very deep network is that it can represent very complex functions. It can also learn features at many different levels of abstraction, from edges (at the lower layers) to very complex features (at the deeper layers). However, using a deeper network doesn't always help. A huge barrier to training them is vanishing gradients: very deep networks often have a gradient signal that goes to zero quickly, thus making gradient descent unbearably slow. More specifically, during gradient descent, as you backprop from the final layer back to the first layer, you are multiplying by the weight matrix on each step, and thus the gradient can decrease exponentially quickly to zero (or, in rare cases, grow exponentially quickly and "explode" to take very large values).
During training, you might therefore see the magnitude (or norm) of the gradient for the earlier layers descrease to zero very rapidly as training proceeds:

   Figure 1   : Vanishing gradient
 The speed of learning decreases very rapidly for the early layers as the network trains

You are now going to solve this problem by building a Residual Network!
2 - Building a Residual Network
In ResNets, a "shortcut" or a "skip connection" allows the gradient to be directly backpropagated to earlier layers:

   Figure 2   : A ResNet block showing a skip-connection

The image on the left shows the "main path" through the network. The image on the right adds a...
Can you write Python code for it?

# GRADED FUNCTION: identity_block

def identity_block(X, f, filters, stage, block):
    """
    Implementation of the identity block as defined in Figure 3

    Arguments:
    X -- input tensor of shape (m, n_H_prev, n_W_prev, n_C_prev)
    f -- integer, specifying the shape of the middle CONV's window for the main path
    filters -- python list of integers, defining the number of filters in the CONV layers of the main path
    stage -- integer, used to name the layers, depending on their position in the network
    block -- string/character, used to name the layers, depending on their position in the network

    Returns:
    X -- output of the identity block, tensor of shape (n_H, n_W, n_C)
    """

    # defining name basis
    conv_name_base = 'res' + str(stage) + block + '_branch'
    bn_name_base = 'bn' + str(stage) + block + '_branch'

    # Retrieve Filters
    F1, F2, F3 = filters

    # Save the input value. You'll need this later to add back to the main path.
    X_shortcut = X

    # First component of main path
    X = Conv2D(filters = F1, kernel_size = (1, 1), strides = (1,1), padding = 'valid', name = conv_name_base + '2a', kernel_initializer = glorot_uniform(seed=0))(X)
    X = BatchNormalization(axis = 3, name = bn_name_base + '2a')(X)
    X = Activation('relu')(X)

    ### START CODE HERE ###

    # Second component of main path (≈3 lines)
    X = Conv2D(filters = F2, kernel_size = (f, f), strides = (1,1), padding = 'same', name = conv_name_base + '2b', kernel_initializer = glorot_uniform(seed=0))(X)
    X = BatchNormalization(axis = 3, name = bn_name_base + '2b')(X)
    X = Activation('relu')(X)

    # Third component of main path (≈2 lines)
    X = Conv2D(filters = F3, kernel_size = (1, 1), strides = (1,1), padding = 'valid', name = conv_name_base + '2c', kernel_initializer = glorot_uniform(seed=0))(X)
    X = BatchNormalization(axis = 3, name = bn_name_base + '2c')(X)

    # Final step: Add shortcut value to main path, and pass it through a RELU...


I am working on the file "Convolutional_Neural_Network/Residual+Networks+-+v2.ipynb".
The first task is:
Expected Output:

            ** Epoch 1/2**

           loss: between 1 and 5, acc: between 0.2 and 0.5, although your results can be different from ours.

            ** Epoch 2/2**

           loss: between 1 and 5, acc: between 0.2 and 0.5, you should see your loss decreasing and the accuracy increasing.

Let's see how this model (trained on only two epochs) performs on the test set.
Can you write Python code for it?

preds = model.evaluate(X_test, Y_test)
print ("Loss = " + str(preds[0]))
print ("Test Accuracy = " + str(preds[1]))


I am working on the file "Convolutional_Neural_Network/Residual+Networks+-+v2.ipynb".
The first task is:
Expected Output:

            **Test Accuracy**

           between 0.16 and 0.25

For the purpose of this assignment, we've asked you to train the model only for two epochs. You can see that it achieves poor performances. Please go ahead and submit your assignment; to check correctness, the online grader will run your code only for a small number of epochs as well.
After you have finished this official (graded) part of this assignment, you can also optionally train the ResNet for more iterations, if you want. We get a lot better performance when we train for ~20 epochs, but this will take more than an hour when training on a CPU.
Using a GPU, we've trained our own ResNet50 model's weights on the SIGNS dataset. You can load and run our trained model on the test set in the cells below. It may take ≈1min to load the model.
Can you write Python code for it?

model = load_model('ResNet50.h5')

preds = model.evaluate(X_test, Y_test)
print ("Loss = " + str(preds[0]))
print ("Test Accuracy = " + str(preds[1]))


I am working on the file "Convolutional_Neural_Network/Residual+Networks+-+v2.ipynb".
The first task is:
ResNet50 is a powerful model for image classification when it is trained for an adequate number of iterations. We hope you can use what you've learnt and apply it to your own classification problem to perform state-of-the-art accuracy.
Congratulations on finishing this assignment! You've now implemented a state-of-the-art image classification system!
4 - Test on your own image (Optional/Ungraded)
If you wish, you can also take a picture of your own hand and see the output of the model. To do this:
    1. Click on "File" in the upper bar of this notebook, then click "Open" to go on your Coursera Hub.
    2. Add your image to this Jupyter Notebook's directory, in the "images" folder
    3. Write your image's name in the following code
    4. Run the code and check if the algorithm is right!
Can you write Python code for it?

img_path = 'images/image222.jpg'
img = image.load_img(img_path, target_size=(64, 64))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)
print('Input image shape:', x.shape)
my_image = scipy.misc.imread(img_path)
imshow(my_image)
print("class prediction vector [p(0), p(1), p(2), p(3), p(4), p(5)] = ")
print(model.predict(x))


I am working on the file "Convolutional_Neural_Network/Residual+Networks+-+v2.ipynb".
The first task is:
You can also print a summary of your model by running the following code.
Can you write Python code for it?


I am working on the file "Convolutions_Sidebar.ipynb".
The first task is:
Next, we can use the pyplot library to draw the image so we know what it looks like.
Can you write Python code for it?

import matplotlib.pyplot as plt
plt.grid(False)
plt.gray()
plt.axis('off')
plt.imshow(i)
plt.show()


I am working on the file "Convolutions_Sidebar.ipynb".
The first task is:
Now we can plot the image to see the effect of the convolution!
Can you write Python code for it?

# Plot the image. Note the size of the axes -- they are 512 by 512
plt.gray()
plt.grid(False)
plt.imshow(i_transformed)
#plt.axis('off')
plt.show()


I am working on the file "Convolutions_Sidebar.ipynb".
The first task is:
This code will show a (2, 2) pooling. The idea here is to iterate over the image, and look at the pixel and it's immediate neighbors to the right, beneath, and right-beneath. Take the largest of them and load it into the new image. Thus the new image will be 1/4 the size of the old -- with the dimensions on X and Y being halved by this process. You'll see that the features get maintained despite this compression!
Can you write Python code for it?

new_x = int(size_x/2)
new_y = int(size_y/2)
newImage = np.zeros((new_x, new_y))
for x in range(0, size_x, 2):
  for y in range(0, size_y, 2):
    pixels = []
    pixels.append(i_transformed[x, y])
    pixels.append(i_transformed[x+1, y])
    pixels.append(i_transformed[x, y+1])
    pixels.append(i_transformed[x+1, y+1])
    newImage[int(x/2),int(y/2)] = max(pixels)

# Plot the image. Note the size of the axes -- now 256 pixels instead of 512
plt.gray()
plt.grid(False)
plt.imshow(newImage)
#plt.axis('off')
plt.show()


I am working on the file "Coordinate_geometry_in_the_(xy)_plane/Coordinate_geometry_in_the_xy_plane.ipynb".
The first task is:
Contents
1.Straight Lines
- Equations
- Line conditions
- Mid points
- Distance between points
2.Coordinate Geometry of the Circle
- Radius equation
- Circle equation
- Rules
3.Parametric Equations
- Circle
- Parabola
- Ellipse
- Rectangular hyperbola
- Conversion between Cartesian and parametric forms

Straight Lines

Equations
Main Equation:
$y = Mx + c$
Can be re-formated though:
$ax + by + c = 0$
Can find equation of line given known point and gradient
Example:
x = 3
y = 6
M = 4
$6 = 4\times 3 + c$
$c = -6$
$y = 4x - 6$
Gradient Equation:
$M = \frac{y_2 - y_1}{x_2 - x_1}$ or $M = \frac{dy}{dx}$
Can be used to find gradient from 2 known points (and used to find equation after)
Example
$x_1 = 2$
$y_1 = -1$
$x_2 = 7$
$y_2 = -15$
$M = \frac{-15 + 1}{7 - 2}$
$M = \frac{-14}{5}$
$M = -2\frac{4}{5}$
We can also use this re-formated equation to find the full equation of the line
$(y - y_1) = M(x - x_1)$
With $x_1$ and $y_1$ and $M$ as our known variables
Example
$(y - 6) = 4(x - 3)$
$y = 4x - 6$
Can you write Python code for it?

# x data
x = np.linspace(-10, 10, 2001).astype(np.float32)

# variables
x1 = 2
x2 = 7
y1 = -1
y2 = -15

# gradient function
def grad(x1, x2, y1, y2):
    return((y2-y1)/(x2-x1))

# line function
def f(y1, x1, x, M):
    c = y1 - M*x1
    return(M*x + c, c)

# get the gradient
M = grad(x1, x2, y1, y2)

# get the y values
# also get the c value to use in our plot key
y, c = f(y1, x1, x, M)

# set up plot
fig, ax = plt.subplots(figsize=(8,4))

ax.plot(x,y, label='y = %sx + %s' % (M, c))

ax.legend()
ax.grid(True)

plt.show()


I am working on the file "Copia_de_Copia_de_Preprocesamiento_y_Red_test.ipynb".
The first task is:
Al parecer este bloque no hace nada a las variables x_train y similares
Can you write Python code for it?

x_train = x_train.astype('float32')
x_val = x_val.astype('float32')
x_train /= 255
x_val /= 255
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_val.shape[0], 'test samples')


I am working on the file "Copia_de_Copia_de_Preprocesamiento_y_Red_test.ipynb".
The first task is:
Balance de Clases, definir conjuntos de train, val, y test
Can you write Python code for it?

"Objetos a tratar"

" Clases = {0: 'AGN', 1:'SN', 2:'VS', 3:'asteroid', 4:'bogus'}"
"AGN: Active Galactic Nuclei"
"SN: Supernova"
"VS: Variable Star"
"bogus: Artifacts"

img_no_nan # imagenes, np array
label_sq # labels, np array

unique, counts = np.unique(label_sq, return_counts=True)
class_counter = dict(zip(unique, counts)) # muestra cuantas veces se repite cada clase
print('Numero de muestras por clase:', class_counter)

#test = np.concatenate((img_no_nan, label_sq), axis=0)
#test_1 = np.concatenate((img_no_nan, label_sq), axis=1)

s_index = np.arange(label_sq.shape[0]) # indexar arreglos

np.random.shuffle(s_index) # revolver

# label_sq.shape[0], label_sq.shape

img_no_nan = img_no_nan[s_index] # reemplazar arreglos originales por arreglos revueltos
label_sq = label_sq[s_index]

"Encontrar ubicaciones de todos los tipos de objeto"

agn_ind = np.squeeze(np.argwhere(label_sq == 0))
sn_ind = np.squeeze(np.argwhere(label_sq == 1))
vs_ind = np.squeeze(np.argwhere(label_sq == 2))
ast_ind = np.squeeze(np.argwhere(label_sq == 3))
bog_ind = np.squeeze(np.argwhere(label_sq == 4))

"Una lista para cada tipo de objeto"

img_AGN = img_no_nan[agn_ind]
img_SN = img_no_nan[sn_ind]
img_VS = img_no_nan[vs_ind]
img_AST = img_no_nan[ast_ind]
img_bog = img_no_nan[bog_ind]

# Separación de la base de datos en un conjunto de entrenamiento y validación.
train_size = int(len(img_no_nan)*.75) # Definiendo el tamaño del conjunto de entrenamiento con los requisitos solicitados
train_total = int(len(img_no_nan)) # Tamaño total del conjunto original.
x_train = img_no_nan[0:train_size,:] # Nuevo conjunto horizontal de entrenamiento.
x_val = img_no_nan[train_size:train_total,:] # Conjunto de validacion horizontal. (imagenes)
y_train = y_data[:train_size] # Nuevo conjunto vertical de entrenamiento. (clases)
y_val = y_data[train_size:] # Conjunto de validacion vertical (clases)

input_shape = (img_rows, img_cols, 3)


I am working on the file "Copia_de_Copia_de_Preprocesamiento_y_Red_test.ipynb".
The first task is:
Cargar base de datos
Can you write Python code for it?

# Cargando base de datos Alerce.
data = pd.read_pickle('/content/drive/My Drive/Supernova/ALERCE_stamps.pkl') # Base de datos original
#data = pd.read_pickle('/content/drive/My Drive/Supernova/ALERCE_stamps_1k.pkl') # Base de datos reducida

imgs = data['images'] # Lista de 31727 imagenes
labels = data['labels'] # Lista de 31727 labels

img_0 = imgs[0] # Primera imagen de imgs, ndarray 63x63, 3 canales
label_0 = labels[0] # Primer label, entero

np.shape(imgs)


I am working on the file "Copia_de_Copia_de_Preprocesamiento_y_Red_test.ipynb".
The first task is:
Configurar Red!
Can you write Python code for it?

""" PROGRAMACIÓN DE LA RED NEURONAL CONVOLUCIONAL!!
    En los siguientes bloques se configura una CNN usando la estructura entregada en el paper.
"""

"Convierten a one hot encoding"
y_train_1 = keras.utils.to_categorical(y_train, num_classes) # Categorización de las muestras del conjunto vertical de entrenamiento.
y_val_1 = keras.utils.to_categorical(y_val, num_classes) # Categorización de las muestras del conjunto vertical de prueba.
"Convierten a one hot encoding"

time_i = time()
model = Sequential() # Inicio de la configuracion del clasificador , a partir de este punto se le añadiran las capas que conformaran la CNN
model.add(Conv2D(32, kernel_size=(4, 4), # Añadiendo una capa convolucional al modelo con un kernel  de 3x3
                 activation = None, # Con un funcion de activacion relu  y las dimensiones configuradas en el bloque anterior
                 input_shape=input_shape))
model.add(Conv2D(32, (3, 3), activation = None)) # Añadiendo otra capa convolucional  con una funcion de activacion relu
model.add(MaxPooling2D(pool_size=(2, 2),strides = 2)) # Utilizando el método maxpooling en esta red.
model.add(Conv2D(64, (3, 3), activation = None))
model.add(Conv2D(64, (3, 3), activation = None))
model.add(Conv2D(64, (3, 3), activation = None))
model.add(MaxPooling2D(pool_size=(2, 2), strides = 2))
#model.add(Dropout(0.5)) # Añadiendo Dropout a la red
model.add(Flatten()) # Este comando remueve todas las dimensiones de un tensor exceptuano una.
#model.add(Dense(128, activation=LeakyReLU)) # En esta capa se implementa la funcion de activacion relu a la capa anterior de la la red neuronal.
model.add(Dense(64)) # 64 salidas
model.add(LeakyReLU()) # LeakyReLU
model.add(Dropout(0.5)) # Se agrega la tecnica de Dropout para abordar el overfitting.
model.add(Dense(64))
model.add(LeakyReLU()) # LeakyReLU
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax')) # Se aplica a la red siendo esta vez usando la funcion softmax.

model.compile(loss=keras.losses.categorical_crossentropy,
  ...


I am working on the file "Copia_de_Copia_de_Preprocesamiento_y_Red_test.ipynb".
The first task is:
Desechar imágenes no cuadradas
Can you write Python code for it?

shap_def = (63, 63, 3) # Estructura correcta de imagenes, cuadradas, 3 canales

img_sq = [] # Inicializar lista vacia para almacenar imagenes que tienen
            # estructura correcta
label_sq = [] # Inicializar lista vacia para almacenar labels de imagenes que
              # tienen estructura correcta

for j in list(range(len(imgs))): # recorrer imagenes
    #print('Revisando si la imagen '+str(j)+' es cuadrada')
    if np.shape(imgs[j]) == shap_def: # si tiene estructura correcta, agregar

        img_sq.append(imgs[j])
        label_sq.append(labels[j])


I am working on the file "Copia_de_Copia_de_Preprocesamiento_y_Red_test.ipynb".
The first task is:
Entrenamiento
Can you write Python code for it?

# Entrenamiento del modelo

model_train = model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_val, y_val)) # Comando para entrenar el modelo.
model_train
score = model.evaluate(x_val, y_val, verbose=0) # Con este comando se obtiene el valor de perdida y de las metricas en el modo de prueba.
time_f = time()
time_execution = time_f - time_i # Con este comando se obtiene el valor de perdida y de las metricas en el modo de prueba.
print('Valid loss:', score[0]) # Se imprime la perdida para la prueba realizada.
print('Valid accuracy:', score[1]) # Se imprimre el accuracy para la prueba realizada.
print('Tiempo de entrenamiento ' + str(time_execution))

import numpy as np

x = np.array([0, 3, 1, 2, 4, 1, 0, 4])
# x = np.reshape(x, (4, 25))
print(x)

ind = np.squeeze(np.argwhere(x%7 == 0))
print(ind)

#y = np.squeeze(x[ind])
y = x[ind]
print(y)

z = keras.utils.to_categorical(x, 5)

print(z)


I am working on the file "Copia_de_Copia_de_Preprocesamiento_y_Red_test.ipynb".
The first task is:
Importar librerías
Can you write Python code for it?

import numpy as np
import pandas as pd
import matplotlib
#%matplotlib inline
import matplotlib.pyplot as plt

import matplotlib.cm as cm
from skimage import io, img_as_float
import scipy.interpolate as itp

import random

from __future__ import print_function
from time import time

from sklearn import metrics
from sklearn.cluster import KMeans
from sklearn.cluster import DBSCAN
from sklearn.cluster import AgglomerativeClustering
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.preprocessing import scale

from warnings import simplefilter

simplefilter(action='ignore', category=FutureWarning) # Comando para ignorar los warnings

import keras  # Importando la libreria keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten # Comandos de keras que importan las capas dense, dropout y flatten
from keras.layers import Conv2D, MaxPooling2D, ZeroPadding2D # Comandos de keras que importan las capas convolucionales y maxpooling2D
from keras import backend as K
from keras.layers.advanced_activations import LeakyReLU


I am working on the file "Copia_de_Copia_de_Preprocesamiento_y_Red_test.ipynb".
The first task is:
Normalizar
Can you write Python code for it?

img_sq = np.asarray(img_sq)
label_sq = np.asarray(label_sq)

# Funcion auxiliar que normaliza UNA matriz
def Norm_Data_matrix(matrix):
    max_data = np.nanmax(matrix)
    min_data = np.nanmin(matrix)
    Norm_matrix = (matrix-min_data)/(max_data-min_data)
    return Norm_matrix

# Normalizacion de las imagenes
img_norm = []
for i in range(len(img_sq)):
    muestra = img_sq[i]
    #print('Normalizando muestra ' + str(i))
    muestra[:, :, 0] = Norm_Data_matrix(muestra[:, :, 0])
    muestra[:, :, 1] = Norm_Data_matrix(muestra[:, :, 1])
    muestra[:, :, 2] = Norm_Data_matrix(muestra[:, :, 2])
    img_norm.append(muestra)

img_norm = np.array(img_norm) # Convertir a arreglo numpy

nan_index = np.argwhere(np.isnan(img_norm))
#print('Indices de nan en arreglo normalizado', nan_index)

img_no_nan = np.nan_to_num(img_norm) # Convertir nan a cero

nonan_index = np.argwhere(np.isnan(img_no_nan))
#print('Indices de nan en arreglo al eliminar nan', nonan_index)

np.shape(img_no_nan) # Dimensiones de la base corregida


I am working on the file "Copia_de_Copia_de_Preprocesamiento_y_Red_test.ipynb".
The first task is:
Red Neuronal
Preliminares
Can you write Python code for it?

# Para poder procesar las clases es necesario pasarlas a one hot encoding

# Funcion para tener las clases como one hot encoding
def one_hot(a, num_classes):
    return np.squeeze(np.eye(num_classes)[a.reshape(-1)])

onehot_labels = one_hot(label_sq, 5)

batch_size = 128 # Definiendo el tamaño de los batches
num_classes = 5 # Numero de clases del conjunto
epochs = 12 # Cantidad de epocas
#pool_value = 0.5

# input image dimensions
img_rows, img_cols = 63, 63  # Definiendo las filas y columnas de las imagenes
y_data = onehot_labels

np.shape(img_no_nan)


I am working on the file "Copula Packages Notebook.ipynb".
The first task is:

Independent (unrelated) data

We will draw samples from a beta distribution for (x), and samples from a lognormal for (y). The samples are pseudo-independent (we know there is no real independence if you use a computer to draw the samples, but well, reasonably independent).

Can you write Python code for it?

#unrelated data: a beta (x) and a lognormal (y)
a= 0.45#2. #alpha
b=0.25#5. #beta
x=beta.rvs(a,b,loc=loc,size=sz)

#plot unrelated x and y
t = np.linspace(0, lognorm.ppf(0.99, sc), sz)
plt.plot(t, beta.pdf(t,a,b), lw=5, alpha=0.6, label='x:beta')
plt.plot(t, lognorm.pdf(t, sc), lw=5, alpha=0.6, label='y:lognormal')
plt.legend()

#plot copulas built from unrelated x and y
title='Copulas from unrelated data x: beta, alpha {} beta {}, y: lognormal, mu {}, sigma {}'.format(a,b,loc,sc)
generateAndPlot(title,x,y,pseudoobs)


I am working on the file "Copula Packages Notebook.ipynb".
The first task is:

Note: at that point I was lazily pasting the structure for some plotting ;).

Testing the package

The first samples (x) are generated from a beta distribution and (y) from a lognormal. Beta distributions have finite bounds for support, while the right-side support for lognormals is in the infinity. An interesting property ofcopulas: BOTH marginals are transformed to the unit range. Isn't that cool?

To try the package, we fitted copulas of three archimedean families (Frank, Clayton, Gumbel) to samples x and y. We then extracted some samples from the fitted copulas, and plotted the sampled outputs together with the original samples, in order to look at how they compare with each other.

feel free to try it out on you own, and do comment if you feel like!

Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
from copulalib.copulalib import Copula as copula
from scipy.stats import beta, lognorm, gaussian_kde

plt.style.use('ggplot')

def ppf(var,q):
    #the equivalent of ppf but built directly from data
    sortedvar=np.sort(var)
    rcdf=np.linspace(0,1, num=len(var))
    returnable=np.array([sortedvar[np.where(rcdf<=qx)[0][-1]] for qx in q])
    #print returnable.shape
    return returnable

def generateAndPlot(title,x,y,howmany):
    #generate and plot
    fig, axs = plt.subplots(3,2,figsize=(12., 18.))
    fig.suptitle(title)
    for index,family in enumerate(['frank','clayton','gumbel']):
        copula_f = copula(x,y,family=family)
        try:
            #get pseudo observations
            u,v = copula_f.generate_uv(howmany)
        except Exception as exx:
            print "Could not extract pseudo observations for {}\
                   because {}".format(family, exx.message)
            continue
        #plot pseudo observations
        axs[index][0].scatter(u,v,marker='o',alpha=0.7)
        axs[index][0].set_title('pseudo observations {}'.format(family),
                                 fontsize=12)
        #plot inverse ppf
        #ux = beta.ppf(u,a,b,loc=loc)
        #vy= lognorm.ppf(v,sc,loc=loc)
        ux=ppf(x,u)
        vy=ppf(y,v)
        axs[index][1].scatter(x,y,marker='*',color='r', alpha=0.5) #original samples
        axs[index][1].scatter(ux,vy,marker='s', alpha=0.3) #simulated realisations
        axs[index][1].set_title('simulated (blue) vs original (red) {}'.format(family),
                                fontsize=12)

    plt.show()

#total samples vs pseudo observations
pseudoobs=100
sz=300
loc=0.0 #needed for most distributions
#lognormal param
sc=0.5
y=lognorm.rvs(sc,loc=loc, size=sz)


I am working on the file "Copy_of_Hello,_Colaboratory.ipynb".
The first task is:
Forms
Forms can be used to parameterize code. See the forms example notebook for more details.
Can you write Python code for it?

#@title Examples

text = 'value' #@param
date_input = '2018-03-22' #@param {type:"date"}
number_slider = 0 #@param {type:"slider", min:-1, max:1, step:0.1}
dropdown = '1st option' #@param ["1st option", "2nd option", "3rd option"]


I am working on the file "Copy_of_Hello,_Colaboratory.ipynb".
The first task is:
GitHub
You can save a copy of your Colab notebook to Github by using File > Save a copy to GitHub…
You can load any .ipynb on GitHub by just adding the path to colab.research.google.com/github/ . For example, colab.research.google.com/github/tensorflow/models/blob/master/samples/core/get_started/_index.ipynb will load this .ipynb on GitHub.
Visualization
Colaboratory includes widely used libraries like matplotlib, simplifying visualization.
Can you write Python code for it?

import matplotlib.pyplot as plt
import numpy as np

x = np.arange(20)
y = [x_i + np.random.randn(1) for x_i in x]
a, b = np.polyfit(x, y, 1)
_ = plt.plot(x, y, 'o', np.arange(20), a*np.arange(20)+b, '-')


I am working on the file "Copy_of_Hello,_Colaboratory.ipynb".
The first task is:
Want to use a new library?  pip install it at the top of the notebook. Then that library can be used anywhere else in the notebook. For recipes to import commonly used libraries, refer to the importing libraries example notebook.
Can you write Python code for it?

!pip install -q matplotlib-venn

from matplotlib_venn import venn2
_ = venn2(subsets = (3, 2, 1))


I am working on the file "Copy_of_Introduction_to_CNNs_Handout.ipynb".
The first task is:
Again create an instance of your model and transfer it to the GPU (using the nn.Module.cuda method).
Now chose your optimizer. Take a look at http://pytorch.org/docs/master/optim.html#algorithms for a list of choices, but if you need a recomendation use torch.optim.SGD
Next we need to pick a loss function.  Here is a list http://pytorch.org/docs/master/nn.html#id46
If you are not sure what to choose then use torch.nn.functional.nll_loss which is negative log likelihood loss ($L(y) = -log(y)$) a common function to use with softmax
Can you write Python code for it?

mlp_model = MLP()
mlp_model.cuda()
# Change this to whatever optimizer you want to try
mlp_optimizer = optim.SGD(mlp_model.parameters(), lr=LEARNING_RATE, momentum=SGD_MOMENTUM)
print(mlp_model)
#Change the last argument to whatever loss function you want to try
MLP_TRAINING_LOSS, MLP_TEST_LOSS, MLP_TEST_ACCURACY = learn(mlp_model, mlp_optimizer, F.nll_loss)


I am working on the file "Copy_of_Introduction_to_CNNs_Handout.ipynb".
The first task is:
But what if I give you this image?
Can you write Python code for it?

img, label = next(iter(test_loader))
img = img[0][0]
label = label[0]
img_transpose = img.transpose(0,1)
img_view = img_transpose.cpu().numpy()
plt.imshow(img_view.reshape(28,28))
print("It is supposed to be a " + str(label))

print("The model thinks it is a " + str(classify(cnn_model, img_transpose.unsqueeze(0).unsqueeze(0))[0]))


I am working on the file "Copy_of_Introduction_to_CNNs_Handout.ipynb".
The first task is:
Congratulations!
You just trained 3 different machine learning models and all of them are relatively good at detecting handwritting. There is a big world of models, tasks and strategies to explore. But one more thing before we finish this tutorial...
Limits of your model
You trained a model, it gets something like 98% accuracy on your testing dataset. But is it ready to go out into the world and classify digits?
Lets take a look at an example again...
Can you write Python code for it?


I am working on the file "Copy_of_Introduction_to_CNNs_Handout.ipynb".
The first task is:
Creating the Learning Pipeline
We are going now to layout all the systems to manage training and testing our model
The Training Pipeline
First we are going to define our training pipeline. This is the order of steps in which you go though to train your model on how to complete its task.
First we start with setting the model to train mode. This allows the parameters in the model to change. This function represents 1 training epoch. An epoch is one full interation through the data set. We use the training dataset we created earlier to generate batches of data and their associated true labels.
For each batch we go through the following steps:

Move our data to our GPU
Declare our data as Variables in the network
Zero all the gradients in the network
Run our data through the network and get the predictions
Using out chosen loss function, calculate the loss function
Calcuate the gradients $\nabla_W$ of the loss with respect to the weights at each node in the network
Update the weights in the network using $W_{t+1} = W_t - \alpha \nabla_W\mathcal L$
Can you write Python code for it?

def train(model, optimizer, loss_func, epoch, training_history):
    model.train()                                            #Set training mode
    for batch, (data, target) in enumerate(train_loader):
        data, target = data.cuda(), target.cuda()            #Transfer data and correct result to the GPU
        data, target = Variable(data), Variable(target)      #Make data and correct answers variables in the Network
        optimizer.zero_grad()                                #zero the gradients
        output = model(data)                                 #classify the data
        loss = loss_func(output, target)                     #calculate the loss
        loss.backward()                                      #propogate the weight updates through the network
        optimizer.step()
        if batch % LOG_INTERVAL == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(epoch, batch * len(data), len(train_loader.dataset), 100. * batch / len(train_loader), loss.data[0]))
            training_history.append(((len(train_loader.dataset) * epoch) + batch * len(data), loss.data[0]))


I am working on the file "Copy_of_Introduction_to_CNNs_Handout.ipynb".
The first task is:
Helpful Utilities
The following function is a function to visualize the learning process, plotting the training loss, testing loss and testing accuracy over time.
Can you write Python code for it?

def visualize_learning(training_loss, test_loss, test_accuracy):
    f1 = plt.figure()
    f2 = plt.figure()
    f3 = plt.figure()
    ax1 = f1.add_subplot(111)
    ax2 = f2.add_subplot(111)
    ax3 = f3.add_subplot(111)

    training_loss_batch, training_loss_values = zip(*training_loss)
    ax1.plot(training_loss_batch, training_loss_values)
    ax1.set_title('Training Loss')
    ax1.set_xlabel("Batch")
    ax1.set_ylabel("Loss")

    test_loss_epoch, test_loss_values = zip(*test_loss)
    ax2.plot(test_loss_epoch, test_loss_values)
    ax2.set_title("Testing Loss")
    ax2.set_xlabel("Batch")
    ax2.set_ylabel("Loss")

    test_accuracy_epoch, test_accuracy_values = zip(*test_accuracy)
    ax3.plot(test_accuracy_epoch, test_accuracy_values)
    ax3.set_title("Testing Accuracy")
    ax3.set_xlabel("Batch")
    ax3.set_ylabel("Accuracy (%)")

    plt.show()


I am working on the file "Copy_of_Introduction_to_CNNs_Handout.ipynb".
The first task is:
Implementing Multilayer Perceptron
Next we are going to implement a mulilayer perceptron. Each layer weights the importance of the data from the previous layer, eventually terminating in the same sigmoid function to convert the representation vector of the input data into a probablity. Between each layer we now include a non-linearity or activation function, a function that allows the model to approximate non linear functions and filter data between layers.
Each layer of the MLP now looks like this $relu(w^Tx +b)$ and the full model becomes $\sigma(w^T relu(w^T relu(w^Tx + b)+ b) + b)$
Again we train this model with gradient decent
Defining the Model
Once again we are going to create a sub class of the pytorch class torch.nn.Module listing out the layers of our model in our __init__ method and the operations between those layers in the method called forward. The torch.nn.Module super class handles the implementation of the backpropogation.
Go a head and try to fill in what you think a Multilayer Perceptron Model would look like.
Here are some useful functions:

torch.nn.Linear -> Fully Connected Layer / Impliments $F(\boldsymbol x) = \boldsymbol w^T \boldsymbol x + \boldsymbol b$
torch.nn.Functional.relu -> Rectifying Linear Unit / Implements $max(0,x)$
Other activation functions can be found here: http://pytorch.org/docs/master/nn.html#non-linear-activation-functions

torch.nn.Functional.softmax -> Softmax function / Implements $F(x) = \frac{e^{ \boldsymbol x}} {\sum_\limits{i \in dim \boldsymbol x} e^{\boldsymbol x_i} }$
torch.nn.Functional.log_softmax -> Log of the softmax function / Implements $F(x) = log(\frac{e^{ \boldsymbol x}} {\sum_\limits{i \in dim \boldsymbol x} e^{\boldsymbol x_i} })$
torch.Tensor.view -> allows you to reshape a Tensor (multi-dimentional vector)
Can you write Python code for it?

class MLP(nn.Module):
    def __init__(self):
        super(MLP, self).__init__()
        '''
        INSERT YOUR MODEL COMPONENTS HERE
        '''
        return x
    def forward(self, x):
        '''
        LINK YOUR COMPONENTS HERE
        '''
        return x


I am working on the file "Copy_of_Introduction_to_CNNs_Handout.ipynb".
The first task is:
Implementing Sofmax Regression
Our first model of the day is going to be softmax regression. This model can be summed up as simultaneously calculating the probablity that an image is any of the 10 classes at the same time using a set of parameters that represent the importance of particular pixels in the image.
We use the function $P(y' = {1..10}) = \sigma(w^Tx + b)$ where $\sigma = \frac{e^{ \boldsymbol x}} {\sum_\limits{i \in dim \boldsymbol x} e^{\boldsymbol x_i}}$ to calculate the probablity and we use gradient decent to find the weights.
Defining the Model
Pytorch has a super easy way to define machine learning models. We create a sub class of the pytorch class torch.nn.Module listing out the layers of our model in our __init__ method and the operations between those layers in the method called forward. The torch.nn.Module super class handles the implementation of the backpropogation.
Go a head and try to fill in what you think a SoftmaxRegression Model would look like.
Here are some useful functions:

torch.nn.Linear -> Fully Connected Layer / Impliments $F(\boldsymbol x) = \boldsymbol w^T \boldsymbol x + \boldsymbol b$
torch.nn.Functional.softmax -> Softmax function / Implements $F(x) = \frac{e^{ \boldsymbol x}} {\sum_\limits{i \in dim \boldsymbol x} e^{\boldsymbol x_i} }$
torch.nn.Functional.log_softmax -> Log of the softmax function / Implements $F(x) = log(\frac{e^{ \boldsymbol x}} {\sum_\limits{i \in dim \boldsymbol x} e^{\boldsymbol x_i} })$
torch.Tensor.view -> allows you to reshape a Tensor (multi-dimentional vector)
Can you write Python code for it?

class SoftmaxRegression(nn.Module):
    def __init__(self):
        super(SoftmaxRegression, self).__init__()
        '''
        INSERT YOUR MODEL COMPONENTS HERE
        '''
    def forward(self, x):
        '''
        LINK YOUR COMPONENTS HERE
        '''
        return x


I am working on the file "Copy_of_Introduction_to_CNNs_Handout.ipynb".
The first task is:
Loading Helpful Packages
First thing in any app is to include all of the dependencies for the project.

torch (PyTorch) and its specific sub-libraries - This handles all of our neural network setup, training and testing
torchvision - This is a specific sub-library for PyTorch for computer vision tasks
numpy - Generic matrix library for python
matplotlib and seaborn - graphing libraries for python
Can you write Python code for it?

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.autograd import Variable
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns


I am working on the file "Copy_of_Introduction_to_CNNs_Handout.ipynb".
The first task is:
Now we can create an instance of our model and transfer it to the GPU (using the nn.Module.cuda method).
Now chose your optimizer, there are a bunch of choices (the most common one is Stocastic Gradient Decent, but others include ADAM, ADA and more). Take a look at http://pytorch.org/docs/master/optim.html#algorithms for a list of choices, but if you need a recomendation use torch.optim.SGD
Next we need to pick a loss function. This function calcaluates how far off our prediction was from the expected value. Again there are a whole bunch of options. Here is a list http://pytorch.org/docs/master/nn.html#id46
If you are not sure what to choose then use torch.nn.functional.nll_loss which is negative log likelihood loss ($L(y) = -log(y)$) a common function to use with softmax
Can you write Python code for it?

sr_model = SoftmaxRegression()
sr_model.cuda()
# Change this to whatever optimizer you want to try
sr_optimizer = optim.SGD(sr_model.parameters(), lr=LEARNING_RATE, momentum=SGD_MOMENTUM)
print(sr_model)
#Change the last argument to whatever loss function you want to try
SR_TRAINING_LOSS, SR_TEST_LOSS, SR_TEST_ACCURACY = learn(sr_model, sr_optimizer, F.nll_loss) #negative loss values may be due to your final layer


I am working on the file "Copy_of_Introduction_to_CNNs_Handout.ipynb".
The first task is:
Testing Pipeline
Next we are going to define our testing or inference pipeline. During our training we want to intermittently chech the accuracy of our model. Since we don't want to do this with data the model has already seen we will show it new data that it will not remember since we will not be doing backpropogation here. You can see that the pipeline is almost the same with the changes after the data passes through the model and loss is calculated. Here we look at the probablilites outputed by the final layer of the models and find the index of the one with the highest. Each node corressponds to its index's value (e.g. index 0 means 0). We then compare the index of what was predicted to what was expected and calcuated the accuracy.
Can you write Python code for it?

def test(model, loss_func, epoch, test_loss_history, test_accuracy_history):
    model.eval()
    test_loss = 0
    correct = 0
    for data, target in test_loader:
        data, target = data.cuda(), target.cuda()                      #Transfer data and correct result to the GPU
        data, target = Variable(data, volatile=True), Variable(target) #Make data and correct answers variables in the Network
        output = model(data)                                           #classify the data
        test_loss += loss_func(output, target).data[0]                 #calculate loss
        pred = output.data.max(1)[1]                                   #get predictions for the batch using argmax
        correct += pred.eq(target.data).cpu().sum()                    #total correct anwsers
    test_loss /= len(test_loader)
    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(test_loss, correct, len(test_loader.dataset), 100. * correct / len(test_loader.dataset)))
    test_loss_history.append((epoch, test_loss))
    test_accuracy_history.append((epoch, 100. * correct / len(test_loader.dataset)))


I am working on the file "Copy_of_Introduction_to_CNNs_Handout.ipynb".
The first task is:
This function takes an image from the testing set and runs inference on it
Can you write Python code for it?

def classify_an_example(model):
    img = next(iter(test_loader))[0]
    img_np = img.cpu().numpy()[0]
    plt.imshow(img_np.reshape(28,28))
    print()
    print("The image is probably: {}".format(classify(model, img)[0]))
    print()


I am working on the file "Copy_of_Introduction_to_CNNs_Handout.ipynb".
The first task is:
Training the Model
Again we are going to set our training hyper parameters

epochs -> The number of times we show the training dataset to the model
learning rate -> The size of the step we take each time we go through backpropogation (i.e. the $\alpha$ in $W_{t+1} = W_t - \alpha \nabla_W\mathcal L$)
momentum (Note: this will only apply if you use SGD) -> We don't want to get stuck in local minima in the loss landscape so we may want to not let a particularly bad batch prevent otherwise good progress. We can redefine the parameter update procedure as $W_{t+1} = W_t - \alpha V_t$ and $V_t = \beta V_{t-1} + (1 - \beta)\nabla_W\mathcal L$ where $\beta$
is our notion of momentum

Set these values to something you think might be reasonable and see what happens. Make sure to shift+enter to lock them in.
Can you write Python code for it?

#@title Training Hyper Parameters
EPOCHS = 5                #@param {type:"integer"}           #Number of times to go through the data set
SGD_MOMENTUM = 0.5        #@param {type:"number"}            #How much it takes to change the direction of the gradient
LEARNING_RATE = 0.001     #@param {type:"number"}            #How far each update pushes the weights


#@title Training Hyper Parameters
EPOCHS = 10                #@param {type:"integer"}           #Number of times to go through the data set
SGD_MOMENTUM = 0.5        #@param {type:"number"}            #How much it takes to change the direction of the gradient
LEARNING_RATE = 0.001     #@param {type:"number"}            #How far each update pushes the weights


I am working on the file "Copy_of_Introduction_to_CNNs_Handout.ipynb".
The first task is:
Trying an Example on Your Trained Model
Use the classify_an_example helper function to classify an example from the testing set
Can you write Python code for it?


I am working on the file "Copy_of_Introduction_to_CNNs_Handout.ipynb".
The first task is:
Visualizing How Well the Training Went
Now we should have a reasonably well trained model. Lets see how the train went. Use the visualization function to plot the training loss, testing loss and testing accuracy over the course of the training.
Notice: We do not measure the training accuracy, since it is not a good measure of how good the model is since we are looking for a model that has good performace on data not yet seen
We should see the training loss quickly decline then plateau.
  - If you are seeing jagged training loss perhaps increase the batch size or decrease the learning rate
You should see that the testing loss more gradually reduces
  - if you see the testing loss starting to go up, you are now overfitting - probably reduce the number of epochs
You should also see the testing accuracy increase at the same rate the testing loss decreases
Can you write Python code for it?


I am working on the file "Copy_of_LP20221.ipynb".
The first task is:
Altere o exemplo anterior para utilizar funções.
Can you write Python code for it?

def valor_total(valor_unitario, qtde, desconto):
  return valor_unitario * qtde * (1-desconto)

nome = input("Digite o nome do produto: ")
qtde = int(input("Digite a qtde do produto: "))
valor_produto = float(input("Digite o valor unitário do produto: "))
print(f"Nome: {nome}")
print(f"Qtde: {qtde}")
print(f"Valor Unitário: {valor_produto}")
print(f"Valor Total: {valor_total(valor_produto,qtde,0)}")

categoria_cliente = input("Informe sua categoria(bronze, prata , ouro): ")
categoria_cliente = categoria_cliente.strip().lower()
desconto = 0.0
if categoria_cliente == "bronze":
  desconto = 5/100                # desconto = valor_total * (5/100)
elif categoria_cliente == "prata":
  desconto = 7/100
elif categoria_cliente == "ouro":
  desconto - 10/100
else:
  desconto = 0.0

print(f"Valor total com desconto para cliente {categoria_cliente}: {valor_total(valor_produto,qtde,desconto)}")


I am working on the file "Copy_of_LP20221.ipynb".
The first task is:
Conversão de tipo:
Can you write Python code for it?

x = 5             # criamos uma variável com valor inteiro
y = 4.3           # criamos uma variável com valor float (real)

print(type(x))    # apresentando o tipo de x
print(type(y))    # apresentando o tipo de y

print(f"x: {x}")  # exibindo o valor de x
print(f"y: {y}")  # exibindo o valor de y

z = float(x)      # conversão do valor de x para float
w = int(y)        # conversão do valor de y para int

print(f"z: {z}")  # exibindo o valor de z que é o valor de x convertido em float
print(f"w: {w}")  # exibindo o valor de w que é o valor de y convertido em float


I am working on the file "Copy_of_LP20221.ipynb".
The first task is:
Dada uma lista LISTA1 com 50 números aleatórios (de 0 a 50), criar uma lista LISTA2 com apenas os valores pares de LISTA1. Ao final o programa deverá apresentar os elementos de LISTA1 e LISTA2 para conferência.
Can you write Python code for it?

import random

lista1 = []
for x in range(50):
  numero_aleatorio = random.randrange(0,51)
  lista1.append(numero_aleatorio)         # lista1+=numero_aleatorio

lista2 = []
for numero in lista1:
  if numero%2 == 0:       # se o resto da divisão por 2 for zero o número é par
    lista2.append(numero)

print (lista1)
print (lista2)


I am working on the file "Copy_of_LP20221.ipynb".
The first task is:
Dada uma lista LISTA1 com 50 números aleatórios (de 0 a 50), criar uma lista LISTA2 com os valores de LISTA1 nas posições pares de LISTA2. Ao final o programa deverá apresentar os elementos de LISTA1 e os elementos de todas as posições de LISTA2 para conferência.
Can you write Python code for it?

import random

lista1 = []
for x in range(50):
  numero_aleatorio = random.randrange(0,51)
  lista1.append(numero_aleatorio)         # lista1+=numero_aleatorio

lista2 = []
for numero in lista1:
  lista2.append(numero)     # insere o número na posição par
  lista2.append(None)         # insere vazio na posição ímpar

print(lista1)
print(lista2)


I am working on the file "Copy_of_LP20221.ipynb".
The first task is:
Dada uma lista com 20 palavras, faça um programa que crie um dicionário onde a chave representa a primeira letra das palavras da lista e os valores representam todas as palavras da lista que começam com a letra da chave. Ao final do programa apresentar apenas o dicionário (chaves e valores).
Can you write Python code for it?

import logging

logging.basicConfig(level=logging.DEBUG, format='%(levelname)s - %(message)s')

logging.info("Carregando 20 palavras")

palavras = ["armário","cadeira","computador","caneta","lápis",
            "borracha","monitor","aleatório","número","final",
            "programa","papel","hardware","conjunto","presente",
            "mouse","teclado","relógio","tênis","camiseta"]

logging.debug(f"palavras[]: {palavras}")
logging.debug(f"len(palavras): {len(palavras)}")

logging.info("Indexando palavras")

dicionario = {}
logging.debug(f"dicionario: {dicionario}")

for palavra in palavras:
  letra = palavra[0]
  logging.debug(f"palavra: {palavra}")
  logging.debug(f"letra: {letra}")
  if letra in dicionario.keys():
    dicionario[letra].append(palavra) # se já existir a letra no dicionário, basta fazer o append para adicionar a nova palavra
  else: # se não existir a letra no dicionário, é preciso criar uma lista e guardar a palavra nela
    dicionario[letra] = [palavra]

logging.debug(f"dicionario{{}}: {dicionario}")


I am working on the file "Copy_of_LP20221.ipynb".
The first task is:
Dada uma lista com 50 números aleatórios (de 0 a 20), criar um dicionário com a contagem de cada elemento da lista. Ao final o programa deverá apresentar os elementos da lista e o dicionário (chaves e valores) para conferência.
Can you write Python code for it?

import logging
import random

logging.basicConfig(level=logging.DEBUG, format='%(levelname)s - %(message)s')

numeros = []
logging.debug(f"numeros[]: {numeros}")
logging.info("Criando 50 números aleatórios")
for x in range(50):
  logging.debug(f"x: {x}")
  numeros.append(random.randrange(0,21))  # adicionando 50 números de 0-9
  logging.debug(f"numeros[]: {numeros}")

numero = {}                      # dicionário que guarda a qtde de repetições de cada número
logging.info("Verificando a qtde de repetições de cada número")
logging.debug(f"numero{{}}: {numero}")
for n in range(21):               # percorrendo os 10 números (0-9)
  logging.debug(f"n: {n}")
  logging.debug(f"count({n}): {numeros.count(n)}")
  numero[n] = numeros.count(n) # contando as repetições de cada número (0-9)
  logging.debug(f"numero{{}}: {numero}")

print(f"50 números aleatórios: {numeros}")
print(f"Qtde de repetições de cada número:{numero}")


I am working on the file "Copy_of_LP20221.ipynb".
The first task is:
Dada uma lista com 50 números aleatórios (de 0 a 9), apresentar ao final do programa  o número que mais se repetiu e o número que menos se repetiu (caso um número não tenha aparecido, este deve ser o que menos se repetiu).
Can you write Python code for it?

import logging
import random

logging.basicConfig(level=logging.DEBUG, format='%(levelname)s - %(message)s')

numeros = []
logging.debug(f"numeros[]: {numeros}")
logging.info("Criando 50 números aleatórios")
for x in range(50):
  logging.debug(f"x: {x}")
  numeros.append(random.randrange(0,10))  # adicionando 50 números de 0-9
  logging.debug(f"numeros[]: {numeros}")

numero = []                       # lista que guarda a qtde de repetições de cada número
logging.info("Verificando a qtde de repetições de cada número")
logging.debug(f"numero[]: {numero}")
for n in range(10):               # percorrendo os 10 números (0-9)
  logging.debug(f"n: {n}")
  logging.debug(f"count({n}): {numeros.count(n)}")
  numero.append(numeros.count(n)) # contando as repetições de cada número (0-9)
  logging.debug(f"numero[]: {numero}")

print(f"50 números aleatórios: {numeros}")
print(f"Qtde de repetições de cada número:{numero}")

qtdeMaiorRepeticao = -1
logging.debug(f"qtdeMaiorRepeticao: {qtdeMaiorRepeticao}")
qtdeMenorRepeticao = 1000000
logging.debug(f"qtdeMenorRepeticao: {qtdeMenorRepeticao}")

logging.info("Descobrindo a menor e a maior repetição")
for n in numero:
  logging.debug(f"n: {n}")
  logging.debug(f"qtdeMenorRepeticao: {qtdeMenorRepeticao}")
  logging.debug(f"qtdeMaiorRepeticao: {qtdeMaiorRepeticao}")

  if n > qtdeMaiorRepeticao:
    qtdeMaiorRepeticao = n
  if n < qtdeMenorRepeticao:
    qtdeMenorRepeticao = n

print(f"Número que mais se repetiu: {numero.index(qtdeMaiorRepeticao)}")
print(f"Número que menos se repetiu: {numero.index(qtdeMenorRepeticao)}")


I am working on the file "Copy_of_LP20221.ipynb".
The first task is:
Dados dois conjuntos SET1 e SET2, cada qual contendo números aleatórios (de 0 a 50) verificar quais números estão presentes nos dois conjuntos e quais números não aparecem em nenhum dos dois conjuntos. Ao final do programa apresentar todos esses dados.
Can you write Python code for it?

import logging
import random

logging.basicConfig(level=logging.DEBUG, format='%(levelname)s - %(message)s')

set1 = set()
set2 = set()
todos_numeros = set(range(51))

logging.info("Gerando os conjuntos set1 e set2 com 20 números aleatórios (0-50)")
for numero in range(20):
  set1.add(random.randrange(0,51))  # adicionando número de 0-50
  set2.add(random.randrange(0,51))  # adicionando número de 0-50

logging.debug(f"set1{{}}: {set1}")
logging.debug(f"set2{{}}: {set2}")
logging.debug(f"todos_numeros{{}}: {todos_numeros}")

logging.info(f"Números presentes nos dois conjuntos: {set1 & set2}")
logging.info(f"Números não presentes nos dois conjuntos: {todos_numeros - (set1 | set2)}")


I am working on the file "Copy_of_LP20221.ipynb".
The first task is:
Dados três conjuntos SET1, SET2 e SET3, cada qual contendo números aleatórios (de 0 a 50) verificar quais números estão presentes nos três conjuntos e quais números estão presentes apenas em SET1, ou seja, estes números não podem estar aparecendo em SET2 ou SET3. Ao final do programa apresentar todos esses dados.
Can you write Python code for it?

import logging
import random

logging.basicConfig(level=logging.DEBUG, format='%(levelname)s - %(message)s')

set1 = set()
set2 = set()
set3 = set()

logging.info("Gerando os conjuntos set1, set2 e set3 com 20 números aleatórios (0-50)")
for numero in range(20):
  set1.add(random.randrange(0,51))  # adicionando número de 0-50
  set2.add(random.randrange(0,51))
  set3.add(random.randrange(0,51))

logging.debug(f"set1{{}}: {set1}")
logging.debug(f"set2{{}}: {set2}")
logging.debug(f"set3{{}}: {set3}")

logging.info(f"Números presentes nos 3 conjuntos: {set1&set2&set3}")
logging.info(f"Números presentes apenas em set1: {set1 - (set2|set3)}")


I am working on the file "Copy_of_LP20221.ipynb".
The first task is:
Faça um programa para manipular um cadastro de produtos de informática.
Para cada produto deverá ser registrado o nome, qtde em estoque, preço unitário e tipo.
O programa deverá permitir operações de inclusão, exclusão, busca por nome, busca por tipo e exibição ordenada dos dados.
Can you write Python code for it?

import logging

logging.basicConfig(level=logging.DEBUG, format='%(levelname)s - %(message)s')

def menu():
  print("="*50)
  print("  SISTEMA DE CONTROLE DE PRODUTOS DE INFORMÁTICA")
  print("="*50)
  print("  Opções: ")
  print("  [1] - Cadastrar")
  print("  [2] - Procurar por nome")
  print("  [3] - Procurar por tipo")
  print("  [4] - Excluir")
  print("  [5] - Exibir produtos")
  print("  [0] - Sair")
  print(f"  Qtde de produtos: {len(produtos)}")
  print("="*50)

def show(produto):
  print(f"{produto.get('nome')}\t{produto.get('qtde')}\t{produto.get('preco')}\t{produto.get('tipo')}")

def showAll(produtos):
  print("NOME\tQTDE\tPRECO\tTIPO")
  produtos_ordenados = sorted(produtos, key=lambda produto:produto["nome"])
  for produto in produtos_ordenados:
    show(produto)

def cadastrar():
  logging.info("Cadastrando produto de informática")
  nome = input("Nome do produto: ")
  qtde = int(input("Qtde: "))
  preco = float(input("Preço: "))
  tipo = input("Tipo: ")
  produto = {
      "nome":nome,
      "qtde":qtde,
      "preco":preco,
      "tipo":tipo
  }
  produtos.append(produto)

def procurar_nome():
  logging.info("Pesquisando por nome do produto")
  nome = input("Nome do produto a ser localizado: ").lower()
  encontrado = False
  for produto in produtos:
    if nome in produto.get("nome").lower():
      encontrado = True
      show(produto)
  if encontrado==False:
    print("Produto não localizado")

def procurar_tipo():
  logging.info("Pesquisando por tipo do produto")
  tipo = input("Tipo do produto a ser localizado: ").lower()
  encontrado = False
  for produto in produtos:
    if tipo in produto.get("tipo").lower():
      encontando=True
      show(produto)
  if encontrado==False:
    print("Tipo não localizado")

def excluir():
  logging.info("Excluindo produto")
  nome = input("Nome do produto a ser excluído: ").lower()
  posicao = 0
  encontrado = -1
  for produto in produtos:
    if nome == produto.get("nome").lower():
      encontrado = posicao
      show(produto)
    posicao+=1
  if...


I am working on the file "Copy_of_LP20221.ipynb".
The first task is:
Faça um programa que receba do usuário um arquivo texto e mostre na tela quantas vezes cada letra do alfabeto aparece dentro do arquivo.
Can you write Python code for it?

import string

arquivo_nome = input("Arquivo: ")
alfabeto = list(string.ascii_lowercase)
qtdes = []

for contador in range(len(alfabeto)):
  qtdes.append(0)

with open(arquivo_nome, 'r') as arquivo:
  for linha in arquivo:
    for letra in alfabeto:
      qtdes[alfabeto.index(letra)]+=linha.count(letra)

print(alfabeto)
print(qtdes)


I am working on the file "Copy_of_LP20221.ipynb".
The first task is:
Faça um programa que receba do usuário um arquivo texto e um caracter. Mostre na tela quantas vezes aquele caractere ocorre dentro do arquivo.
Can you write Python code for it?

arquivo_nome = input("Arquivo: ")
caractere = input("Caractere: ")

qtde = 0
with open(arquivo_nome, 'r') as arquivo:
  for linha in arquivo:
    qtde = qtde + linha.count(caractere)
print(qtde)

arquivo_nome = input("Arquivo: ")
caractere = input("Caractere: ")

qtde = 0
with open(arquivo_nome, 'r') as arquivo:
  for linha in arquivo:
    for letra in linha:
      if (letra == caractere):
        qtde = qtde + 1
print(qtde)


I am working on the file "Copy_of_LP20221.ipynb".
The first task is:
Faça um programa que use a função valor_pagamento para determinar o valor a ser pago por uma prestação de uma conta. O programa deverá solicitar ao usuário o valor da prestação e o número de dias em atraso e passar estes valores para a função valorPagamento, que calculará o valor a ser pago e devolverá este valor ao programa que a chamou. O programa deverá então exibir o valor a ser pago na tela. Após a execução o programa deverá voltar a pedir outro valor de prestação e assim continuar até que seja informado um valor igual a zero para a prestação. Neste momento o programa deverá ser encerrado, exibindo o relatório do dia, que conterá a quantidade e o valor total de prestações pagas no dia. O cálculo do valor a ser pago é feito da seguinte forma. Para pagamentos sem atraso, cobrar o valor da prestação. Quando houver atraso, cobrar 3% de multa, mais 0,1% de juros por dia de atraso.
Can you write Python code for it?

def valor_pagamento(valorPrestacao, diasAtraso):
  if diasAtraso<=0:
    multa = 0
  else:
    multa = 3/100+ (0.1/100)*diasAtraso
  return valorPrestacao*(1+multa)

texto = ""
qtde = 0
totalPagamento = 0
while True:
  valorPrestacao = float(input("Valor da prestação: "))
  if valorPrestacao == 0: break;
  qtde += 1
  diasAtraso = int(input("Dias de atraso: "))
  totalPagamento += valor_pagamento(valorPrestacao,diasAtraso)
  texto += f"Prestação de R$ {valorPrestacao}, com {diasAtraso} dias de atraso ficou em R$ {str(valor_pagamento(valorPrestacao,diasAtraso))}\n"

print(texto)
print(f"Qtde de pagamentos realizados: {qtde}")
print(f"Valor total dos pagamentos: {totalPagamento}")


I am working on the file "Copy_of_LP20221.ipynb".
The first task is:
Faça um programa, com uma função que necessite de três argumentos, e que forneça a soma desses três argumentos.
Can you write Python code for it?

def  soma (valor1, valor2, valor3):
  return valor1+valor2+valor3

print(soma(1,2,3))


I am working on the file "Copy_of_LP20221.ipynb".
The first task is:
Nome ao contrário em maiúsculas. Faça um programa que permita ao usuário digitar o seu nome e em seguida mostre o nome do usuário de trás para frente utilizando somente letras maiúsculas. Dica: lembre−se que ao solicitar o nome, o usuário pode digitar letras maiúsculas ou minúsculas.
Can you write Python code for it?

nome = input("Nome: ").upper()
print(nome[::-1])


I am working on the file "Copy_of_LP20221.ipynb".
The first task is:
Operações com String (texto)
Can you write Python code for it?

nome = "   João Paulo   "
print(len(nome))             #tamanho da string
nome = nome.strip()          #remove os espaços em branco nas extremidades
print(len(nome))
print(nome[1])               #o primeiro caractere é o de posição 0
print(nome[2:4])             #de 2 a 3 (último não incluso)
print(nome[-5:-1])           #começa a contagem pelo final da string
print(nome[0:10:2])          ##de 0 a 10 (último não incluso) saltando duas posições a cada caractere
print(nome.lower())          #transforma em minúsculo
print(nome.upper())          #transforma em maiúsculo
print(nome.replace("o","0")) #substitui a string
print(nome.split(" "))       #returns ["João","Paulo"]
print("au" in nome)          #retorna True se existe na string
print("au" not in nome)      #retorna True se não existe na string
idade = 40
texto = "Meu nome é {} e tenho {} anos."
print(texto.format(nome,idade))

nota = float(input("Digite a nota do estudante: "))

if nota == 10:
  print("Aprovado com Mérito")
elif nota >= 6:
  print("Aprovado")
elif nota >= 5:
  print("Em recuperação")
  notapf = float(input("Digite a nota da PF: "))
  media = (nota + notapf)/2
  if (media >= 5):
    print("Aprovado na Recuperação")
  else:
    print("Reprovado na Recuperação")
else:
  print("Reprovado")


I am working on the file "Copy_of_LP20221.ipynb".
The first task is:
Verificar se a entrada do usuário é um CPF em formato válido utilizando expressões regulares.
Can you write Python code for it?

import re

cpf = input("CPF: ")
expCPF = "\d{3}[.]\d{3}[.]\d{3}[-]\d{2}"

if re.match(expCPF, cpf):
  print("Formato Válido")
else:
  print("Formato Inválido")


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
Calendar heatmap
Can you write Python code for it?

alt.Chart(snPdsiDf).mark_rect().encode(
  x = 'Year:O',
  y = 'Month:O',
  color = alt.Color('PDSI:Q', scale=alt.Scale(scheme="redblue", domain=(-5, 5))),
  tooltip = [
    alt.Tooltip('Year:O', title='Year'),
    alt.Tooltip('mean(Month):O', title='Month'),
    alt.Tooltip('PDSI:Q', title='PDSI')
  ]).properties(width=700, height=300)


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
Convert list of lists to dataframe

Convert the list to a Pandas dataframe.
Print the shape of the new dataframe. shape returns (n rows, n columns)
Can you write Python code for it?

snPdsiDf = pd.DataFrame(snPdsiList)
print(snPdsiDf.shape)


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
Define a data structure translator
The above data structure transformation will be used again multiple times; make it a function.
Can you write Python code for it?

cols = {'Year': int, 'Month': int, 'DOY': int, 'Date': str, 'PDSI': float}

def eeList2Df(list, cols):
  df = pd.DataFrame(list).transpose()
  df.columns = [k for k in cols.keys()]
  return(df.astype(cols))


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
Display intra-annual reduction a line plot.
Can you write Python code for it?

alt.Chart(pointNDVIDf).mark_line().encode(
  x='Date:T',
  y='MSAVI2:Q',
  tooltip=[
    alt.Tooltip('Date:T', title='Date'),
    alt.Tooltip('NDVI:Q', title='MSAVI2')
  ]
).interactive().properties(width=700, height=300)


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
Format the table
Check the format
Print the list dimensions. These data are in memory on the Colab VM.
Can you write Python code for it?

# Print the dimensions of the returned list.
print('n variables:', len(snPdsiList))
print('n observations:', len(snPdsiList[0]))


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
Import other libraries
Can you write Python code for it?

import numpy as np
import pandas as pd
import altair as alt

!pip install geopandas
import geopandas

!pip install eeconvert
import eeconvert


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
It's missing column names - add some and preview again.
Can you write Python code for it?

snPdsiDf.columns = ['Year', 'Month', 'DOY', 'Date', 'PDSI']
snPdsiDf.head(5)


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
Now check the data type of each column.
Can you write Python code for it?


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
Perform a join on 'Year' to combine the two dataframes.
Can you write Python code for it?

ndviPdsiCombo = pd.merge(snNdviDfSub, snPdsiDfSub, how='left', on='Year') \
  .drop(columns=['DOY_x', 'DOY_y', 'Month']) \
  .reset_index()

ndviPdsiCombo.head(5)


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
Prepare a Landsat surface reflectance collection
Edit date window inputs based on NDVI curve plotted previously and set latitude and longitude variables from the map above.
Can you write Python code for it?

# Start and end dates.
startDay = 1
endDay = 365


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
Prepare a Landsat surface reflectance collection 2017-present.
Can you write Python code for it?

# Make lat and lon an `ee.Geometry.Point` - AOI.
point = sn.geometry().centroid()
polygon = sn.geometry()
#point = ee.Geometry.Point([Longitude, Latitude])

# Define function to get and rename bands of interest from OLI.
def renameOLI(img):
  return(img.select(
    ee.List(['B2', 'B3', 'B4', 'B5', 'B6', 'B7', 'pixel_qa']),
		ee.List(['Blue', 'Green', 'Red', 'NIR', 'SWIR1', 'SWIR2', 'pixel_qa'])))

# Define function to get and rename bands of interest from ETM+.
def renameETM(img):
  return(img.select(
		ee.List(['B1', 'B2', 'B3', 'B4', 'B5', 'B7', 'pixel_qa']),
		ee.List(['Blue', 'Green', 'Red', 'NIR', 'SWIR1', 'SWIR2', 'pixel_qa'])))

# Define function to get and rename bands of interest from Sentinnel2
def renameS2(img):
  return(img.select(
		ee.List(['B2', 'B3', 'B4', 'B8A', 'B11', 'B12', 'SCL']),
		ee.List(['Blue', 'Green', 'Red', 'NIR', 'SWIR1', 'SWIR2', 'pixel_qa'])))

# Define function to mask out clouds and cloud shadows.
def fmask(img):
  cloudShadowBitMask = 1 << 3
  cloudsBitMask = 1 << 5
  qa = img.select('pixel_qa')
  mask = qa.bitwiseAnd(cloudShadowBitMask).eq(0)\
    .And(qa.bitwiseAnd(cloudsBitMask).eq(0))
  return(img.updateMask(mask))

# Define function to mask out clouds and cloud shadows.
def sclMask(img):
  qa = img.select('pixel_qa')
  mask = qa.eq(4)\
    .Or(qa.eq(5))\
    .Or(qa.eq(6))\
    .Or(qa.eq(7))\
    .Or(qa.eq(11))
  return(img.updateMask(mask))

# Define function to add year.
def setYear(img):
  year = ee.Image(img).date().get('year')
  return(img.set('Year', year))

def setMonth(img):
  month = ee.Image(img).date().get('month')
  return img.set('Month', month)

# Define function to calculate NBR.
def calcNBR(img):
  return(img.normalizedDifference(ee.List(['NIR', 'SWIR2'])).rename('NBR'))

# Define function to calculate NDVI.
def calcNDVI(img):
  return(img.normalizedDifference(ee.List(['NIR', 'Red'])).rename('NDVI'))

def calcMSAVI2(img):
  nir = img.select('NIR')
  red = img.select('Red')

  return img.expression(
  '(2 * NIR + 1 - sqrt(pow((2 * NIR + 1), 2) - 8 * (NIR...


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
Preview the first 5 rows of the dataframe.
Can you write Python code for it?


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
Reduce NDVI by region
A measure of photosynthetic capacity from MODIS.
Get data and reduce it

Load MODIS NDVI data as an Earth Engine ee.ImageCollection.
Set the region reduce scale argument to 1000 (meters).
Get the data from Earth Engine to the Colab VM as a list of lists.
Can you write Python code for it?

def calculate_ndvi(image):
  ndvi = image.normalizedDifference(['B8', 'B4']).rename('NDVI')
  return image.addBands(ndvi)

ndvi = ee.ImageCollection('MODIS/006/MOD13A2').filterDate('2017-01-01', '2020-03-25').select('NDVI')

reReArgs['scale'] = 1000
snNdviList = getReReList(ndvi, ['Year', 'DOY', 'NDVI'])


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
Reduce PDSI by AOI

Set the reduce region arguments.
Reduce PDSI for the AOI and return the results as a list. (return all properties in the provided list)
Can you write Python code for it?

# Reset global arguments.
reReArgs['scale'] = 5000
reReArgs['reducer'] = ee.Reducer.mean()
reReArgs['geometry'] = sn
reReArgs['crs'] = 'EPSG:3310'

# Get a list containing a series of lists, one for each property given.
snPdsiList = getReReList(pdsi, ['Year', 'Month', 'DOY', 'Date', 'pdsi'])


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
Reduce the plot to median and interquartile range.
Can you write Python code for it?

line = alt.Chart(snNdviDf).mark_line().encode(
  x='DOY:O',
  y='median(NDVI):Q',
).interactive()

band = alt.Chart(snNdviDf).mark_errorband(extent='iqr').encode(
  x='DOY:O',
  y=alt.Y('NDVI:Q', scale=alt.Scale(domain=(0.1, 0.7))),
).interactive().properties(width=700, height=300)

band + line


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
Reduce the region.
Can you write Python code for it?

# reset global args
reReArgs['reducer'] = ee.Reducer.mean()
reReArgs['scale'] = 30
reReArgs['geometry'] = polygon

# Get a list containing a series of lists, one for each property given.
pointNDVIList = getReReList(landsatCol, ['Date', 'MSAVI2'])

#pointNbrList = getReReList(landsatCol, ['Date', 'NBR'])


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
Reduction function
This function reduces the pixels intersecting a region to a statistic. It operates on a single ee.Image and returns an ee.Feature (after the reduction, the pixel values and geometry matter not, so just return an aspatial feature). The plan is to map this over an ee.ImageCollection.

Derive a series of date properties that will be used in charting.
Apply the reduceRegion method to the image using arguments provided by the global reReArgs dictionary.
Return an ee.Feature without geometry that contains the region reduction statistic result (an ee.dictionary), all image properties, and the derived date properties.
Can you write Python code for it?

def regionReduce(img):
  eeDate = img.date()
  year = eeDate.get('year')
  month = eeDate.getRelative('month', 'year')
  doy = eeDate.getRelative('day', 'year')
  date = eeDate.format('YYYY-MM-dd')

  stat = img.reduceRegion(
    reducer = reReArgs['reducer'],
    geometry = reReArgs['geometry'],
    scale = reReArgs['scale'],
    crs = reReArgs['crs'],
    bestEffort = reReArgs['bestEffort'],
    maxPixels = reReArgs['maxPixels'],
    tileScale = reReArgs['tileScale'])

  return(ee.Feature(None, stat) \
    .copyProperties(img, img.propertyNames())
    .set({
      'DOY': doy,
      'Month': month,
      'Year': year,
      'Date': date}))


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
Subset the PDSI data to include all days prior to the end of the above defined NDVI DOY range.
Reduce the values within a year by mean of observations.
Can you write Python code for it?

pdsiDoy = [1, 272]

snPdsiDfSub = snPdsiDf[
  (snPdsiDf['DOY'] >= pdsiDoy[0]) & (snPdsiDf['DOY'] <= pdsiDoy[1])
]

snPdsiDfSub = snPdsiDfSub.groupby('Year').agg('mean')


I am working on the file "Copy_of_time_series_visualization.ipynb".
The first task is:
Transform and tidy the table.
Can you write Python code for it?

cols = {'Date': str, 'MSAVI2': float}
pointNDVIDf = eeList2Df(pointNDVIList, cols)
pointNDVIDf.tail(5)