I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/07-Classification-Exercise.ipynb".
The first task is:
Train your model on the data, for at least 5000 steps.
Can you write Python code for it?

model.train(input_fn = input_func,
            steps = 5000)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/00-MNIST-Data-Basic-Approach.ipynb".
The first task is:
Create Session
Can you write Python code for it?

# Intializing all variables
init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)

    # Train the model for 1000 steps on the training set
    # Using built in batch feeder from mnist for convenience
    for step in range(1000):
        # Training on a batch of 100 examples
        batch_x , batch_y = mnist.train.next_batch(100)

        sess.run(train, feed_dict = {x : batch_x,
                                     y_true : batch_y})

    # Calculating the number of matches
    matches = tf.equal(tf.argmax(y, 1),
                       tf.argmax(y_true, 1))

    acc = tf.reduce_mean(tf.cast(matches, tf.float32))

    # Calculating the accuracy
    print(sess.run(acc, feed_dict = {x : mnist.test.images,
                                     y_true : mnist.test.labels}))


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/00-MNIST-Data-Basic-Approach.ipynb".
The first task is:
Create the Model
Can you write Python code for it?

# Initializing a Placeholder of shape None (number of inputs) by 784
# Tensorflow requires float32
x = tf.placeholder(tf.float32,
                shape = [None, 784])

# Initializing weights between the input layer and the output layer
# It is of shape number_of_features by number_of_neurons_in_the_layer
# Initializing with zeros, which is meh, but we will use it for simplicity

# 10 because 0-9 possible numbers
W = tf.Variable(tf.zeros([784, 10]))

# Initializing biases
b = tf.Variable(tf.zeros([10]))

# Create the Graph
y = tf.matmul(x, W) + b


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/00-MNIST-Data-Basic-Approach.ipynb".
The first task is:
Visualizing the Data
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

# The image is a long array
mnist.train.images[1].shape

# Showing reshaped image
plt.imshow(mnist.train.images[1].reshape(28, 28))

# Showing the image in gist gray scale
plt.imshow(mnist.train.images[1].reshape(28, 28),
           cmap = 'gist_gray')

mnist.train.images[1].max()

plt.imshow(mnist.train.images[1].reshape(784, 1))

plt.imshow(mnist.train.images[1].reshape(784, 1),
           cmap = 'gist_gray',
           aspect = 0.02)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/01-MNIST-with-CNN.ipynb".
The first task is:
Create a 2D convolution using builtin conv2d from TF. From those docs:
Computes a 2-D convolution given 4-D input and filter tensors.
Given an input tensor of shape [batch, in_height, in_width, in_channels]
and a filter / kernel tensor of shape
[filter_height, filter_width, in_channels, out_channels], this op
performs the following:

Flattens the filter to a 2-D matrix with shape
   [filter_height * filter_width * in_channels, output_channels].
Extracts image patches from the input tensor to form a virtual
   tensor of shape [batch, out_height, out_width,
   filter_height * filter_width * in_channels].
For each patch, right-multiplies the filter matrix and the image patch
   vector.
Can you write Python code for it?

def conv2d(x, W):
    return tf.nn.conv2d(x, W,
                        strides = [1, 1, 1, 1],
                        padding = 'SAME')


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/01-MNIST-with-CNN.ipynb".
The first task is:
Helper Functions
Function to help intialize random weights for fully connected or convolutional layers, we leave the shape attribute as a parameter for this.
Can you write Python code for it?

def init_weights(shape):
    init_random_dist = tf.truncated_normal(shape = shape, stddev = 0.1)
    return tf.Variable(init_random_dist)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/01-MNIST-with-CNN.ipynb".
The first task is:
Loss Function
Can you write Python code for it?

cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels = y_true,
                                                                          logits = y_pred))


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/01-MNIST-with-CNN.ipynb".
The first task is:
Optimizer
Can you write Python code for it?

# Training with the Adam Optimizer
optimizer = tf.train.AdamOptimizer(learning_rate = 0.0001)
train = optimizer.minimize(cross_entropy)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/01-MNIST-with-CNN.ipynb".
The first task is:
Same as init_weights, but for the biases
Can you write Python code for it?

def init_bias(shape):
    init_bias_vals = tf.constant(shape =shape, value = 0.1)
    return tf.Variable(init_bias_vals)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/01-MNIST-with-CNN.ipynb".
The first task is:
This is a normal fully connected layer
Can you write Python code for it?

def normal_full_layer(input_layer, size):
    input_size = int(input_layer.get_shape()[1])
    W = init_weights([input_size, size])
    b = init_bias([size])
    return tf.matmul(input_layer, W) + b


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/01-MNIST-with-CNN.ipynb".
The first task is:
Using the conv2d function, we'll return an actual convolutional layer here that uses an ReLu activation.
Can you write Python code for it?

def convolutional_layer(input_x, shape):
    W = init_weights(shape)
    b = init_bias([shape[3]])
    return tf.nn.relu(conv2d(input_x, W) + b)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/02-CNN-Project-Exercise.ipynb".
The first task is:
Create 2 placeholders, x and y_true. Their shapes should be:

x shape = [None,32,32,3]
y_true shape = [None,10]
Can you write Python code for it?

x = tf.placeholder(shape = [None, 32, 32, 3], dtype = tf.float32)
y = tf.placeholder(shape = [None, 10], dtype = tf.float32)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/02-CNN-Project-Exercise.ipynb".
The first task is:
Create a new full layer using the normal_full_layer function and passing in your flattend convolutional 2 layer with size=1024. (You could also choose to reduce this to something like 512)
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/02-CNN-Project-Exercise.ipynb".
The first task is:
Create a variable to intialize all the global tf variables.
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/02-CNN-Project-Exercise.ipynb".
The first task is:
Create the Layers
 Create a convolutional layer and a pooling layer as we did for MNIST.
 Its up to you what the 2d size of the convolution should be, but the last two digits need to be 3 and 32 because of the 3 color channels and 32 pixels. So for example you could use:
    convo_1 = convolutional_layer(x,shape=[4,4,3,32])
Can you write Python code for it?

convo_1 = convolutional_layer(x, shape = [4, 4, 3, 32])
pool_1 = max_pool_2_by_2(convo_1)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/02-CNN-Project-Exercise.ipynb".
The first task is:
Create the next convolutional and pooling layers.  The last two dimensions of the convo_2 layer should be 32,64
Can you write Python code for it?

convo_2 = convolutional_layer(pool_1, shape = [4, 4, 32, 64])
pool_2 = max_pool_2_by_2(convo_2)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/02-CNN-Project-Exercise.ipynb".
The first task is:
Creating the Model
 Import tensorflow
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/02-CNN-Project-Exercise.ipynb".
The first task is:
Finally set the output to y_pred by passing in the dropout layer into the normal_full_layer function. The size should be 10 because of the 10 possible labels
Can you write Python code for it?

y_pred = fully_connected_layer(fully_connected_layer_after_dropout, size = 10)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/02-CNN-Project-Exercise.ipynb".
The first task is:
Graph Session
 Perform the training and test print outs in a Tf session and run your model!
Can you write Python code for it?

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    for i in range(5000):
        batch = ch.next_batch(100)
        sess.run(train, feed_dict={x: batch[0],
                                   y: batch[1],
                                   hold_prob: 0.5})

        # PRINT OUT A MESSAGE EVERY 100 STEPS
        if i%100 == 0:

            print('Currently on step {}'.format(i))
            print('Accuracy is:')
            # Test the Train Model
            matches = tf.equal(tf.argmax(y_pred,1),tf.argmax(y,1))

            acc = tf.reduce_mean(tf.cast(matches,tf.float32))

            print(sess.run(acc,feed_dict={x: ch.test_images,
                                          y: ch.test_labels,
                                          hold_prob: 1.0}))
            print('\n')


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/02-CNN-Project-Exercise.ipynb".
The first task is:
Helper Functions
 Grab the helper functions from MNIST with CNN (or recreate them here yourself for a hard challenge!). You'll need:

init_weights
init_bias
conv2d
max_pool_2by2
convolutional_layer
normal_full_layer
Can you write Python code for it?

def initialize_weights(shape):
    init_random_distribution = tf.truncated_normal(shape, stddev = 0.1)
    return tf.Variable(init_random_distribution)

def initialize_bias(shape):
    init_bias_vals = tf.constant(0.1, shape=shape)
    return tf.Variable(init_bias_vals)

def conv2d(x, W):
    return tf.nn.conv2d(x, W, strides = [1, 1, 1, 1], padding = 'SAME')

def max_pool_2_by_2(x):
    return tf.nn.max_pool(x, ksize = [1, 2, 2, 1], strides = [1, 2, 2, 1], padding = 'SAME')

def convolutional_layer(input_x, shape):
    W = initialize_weights(shape)
    b = initialize_bias([shape[3]])
    return tf.nn.relu(conv2d(input_x, W) + b)

def fully_connected_layer(input_layer, size):
    input_size = int(input_layer.get_shape()[1])
    W = initialize_weights([input_size, size])
    b = initialize_bias([size])
    return tf.matmul(input_layer, W) + b


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/02-CNN-Project-Exercise.ipynb".
The first task is:
How to use the above code:
Can you write Python code for it?

# Before Your tf.Session run these two lines
ch = CifarHelper()
ch.set_up_images()

# During your session to grab the next batch use this line
# (Just like we did for mnist.train.next_batch)
# batch = ch.next_batch(100)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/02-CNN-Project-Exercise.ipynb".
The first task is:
Now create the dropout layer with tf.nn.dropout, remember to pass in your hold_prob placeholder.
Can you write Python code for it?

fully_connected_layer_after_dropout = tf.nn.dropout(x = fully_connected_layer_1,
                                                    keep_prob = hold_prob)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/02-CNN-Project-Exercise.ipynb".
The first task is:
Optimizer
 Create the optimizer using an Adam Optimizer.
Can you write Python code for it?

optimizer = tf.train.AdamOptimizer(learning_rate=0.001)
train = optimizer.minimize(cross_entropy)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/02-CNN-Project-Exercise.ipynb".
The first task is:
The archive contains the files data_batch_1, data_batch_2, ..., data_batch_5, as well as test_batch. Each of these files is a Python "pickled" object produced with cPickle.
 Load the Data. Use the Code Below to load the data:
Can you write Python code for it?

def unpickle(file):
    import pickle
    with open(file, 'rb') as fo:
        cifar_dict = pickle.load(fo, encoding='bytes')
    return cifar_dict

dirs = ['batches.meta','data_batch_1','data_batch_2','data_batch_3','data_batch_4','data_batch_5','test_batch']

all_data = [0, 1, 2, 3, 4, 5, 6]

for i,direc in zip(all_data,dirs):
    all_data[i] = unpickle(CIFAR_DIR + direc)

batch_meta = all_data[0]
data_batch1 = all_data[1]
data_batch2 = all_data[2]
data_batch3 = all_data[3]
data_batch4 = all_data[4]
data_batch5 = all_data[5]
test_batch = all_data[6]

batch_meta


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/03-Convolutional-Neural-Networks/02-CNN-Project-Exercise.ipynb".
The first task is:
Why the 'b's in front of the string?
Bytes literals are always prefixed with 'b' or 'B'; they produce an instance of the bytes type instead of the str type. They may only contain ASCII characters; bytes with a numeric value of 128 or greater must be expressed with escapes.
https://stackoverflow.com/questions/6269765/what-does-the-b-character-do-in-front-of-a-string-literal
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/.ipynb_checkpoints/03-Time-Series-Exercise-Solutions-Final-checkpoint.ipynb".
The first task is:
Batch Function
 We'll need a function that can feed batches of the training data. We'll need to do several things that are listed out as steps in the comments of the function. Remember to reference the previous batch method from the lecture for hints. Try to fill out the function template below, this is a pretty hard step, so feel free to reference the solutions!
Can you write Python code for it?

def next_batch(training_data,batch_size,steps):
    """
    INPUT: Data, Batch Size, Time Steps per batch
    OUTPUT: A tuple of y time series results. y[:,:-1] and y[:,1:]
    """

    # STEP 1: Use np.random.randint to set a random starting point index for the batch.
    # Remember that each batch needs have the same number of steps in it.
    # This means you should limit the starting point to len(data)-steps

    # STEP 2: Now that you have a starting index you'll need to index the data from
    # the random start to random start + steps. Then reshape this data to be (1,steps)

    # STEP 3: Return the batches. You'll have two batches to return y[:,:-1] and y[:,1:]
    # You'll need to reshape these into tensors for the RNN. Depending on your indexing it
    # will be either .reshape(-1,steps-1,1) or .reshape(-1,steps,1)

def next_batch(training_data,batch_size,steps):

    # Grab a random starting point for each batch
    rand_start = np.random.randint(0,len(training_data)-steps)

    # Create Y data for time series in the batches
    y_batch = np.array(training_data[rand_start:rand_start+steps+1]).reshape(1,steps+1)

    return y_batch[:, :-1].reshape(-1, steps, 1), y_batch[:, 1:].reshape(-1, steps, 1)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/.ipynb_checkpoints/03-Time-Series-Exercise-Solutions-Final-checkpoint.ipynb".
The first task is:
Check out the head of the dataframe
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/.ipynb_checkpoints/03-Time-Series-Exercise-Solutions-Final-checkpoint.ipynb".
The first task is:
Create Placeholders for X and y. (You can change the variable names if you want). The shape for these placeholders should be [None,num_time_steps-1,num_inputs] and [None, num_time_steps-1, num_outputs] The reason we use num_time_steps-1 is because each of these will be one step shorter than the original time steps size, because we are training the RNN network to predict one point into the future based on the input sequence.
Can you write Python code for it?

X = tf.placeholder(tf.float32, [None, num_time_steps, num_inputs])
y = tf.placeholder(tf.float32, [None, num_time_steps, num_outputs])


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/.ipynb_checkpoints/03-Time-Series-Exercise-Solutions-Final-checkpoint.ipynb".
The first task is:
Grab the portion of the results that are the generated values and apply inverse_transform on them to turn them back into milk production value units (lbs per cow). Also reshape the results to be (12,1) so we can easily add them to the test_set dataframe.
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/.ipynb_checkpoints/03-Time-Series-Exercise-Solutions-Final-checkpoint.ipynb".
The first task is:
Initialize the global variables
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/.ipynb_checkpoints/03-Time-Series-Exercise-Solutions-Final-checkpoint.ipynb".
The first task is:
Now pass in the cells variable into tf.nn.dynamic_rnn, along with your first placeholder (X)
Can you write Python code for it?

outputs, states = tf.nn.dynamic_rnn(cell, X, dtype=tf.float32)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/.ipynb_checkpoints/03-Time-Series-Exercise-Solutions-Final-checkpoint.ipynb".
The first task is:
Now we want to attempt to predict these 12 months of data, using only the training data we had. To do this we will feed in a seed training_instance of the last 12 months of the training_set of data to predict 12 months into the future. Then we will be able to compare our generated 12 months to our actual true historical values from the test set!
Generative Session
NOTE: Recall that our model is really only trained to predict 1 time step ahead, asking it to generate 12 steps is a big ask, and technically not what it was trained to do! Think of this more as generating new values based off some previous pattern, rather than trying to directly predict the future. You would need to go back to the original model and train the model to predict 12 time steps ahead to really get a higher accuracy on the test data. (Which has its limits due to the smaller size of our data set)
 Fill out the session code below to generate 12 months of data based off the last 12 months of data from the training set. The hardest part about this is adjusting the arrays with their shapes and sizes. Reference the lecture for hints.
Can you write Python code for it?

with tf.Session() as sess:

    # Use your Saver instance to restore your saved rnn time series model
    saver.restore(sess, "./ex_time_series_model")

    # Create a numpy array for your genreative seed from the last 12 months of the
    # training set data. Hint: Just use tail(12) and then pass it to an np.array
    train_seed = list(train_scaled[-12:])

    ## Now create a for loop that
    for iteration in range(12):
        X_batch = np.array(train_seed[-num_time_steps:]).reshape(1, num_time_steps, 1)
        y_pred = sess.run(outputs, feed_dict={X: X_batch})
        train_seed.append(y_pred[0, -1, 0])


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/.ipynb_checkpoints/03-Time-Series-Exercise-Solutions-Final-checkpoint.ipynb".
The first task is:
Plot out the time series data.
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/.ipynb_checkpoints/03-Time-Series-Exercise-Solutions-Final-checkpoint.ipynb".
The first task is:
Plot out the two columns for comparison.
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/.ipynb_checkpoints/03-Time-Series-Exercise-Solutions-Final-checkpoint.ipynb".
The first task is:
Session
 Run a tf.Session that trains on the batches created by your next_batch function. Also add an a loss evaluation for every 100 training iterations. Remember to save your model after you are done training.
Can you write Python code for it?

gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.9)

with tf.Session(config=tf.ConfigProto(gpu_options=gpu_options)) as sess:
    sess.run(init)

    for iteration in range(num_train_iterations):

        X_batch, y_batch = next_batch(train_scaled,batch_size,num_time_steps)
        sess.run(train, feed_dict={X: X_batch, y: y_batch})

        if iteration % 100 == 0:

            mse = loss.eval(feed_dict={X: X_batch, y: y_batch})
            print(iteration, "\tMSE:", mse)

    # Save Model for Later
    saver.save(sess, "./ex_time_series_model")


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/.ipynb_checkpoints/03-Time-Series-Exercise-Solutions-Final-checkpoint.ipynb".
The first task is:
Setting Up The RNN Model
 Import TensorFlow
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/.ipynb_checkpoints/03-Time-Series-Exercise-Solutions-Final-checkpoint.ipynb".
The first task is:
The Constants
 Define the constants in a single cell. You'll need the following (in parenthesis are the values I used in my solution, but you can play with some of these):
* Number of Inputs (1)
* Number of Time Steps (12)
* Number of Neurons per Layer (100)
* Number of Outputs (1)
* Learning Rate (0.003)
* Number of Iterations for Training (4000)
* Batch Size (1)
Can you write Python code for it?

# Just one feature, the time series
num_inputs = 1
# Num of steps in each batch
num_time_steps = 12
# 100 neuron layer, play with this
num_neurons = 100
# Just one output, predicted time series
num_outputs = 1

## You can also try increasing iterations, but decreasing learning rate
# learning rate you can play with this
learning_rate = 0.03
# how many iterations to go through (training steps), you can play with this
num_train_iterations = 4000
# Size of the batch of data
batch_size = 1


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/.ipynb_checkpoints/03-Time-Series-Exercise-Solutions-Final-checkpoint.ipynb".
The first task is:
Train Test Split
 Let's attempt to predict a year's worth of data. (12 months or 12 steps into the future)
 Create a test train split using indexing (hint: use .head() or tail() or .iloc[]). We don't want a random train test split, we want to specify that the test set is the last 3 months of data is the test set, with everything before it is the training.
Can you write Python code for it?

milk.info()

train_set = milk.head(156)

test_set = milk.tail(12)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/.ipynb_checkpoints/03-Time-Series-Exercise-Solutions-Final-checkpoint.ipynb".
The first task is:
Use pandas to read the csv of the monthly-milk-production.csv file and set index_col='Month'
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/.ipynb_checkpoints/03-Time-Series-Exercise-Solutions-Final-checkpoint.ipynb".
The first task is:
View the test_set dataframe.
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/02-Time-Series-Exercise.ipynb".
The first task is:
Batch Function
 We'll need a function that can feed batches of the training data. We'll need to do several things that are listed out as steps in the comments of the function. Remember to reference the previous batch method from the lecture for hints. Try to fill out the function template below, this is a pretty hard step, so feel free to reference the solutions!
Can you write Python code for it?

def next_batch(training_data, batch_size, steps):
    """
    INPUT: Data, Batch Size, Time Steps per batch
    OUTPUT: A tuple of y time series results. y[:,:-1] and y[:,1:]
    """

    # STEP 1: Use np.random.randint to set a random starting point index for the batch.
    # Remember that each batch needs have the same number of steps in it.
    # This means you should limit the starting point to len(data)-steps
    random_start = np.random.randint(0, len(training_data) - steps)

    # STEP 2: Now that you have a starting index you'll need to index the data from
    # the random start to random start + steps + 1. Then reshape this data to be (1,steps+1)
    # Create Y data for time series in the batches
    y_batch = np.array(training_data[random_start : random_start + steps + 1]).reshape(1, steps+1)

    # STEP 3: Return the batches. You'll have two batches to return y[:,:-1] and y[:,1:]
    # You'll need to reshape these into tensors for the RNN to .reshape(-1,steps,1)
    return y_batch[:, :-1].reshape(-1, steps, 1), y_batch[:, 1:].reshape(-1, steps, 1)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/02-Time-Series-Exercise.ipynb".
The first task is:
Create Placeholders for X and y. (You can change the variable names if you want). The shape for these placeholders should be [None,num_time_steps-1,num_inputs] and [None, num_time_steps-1, num_outputs] The reason we use num_time_steps-1 is because each of these will be one step shorter than the original time steps size, because we are training the RNN network to predict one point into the future based on the input sequence.
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/02-Time-Series-Exercise.ipynb".
The first task is:
Initialize the global variables
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/02-Time-Series-Exercise.ipynb".
The first task is:
Make the index a time series by using:
milk.index = pd.to_datetime(milk.index)
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/02-Time-Series-Exercise.ipynb".
The first task is:
Now create the RNN Layer, you have complete freedom over this, use tf.contrib.rnn and choose anything you want, OutputProjectionWrappers, BasicRNNCells, BasicLSTMCells, MultiRNNCell, GRUCell etc... Keep in mind not every combination will work well! (If in doubt, the solutions used an Outputprojection Wrapper around a basic LSTM cell with relu activation.
Can you write Python code for it?

cell = tf.contrib.rnn.OutputProjectionWrapper(tf.contrib.rnn.BasicLSTMCell(num_units = num_neurons, activation = tf.nn.relu), output_size = num_outputs)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/02-Time-Series-Exercise.ipynb".
The first task is:
Now we want to attempt to predict these 12 months of data, using only the training data we had. To do this we will feed in a seed training_instance of the last 12 months of the training_set of data to predict 12 months into the future. Then we will be able to compare our generated 12 months to our actual true historical values from the test set!
Generative Session
NOTE: Recall that our model is really only trained to predict 1 time step ahead, asking it to generate 12 steps is a big ask, and technically not what it was trained to do! Think of this more as generating new values based off some previous pattern, rather than trying to directly predict the future. You would need to go back to the original model and train the model to predict 12 time steps ahead to really get a higher accuracy on the test data. (Which has its limits due to the smaller size of our data set)
 Fill out the session code below to generate 12 months of data based off the last 12 months of data from the training set. The hardest part about this is adjusting the arrays with their shapes and sizes. Reference the lecture for hints.
Can you write Python code for it?

with tf.Session() as sess:

    # Use your Saver instance to restore your saved rnn time series model
    saver.restore(sess, "./checkpoints/ex_time_series_model")

    # Create a numpy array for your genreative seed from the last 12 months of the
    # training set data. Hint: Just use tail(12) and then pass it to an np.array
    train_seed = list(training_set[-12:])

    ## Now create a for loop that
    for iteration in range(12):
        X_batch = np.array(train_seed[-num_time_steps:]).reshape(1, num_time_steps, 1)
        y_pred = sess.run(outputs, feed_dict={X: X_batch})
        train_seed.append(y_pred[0, -1, 0])


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/02-Time-Series-Exercise.ipynb".
The first task is:
Plot out the two columns for comparison.
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/02-Time-Series-Exercise.ipynb".
The first task is:
Predicting Future (Test Data)
 Show the test_set (the last 12 months of your original complete data set)
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/02-Time-Series-Exercise.ipynb".
The first task is:
Scale the Data
 Use sklearn.preprocessing to scale the data using the MinMaxScaler. Remember to only fit_transform on the training data, then transform the test data. You shouldn't fit on the test data as well, otherwise you are assuming you would know about future behavior!
Can you write Python code for it?

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

training_set = scaler.fit_transform(training_set)

test_set_scaled = scaler.transform(test_set)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/02-Time-Series-Exercise.ipynb".
The first task is:
Setting Up The RNN Model
 Import TensorFlow
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/02-Time-Series-Exercise.ipynb".
The first task is:
Show the result of the predictions.
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/02-Time-Series-Exercise.ipynb".
The first task is:
The Constants
 Define the constants in a single cell. You'll need the following (in parenthesis are the values I used in my solution, but you can play with some of these):
* Number of Inputs (1)
* Number of Time Steps (12)
* Number of Neurons per Layer (100)
* Number of Outputs (1)
* Learning Rate (0.03)
* Number of Iterations for Training (4000)
* Batch Size (1)
Can you write Python code for it?

num_inputs = 1

num_time_steps = 12

num_neurons = 100

num_outputs = 1

learning_rate = 0.03

num_train_iter = 4000

batch_size = 1


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/02-Time-Series-Exercise.ipynb".
The first task is:
Train Test Split
 Let's attempt to predict a year's worth of data. (12 months or 12 steps into the future)
 Create a test train split using indexing (hint: use .head() or tail() or .iloc[]). We don't want a random train test split, we want to specify that the test set is the last 12 months of data is the test set, with everything before it is the training.
Can you write Python code for it?

data.info()

training_set = data.head(156)

test_set = data.tail(12)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/03-Time-Series-Exercise-Solutions-Final.ipynb".
The first task is:
Batch Function
 We'll need a function that can feed batches of the training data. We'll need to do several things that are listed out as steps in the comments of the function. Remember to reference the previous batch method from the lecture for hints. Try to fill out the function template below, this is a pretty hard step, so feel free to reference the solutions!
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/03-Time-Series-Exercise-Solutions-Final.ipynb".
The first task is:
Check out the head of the dataframe
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/03-Time-Series-Exercise-Solutions-Final.ipynb".
The first task is:
Create a new column on the test_set called "Generated" and set it equal to the generated results. You may get a warning about this, feel free to ignore it.
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/03-Time-Series-Exercise-Solutions-Final.ipynb".
The first task is:
Create an instance of tf.train.Saver()
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/03-Time-Series-Exercise-Solutions-Final.ipynb".
The first task is:
Grab the portion of the results that are the generated values and apply inverse_transform on them to turn them back into milk production value units (lbs per cow). Also reshape the results to be (12,1) so we can easily add them to the test_set dataframe.
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/03-Time-Series-Exercise-Solutions-Final.ipynb".
The first task is:
Initialize the global variables
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/03-Time-Series-Exercise-Solutions-Final.ipynb".
The first task is:
Loss Function and Optimizer
 Create a Mean Squared Error Loss Function and use it to minimize an AdamOptimizer, remember to pass in your learning rate.
Can you write Python code for it?

loss = tf.reduce_mean(tf.square(outputs - y)) # MSE
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
train = optimizer.minimize(loss)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/03-Time-Series-Exercise-Solutions-Final.ipynb".
The first task is:
Now pass in the cells variable into tf.nn.dynamic_rnn, along with your first placeholder (X)
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/03-Time-Series-Exercise-Solutions-Final.ipynb".
The first task is:
Plot out the time series data.
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/03-Time-Series-Exercise-Solutions-Final.ipynb".
The first task is:
Plot out the two columns for comparison.
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/03-Time-Series-Exercise-Solutions-Final.ipynb".
The first task is:
Predicting Future (Test Data)
 Show the test_set (the last 12 months of your original complete data set)
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/03-Time-Series-Exercise-Solutions-Final.ipynb".
The first task is:
Scale the Data
 Use sklearn.preprocessing to scale the data using the MinMaxScaler. Remember to only fit_transform on the training data, then transform the test data. You shouldn't fit on the test data as well, otherwise you are assuming you would know about future behavior!
Can you write Python code for it?

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

train_scaled = scaler.fit_transform(train_set)

test_scaled = scaler.transform(test_set)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/03-Time-Series-Exercise-Solutions-Final.ipynb".
The first task is:
Session
 Run a tf.Session that trains on the batches created by your next_batch function. Also add an a loss evaluation for every 100 training iterations. Remember to save your model after you are done training.
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/03-Time-Series-Exercise-Solutions-Final.ipynb".
The first task is:
Setting Up The RNN Model
 Import TensorFlow
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/03-Time-Series-Exercise-Solutions-Final.ipynb".
The first task is:
Show the result of the predictions.
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/03-Time-Series-Exercise-Solutions-Final.ipynb".
The first task is:
The Constants
 Define the constants in a single cell. You'll need the following (in parenthesis are the values I used in my solution, but you can play with some of these):
* Number of Inputs (1)
* Number of Time Steps (12)
* Number of Neurons per Layer (100)
* Number of Outputs (1)
* Learning Rate (0.003)
* Number of Iterations for Training (4000)
* Batch Size (1)
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/03-Time-Series-Exercise-Solutions-Final.ipynb".
The first task is:
Train Test Split
 Let's attempt to predict a year's worth of data. (12 months or 12 steps into the future)
 Create a test train split using indexing (hint: use .head() or tail() or .iloc[]). We don't want a random train test split, we want to specify that the test set is the last 3 months of data is the test set, with everything before it is the training.
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/03-Time-Series-Exercise-Solutions-Final.ipynb".
The first task is:
Use pandas to read the csv of the monthly-milk-production.csv file and set index_col='Month'
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/04-Word2Vec.ipynb".
The first task is:
Also check out gensim!
https://radimrehurek.com/gensim/tutorial.html
https://stackoverflow.com/questions/40074412/word2vec-get-nearest-words
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/04-Word2Vec.ipynb".
The first task is:
Constants
Can you write Python code for it?

# Size of the bath
batch_size = 32

# Dimension of embedding vector
embedding_size = 150

# How many words to consider left and right (the bigger, the longer the training)
skip_window = 1

# How many times to reuse an input to generate a label
num_skips = 2

# We pick a random validation set to sample nearest neighbors. Here we limit the
# validation samples to the words that have a low numeric ID, which by
# construction are also the most frequent.

# Random set of words to evaluate similarity on.
valid_size = 16

# Only pick dev samples in the head of the distribution.
valid_window = 100
valid_examples = np.random.choice(valid_window, valid_size, replace = False)

# Number of negative examples to sample.
num_sampled = 64

# Model Learning Rate
learning_rate = 0.01

# How many words in vocab
vocabulary_size = 50000


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/04-Word2Vec.ipynb".
The first task is:
Create Word Data and Vocab
Can you write Python code for it?

def create_counts(vocab_size = 50000):

    # Begin adding vocab counts with Counter
    vocab = [] + Counter(words).most_common(vocab_size )

    # Turn into a numpy array
    vocab = np.array([word for word, _ in vocab])

    dictionary = {word: code for code, word in enumerate(vocab)}
    data = np.array([dictionary.get(word, 0) for word in words])
    return data,vocab

vocab_size = 50000

# This may take awhile
data, vocabulary = create_counts(vocab_size = vocab_size)

data[0]

(words[100],data[100])

vocabulary.shape

vocabulary[np.random.randint(0, 50000)]


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/04-Word2Vec.ipynb".
The first task is:
NCE Loss
Can you write Python code for it?

# Construct the variables for the NCE loss
nce_weights = tf.Variable(tf.truncated_normal([vocabulary_size, embedding_size], stddev = 1.0 / np.sqrt(embedding_size)))
nce_biases = tf.Variable(tf.zeros([vocabulary_size]))

# Compute the average NCE loss for the batch.
# tf.nce_loss automatically draws a new sample of the negative labels each
# time we evaluate the loss.
loss = tf.reduce_mean(
    tf.nn.nce_loss(nce_weights, nce_biases, train_labels, embed,
                   num_sampled, vocabulary_size))


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/04-Word2Vec.ipynb".
The first task is:
Session
Can you write Python code for it?

gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction = 0.9)

# Usually needs to be quite large to get good results,
# training takes a long time!
num_steps = 200001

with tf.Session(config=tf.ConfigProto(gpu_options=gpu_options)) as sess:
    sess.run(init)
    average_loss = 0
    for step in range(num_steps):

        batch_inputs, batch_labels = generate_batch(batch_size, num_skips, skip_window)
        feed_dict = {train_inputs : batch_inputs, train_labels : batch_labels}

        # We perform one update step by evaluating the training op (including it
        # in the list of returned values for session.run()
        empty, loss_val = sess.run([trainer, loss], feed_dict=feed_dict)
        average_loss += loss_val

        if step % 1000 == 0:
            if step > 0:
                average_loss /= 1000
            # The average loss is an estimate of the loss over the last 1000 batches.
            print("Average loss at step ", step, ": ", average_loss)
            average_loss = 0

    final_embeddings = normalized_embeddings.eval()


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/04-Word2Vec.ipynb".
The first task is:
Step 1: The data.
Can you write Python code for it?

data_dir = ".data/word2vec_data/words"
data_url = 'http://mattmahoney.net/dc/text8.zip'

def fetch_words_data(url = data_url, words_data = data_dir):

    # Make the Dir if it does not exist
    os.makedirs(words_data, exist_ok=True)

    # Path to zip file
    zip_path = os.path.join(words_data, "words.zip")

    # If the zip file isn't there, download it from the data url
    if not os.path.exists(zip_path):
        urllib.request.urlretrieve(url, zip_path)

    # Now that the zip file is there, get the data from it
    with zipfile.ZipFile(zip_path) as f:
        data = f.read(f.namelist()[0])

    # Return a list of all the words in the data source.
    return data.decode("ascii").split()

# Use Defaults (this make take awhile!!)
words = fetch_words_data()

# Total words
len(words)

# Random slice of words
words[9000:9040]

for w in words[9000:9040]:
    print(w, end=' ')


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/04-Word2Vec.ipynb".
The first task is:
TSNE

https://lvdmaaten.github.io/tsne/
https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding

Dimensionality reduction to 2-D vectors (down from 150), this takes awhile.
Can you write Python code for it?

type(final_embeddings)

final_embeddings.shape

from  sklearn.manifold import TSNE
tsne = TSNE(perplexity = 30,
            n_components = 2,
            init = 'pca',
            n_iter = 5000)

plot_only = 2000
low_dim_embs = tsne.fit_transform(final_embeddings[:plot_only,:])

labels = [vocabulary[i] for i in range(plot_only)]

low_dim_embs.shape

plot_with_labels(low_dim_embs, labels)

plot_with_labels(low_dim_embs, labels)
plt.xlim(-10,10)
plt.ylim(-10,10)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/04-Word2Vec.ipynb".
The first task is:
Variables
Can you write Python code for it?

# Look up embeddings for inputs.
init_embeds = tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0)
embeddings = tf.Variable(init_embeds)

embed = tf.nn.embedding_lookup(embeddings, train_inputs)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/04-Recurrent-Neural-Networks/04-Word2Vec.ipynb".
The first task is:
Visualizing Results
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

def plot_with_labels(low_dim_embs, labels):
    assert low_dim_embs.shape[0] >= len(labels), "More labels than embeddings"
    plt.figure(figsize=(18, 18))  #in inches
    for i, label in enumerate(labels):
        x, y = low_dim_embs[i,:]
        plt.scatter(x, y)
        plt.annotate(label,
                     xy = (x, y),
                     xytext = (5, 2),
                     textcoords = 'offset points',
                     ha = 'right',
                     va = 'bottom')


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/05-Miscellaneous-Topics/00-Deep-Nets-with-TF-Abstractions.ipynb".
The first task is:
Abstractions
Estimator API
Can you write Python code for it?

import tensorflow as tf

from tensorflow import estimator

X_train.shape

feat_cols = [tf.feature_column.numeric_column("x", shape = [13])]

deep_model = estimator.DNNClassifier(hidden_units = [13,13,13],
                            feature_columns = feat_cols,
                            n_classes = 3,
                            optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01) )

input_fn = estimator.inputs.numpy_input_fn(x = {'x':scaled_x_train},
                                           y = y_train,
                                           shuffle = True,
                                           batch_size = 10,
                                           num_epochs = 5)

deep_model.train(input_fn = input_fn,
                 steps = 500)

input_fn_eval = estimator.inputs.numpy_input_fn(x = {'x': scaled_x_test},
                                                shuffle = False)

preds = list(deep_model.predict(input_fn = input_fn_eval))

predictions = [p['class_ids'][0] for p in preds]

from sklearn.metrics import confusion_matrix, classification_report

print(classification_report(y_test,predictions))


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/05-Miscellaneous-Topics/00-Deep-Nets-with-TF-Abstractions.ipynb".
The first task is:
Activation Function
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/05-Miscellaneous-Topics/00-Deep-Nets-with-TF-Abstractions.ipynb".
The first task is:
Add Layers to the model
Can you write Python code for it?

from tensorflow.contrib.keras import layers

dnn_keras_model.add(layers.Dense(units = 13,
                                 input_dim = 13,
                                 activation = 'relu'))

dnn_keras_model.add(layers.Dense(units = 13,
                                 activation = 'relu'))
dnn_keras_model.add(layers.Dense(units = 13,
                                 activation = 'relu'))

dnn_keras_model.add(layers.Dense(units = 3,
                                 activation = 'softmax'))


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/05-Miscellaneous-Topics/00-Deep-Nets-with-TF-Abstractions.ipynb".
The first task is:
Compile the Model
Can you write Python code for it?

from tensorflow.contrib.keras import losses, optimizers, metrics

losses.sparse_categorical_crossentropy

dnn_keras_model.compile(optimizer = 'adam',
              loss = 'sparse_categorical_crossentropy',
              metrics = ['accuracy'])


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/05-Miscellaneous-Topics/00-Deep-Nets-with-TF-Abstractions.ipynb".
The first task is:
Create Layers
Can you write Python code for it?

hidden1 = fully_connected(X, num_hidden1, activation_fn = actf)

hidden2 = fully_connected(hidden1,num_hidden2,activation_fn = actf)

output = fully_connected(hidden2, num_outputs)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/05-Miscellaneous-Topics/00-Deep-Nets-with-TF-Abstractions.ipynb".
The first task is:
Init
Can you write Python code for it?

init = tf.global_variables_initializer()

training_steps = 1000
with tf.Session() as sess:
    sess.run(init)
    for i in range(training_steps):
        sess.run(train, feed_dict = {X: scaled_x_train,
                                     y_true: y_train})

    # Get Predictions
    logits = output.eval(feed_dict = {X: scaled_x_test})

    preds = tf.argmax(logits,
                      axis = 1)

    results = preds.eval()

from sklearn.metrics import confusion_matrix,classification_report
print(classification_report(results,y_test))


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/05-Miscellaneous-Topics/00-Deep-Nets-with-TF-Abstractions.ipynb".
The first task is:
Optimizer
Can you write Python code for it?

optimizer = tf.train.AdamOptimizer(learning_rate)
train = optimizer.minimize(loss)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/05-Miscellaneous-Topics/00-Deep-Nets-with-TF-Abstractions.ipynb".
The first task is:
Parameters
Can you write Python code for it?

num_feat = 13
num_hidden1 = 13
num_hidden2 = 13
num_outputs = 3
learning_rate = 0.01

import tensorflow as tf
from tensorflow.contrib.layers import fully_connected


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/05-Miscellaneous-Topics/00-Deep-Nets-with-TF-Abstractions.ipynb".
The first task is:
Train Model
Can you write Python code for it?

dnn_keras_model.fit(scaled_x_train,y_train,epochs=50)

predictions = dnn_keras_model.predict_classes(scaled_x_test)

print(classification_report(predictions, y_test))


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/05-Miscellaneous-Topics/00-Deep-Nets-with-TF-Abstractions.ipynb".
The first task is:
Train Test Split
Can you write Python code for it?

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(feat_data,
                                                    labels,
                                                    test_size = 0.3,
                                                   random_state = 101)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/06-Autoencoders/01-Linear-Autoencoder-for-PCA-Exercise.ipynb".
The first task is:
Confirm that your output is now 2 dimensional along the previous axis of 30 features.
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/06-Autoencoders/01-Linear-Autoencoder-for-PCA-Exercise.ipynb".
The first task is:
Fill out the number of inputs to fit the dimensions of the data set and set the hidden number of units to be 2. Also set the number of outputs to match the number of inputs. Also choose a learning_rate value.
Can you write Python code for it?

num_inputs = 30 # FILL ME IN
num_hidden = 2 # FILL ME IN
num_outputs = num_inputs # Must be true for an autoencoder!

learning_rate = 0.01 #FILL ME IN


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/06-Autoencoders/01-Linear-Autoencoder-for-PCA-Exercise.ipynb".
The first task is:
Init
 Create an instance of a global variable intializer.
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/06-Autoencoders/01-Linear-Autoencoder-for-PCA-Exercise.ipynb".
The first task is:
Layers
 Create the hidden layer and the output layers using the fully_connected function. Remember that to perform PCA there is no activation function.
Can you write Python code for it?

hidden_layer = fully_connected(inputs = X,
                               num_outputs = num_hidden,
                               activation_fn = None)
outputs = fully_connected(inputs = hidden_layer,
                         num_outputs = num_outputs,
                         activation_fn = None)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/06-Autoencoders/01-Linear-Autoencoder-for-PCA-Exercise.ipynb".
The first task is:
Loss Function
 Create a Mean Squared Error loss function.
Can you write Python code for it?

loss = tf.reduce_mean(tf.square(outputs - X))


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/06-Autoencoders/01-Linear-Autoencoder-for-PCA-Exercise.ipynb".
The first task is:
Now plot out the reduced dimensional representation of the data. Do you still have clear separation of classes even with the reduction in dimensions? Hint: You definitely should, the classes should still be clearly seperable, even when reduced to 2 dimensions.
Can you write Python code for it?

plt.scatter(output_2d[:, 0],
            output_2d[:, 1],
            c = data['Label'])


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/06-Autoencoders/01-Linear-Autoencoder-for-PCA-Exercise.ipynb".
The first task is:
Running the Session
 Now create a Tensorflow session that runs the optimizer for at least 1000 steps. (You can also use epochs if you prefer, where 1 epoch is defined by one single run through the entire dataset.
Can you write Python code for it?

num_steps = 1000

with tf.Session() as sess:
    sess.run(init)
    for iteration in range(num_steps):
        sess.run(train,
                 feed_dict = {X: X_data})

    # Now ask for the hidden layer output (the 2 dimensional output)
    output_2d = hidden_layer.eval(feed_dict = {X: X_data})


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/06-Autoencoders/01-Linear-Autoencoder-for-PCA-Exercise.ipynb".
The first task is:
Scale the Data
 Use scikit learn to scale the data with a MinMaxScaler. Remember not to scale the Label column, just the data. Save this scaled data as a new variable called scaled_data.
Can you write Python code for it?

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

X_data = scaler.fit_transform(data.drop('Label', axis = 1))

pd.DataFrame(X_data, columns = data.columns[:-1]).describe()


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/06-Autoencoders/01-Linear-Autoencoder-for-PCA-Exercise.ipynb".
The first task is:
The Linear Autoencoder
 Import tensorflow and import fully_connected layers from tensorflow.contrib.layers.
Can you write Python code for it?

import tensorflow as tf
from tensorflow.contrib.layers import fully_connected


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/06-Autoencoders/02-Stacked-Autoencoder-Example.ipynb".
The first task is:
Activation Function and Layers
Can you write Python code for it?

act_func = tf.nn.relu

hid_layer1 = act_func(tf.matmul(X, w1) + b1)
hid_layer2 = act_func(tf.matmul(hid_layer1, w2) + b2)
hid_layer3 = act_func(tf.matmul(hid_layer2, w3) + b3)
output_layer = tf.matmul(hid_layer3, w4) + b4


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/06-Autoencoders/02-Stacked-Autoencoder-Example.ipynb".
The first task is:
Activation function
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/06-Autoencoders/02-Stacked-Autoencoder-Example.ipynb".
The first task is:
Intialize Variables
Can you write Python code for it?

init = tf.global_variables_initializer()

saver = tf.train.Saver()

num_epochs = 5
batch_size = 150

with tf.Session() as sess:
    sess.run(init)

    # Epoch == Entire Training Set
    for epoch in range(num_epochs):
        num_batches = mnist.train.num_examples // batch_size

        # 150 batch size
        for iteration in range(num_batches):
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(train,
                     feed_dict = {X: X_batch})

        training_loss = loss.eval(feed_dict={X: X_batch})

        print("Epoch {} Complete. Training Loss: {}".format(epoch, training_loss))

    saver.save(sess, "./checkpoint/stacked_autoencoder.ckpt")


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/06-Autoencoders/02-Stacked-Autoencoder-Example.ipynb".
The first task is:
Parameters
Can you write Python code for it?

num_inputs = 784 # 28*28
neurons_hid1 = 392
neurons_hid2 = 196
neurons_hid3 = neurons_hid1 # Decoder Begins
num_outputs = num_inputs

learning_rate = 0.01


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/06-Autoencoders/02-Stacked-Autoencoder-Example.ipynb".
The first task is:
Weights
Initializer capable of adapting its scale to the shape of weights tensors.
With distribution="normal", samples are drawn from a truncated normal
distribution centered on zero, with stddev = sqrt(scale / n)
where n is:
  - number of input units in the weight tensor, if mode = "fan_in"
  - number of output units, if mode = "fan_out"
  - average of the numbers of input and output units, if mode = "fan_avg"
With distribution="uniform", samples are drawn from a uniform distribution
within [-limit, limit], with limit = sqrt(3 * scale / n).
Can you write Python code for it?

initializer = tf.variance_scaling_initializer()

w1 = tf.Variable(initializer([num_inputs, neurons_hid1]),
                 dtype = tf.float32)
w2 = tf.Variable(initializer([neurons_hid1, neurons_hid2]),
                 dtype = tf.float32)
w3 = tf.Variable(initializer([neurons_hid2, neurons_hid3]),
                 dtype = tf.float32)
w4 = tf.Variable(initializer([neurons_hid3, num_outputs]),
                 dtype = tf.float32)


I am working on the file "18-10-30-Think-Stats-by-Allen-B-Downey/Notes.ipynb".
The first task is:
Chapter 03
Probability Mass Function
Probability mass function maps each value to its probability.
Probability of a group always adds to one.
Can you write Python code for it?

s = pd.Series([1, 2, 3, 4, 2])

def pmf(series: pd.Series) -> pd.Series:
    return series.value_counts().sort_index() / series.count()

pmf(s)


I am working on the file "18-10-30-Think-Stats-by-Allen-B-Downey/Notes.ipynb".
The first task is:
Chapter 04
Percentile Rank
Percentile rank is a metric that presents how big is the subset of the data that the values in the subset
are equal or below any given value.
Can you write Python code for it?

# Data
a = [1, 2, 3, 3, 4, 5]

stats.percentileofscore(a=a, score=2.5)

stats.percentileofscore(a=a, score=2)

stats.percentileofscore(a=a, score=3)


I am working on the file "18-10-30-Think-Stats-by-Allen-B-Downey/Notes.ipynb".
The first task is:
Chapter 06
Moments
$k^{th}$ central moment is given as:
$$ m_{k} = \frac{1}{n}\sum_{i}^{ }(x_{i} - \bar{x})^{k} $$
Second order momentu, when $k = 2 $, is the variance.
If the measured value is in f.e. $cm$, the first moment is also in $cm$, but the second is in $cm^{2}$, the third in $cm^{3}$, the forth $cm^{4}$, and so on.
Skewness
Skewness is a property that describes the shape of a distribution.

If the distribution is focused around its central tendency, it is unskewed.
If the values focues on the left of the central tendency, it is described as "left skewed".
If the values focues on the right of the central tendency, it is called "right skewed".

Pearson's median skewness coefficient
Pearson's median skewness coefficient is a measure of skewness baed on the difference between the sample mean and median:
$$ g_{p}=3 \frac{(\bar x - \tilde{x})}{\sigma} $$
where:
- $ \tilde{x} $ is the median
- $ \sigma $ is the standard deviation
Robustness of a statistic
A statistic is robust if the outliers have relatively small impact on the value of the statistic.
Chapter 07
Correlation
A correlation is a statistic intended to quantify the strength of the relationship between two variables.
Few challenges present themselves during such analysis:
- usually the variables have different units
- usually the variables come from different distributions
There are two common approaches trying to solve the challenges:
- Transforming each value to a standard score (example: Pearson product-moment correlation coefficient)
- Transforming each value to a rank (example: Spearman rank correlation coefficient)
Covariance
Covariance is a measurement of the tendency of two variables to vary together.
It is given as:
$$ Cov(X, Y) = \frac{1}{n-1}\sum (x_{i} - \bar x)(y_{i} - \bar y) $$
where:
- X and Y are two series of the same lengths
Can you write Python code for it?


I am working on the file "18-10-30-Think-Stats-by-Allen-B-Downey/Notes.ipynb".
The first task is:
Chapter 08
Mean Squared Error
Mean squared error is a way to measure a quality of an estimator.
It is important to mention that it is very sensitive to outliers and large values.
$$ MSE = \frac{1}{n}\sum (Y_{i}-\hat{Y_{i}})^{2} $$
Can you write Python code for it?

from sklearn.metrics import mean_squared_error
y_true = [3, -0.5, 2, 7]
y_pred = [2.5, 0.0, 2, 8]

mean_squared_error(y_true, y_pred)


I am working on the file "18-10-30-Think-Stats-by-Allen-B-Downey/Notes.ipynb".
The first task is:
DataFrame Indexing
Can you write Python code for it?

array = np.random.randn(4, 2)
array

df = pd.DataFrame(array)
df

columns = ['A', 'B']
df = pd.DataFrame(data=array,
                  columns=columns)
df

index = ['a', 'b', 'c', 'd']
df = pd.DataFrame(data=array,
                  columns=columns,
                  index=index)
df

df['A']

df.loc['a']

df.iloc[0]

indices = ['a', 'c']
df.loc[indices]

df['a':'c']

df[0:2]

df[:2]

df['A'].loc['a']


I am working on the file "18-10-30-Think-Stats-by-Allen-B-Downey/Notes.ipynb".
The first task is:
Different scale below:
Can you write Python code for it?

plt.hist(w3, bins = 200)
plt.xlim((-1, 5))
plt.ylim((0, 25000))
plt.show()

plt.hist(w4, bins = 200)
plt.xlim((-1, 5))
plt.ylim((0, 25000))
plt.show()


I am working on the file "18-10-30-Think-Stats-by-Allen-B-Downey/Notes.ipynb".
The first task is:
Effect size - Cohen'd
Having groups G1 and G2, with number of elements given as N1 and N2, the effect size is given as:
$$ Cohen'd = \frac{\overline{G1} - \overline{G2}}{\sqrt{(\sigma (G1) \cdot (N1-1) + \sigma (G2) \cdot (N2-1)) / ((N1-1) + (N2-1))}} $$
Can you write Python code for it?

def effect_size(g1: pd.DataFrame, g2: pd.DataFrame) -> float:
    diff = g1.mean() - g2.mean()
    var_g1, var_g2 = g1.var(ddof=1), g2.var(ddof=1)
    n1, n2 = len(g1), len(g2)

    pooled_var = (var_g1 * (n1 - 1) + var_g2 * (n2 - 1)) / ((n1 - 1) + (n2 - 1))
    cohen_d = diff / np.sqrt(pooled_var)
    return cohen_d


I am working on the file "18-10-30-Think-Stats-by-Allen-B-Downey/Notes.ipynb".
The first task is:
Exponentially-weighted moving average (EWMA)
Another approach is to calculate weighted average where the most recent values has the highest weight and the weights from previous values drop off exponentially.
The span parameter roughly corresponds to the window size of a moving average. It controls how fast the weights drop off, so it determines the number of points that make a non-negligible contribution to each average.
Can you write Python code for it?

ewma2 = season.ewm(span=2).mean()
ewma2.name = 'ewma2'
ewma3 = season.ewm(span=3).mean()
ewma3.name = 'ewma3'
ewma5 = season.ewm(span=5).mean()
ewma5.name = 'ewma5'

plt.figure(figsize=(7, 7))
plt.plot(season)
plt.plot(ewma2)
plt.plot(ewma3)
plt.plot(ewma5)
plt.legend()


I am working on the file "18-10-30-Think-Stats-by-Allen-B-Downey/Notes.ipynb".
The first task is:
Interquartile Range
Interquartile range is the difference between the 7th and 25th percentiles.
It is used as a measure of the spread of a distribution.
Chapter 05
Exponential Distribution
$$ f(x, \lambda ) = \left{\begin{matrix}
\lambda e^{- \lambda x} & x \geq 0
\
0 & x < 0
\end{matrix}\right. $$
Variable $ \lambda $ defines the shape of the distribution.
The exponential distribution is used analyzing a series of events and measure times between them.
If the events are equally likely to occur at any time, the distribution of inverarrival times tends to look like an exponential distribution.
Can you write Python code for it?

r1 = np.random.exponential(scale=1.0, size=100000)
r2 = np.random.exponential(scale=0.5, size=100000)

plt.hist(r1, bins = 200)
plt.xlim((0, 10))
plt.show()

plt.hist(r2, bins = 200)
plt.xlim((0, 10))
plt.show()


I am working on the file "18-10-30-Think-Stats-by-Allen-B-Downey/Notes.ipynb".
The first task is:
It can be plotted using .hist()
Can you write Python code for it?


I am working on the file "18-10-30-Think-Stats-by-Allen-B-Downey/Notes.ipynb".
The first task is:
Pearson's Correlation
The Pearson's correlation is computed by dividing the deviations by the standard deviations:
$$p = \frac{Conv(X, Y)}{\sigma_{X} \sigma_{Y}}$$
Pearson's correlations ia always between -1 and +1.
If the value $p$ is positive, the correlated values change is similar manner, when one is high, the other one tends to be high as well, when one is low, the other one tends to be low.
If the value $p$ is positive, the correlated values change is similar manner, when one is high, the other one tends to be high as well, when one is low, the other one tends to be low.
The magnitude of the correlation, $p$, describes the strength of the correlation when 1 is the perfect, positive correlation.
Pearson's correlation works
Spearman's Rank
Spearman's rank is more robust than the Pearson's correlations. It mitgates the effect of outliers and skewed distributions.
If the relationship is nonlinear, the Pearson'c correlation tends to underestimate the strength of the relationship.
Can you write Python code for it?


I am working on the file "18-10-30-Think-Stats-by-Allen-B-Downey/Notes.ipynb".
The first task is:
Weibull Distribution
The Weibull Distribution is given as:
$$ f(x ; \lambda, a) = \left{\begin{matrix}
\frac{a}{\lambda}(\frac{x}{\lambda})^{a-1}e^{-(\frac{x}{\lambda})^{a}}
 & x \geq  0\
 0 & x < 0
\end{matrix}\right. $$
where:
- a is the shape
- $\lambda$ is the scale
If the quantity X is a "time-to-failure", the Weibull distribution gives a distribution for which the failure rate is proportional to a power of time.
The shape parameter, a, is that power plus one, and so this parameter can be interpreted directly as follows:
1)  a < 1  - indicates that the failure rate decreases over time (Lindy effect). This happens if there is significant "infant mortality", or defective items failing early and the failure rate decreasing over time as the defective items are weeded out of the population. In the context of the diffusion of innovations, this means negative word of mouth: the hazard function is a monotonically decreasing function of the proportion of adopters.
2)  a = 1  - indicates that the failure rate is constant over time. This might suggest random external events are causing mortality, or failure. The Weibull distribution reduces to an exponential distribution.
3)  a > 1  - indicates that the failure rate increases with time. This happens if there is an "aging" process, or parts that are more likely to fail as time goes on. In the context of the diffusion of innovations, this means positive word of mouth: the hazard function is a monotonically increasing function of the proportion of adopters.
Can you write Python code for it?

w1 = np.random.weibull(a=0.8, size=1000000)
w2 = np.random.weibull(a=1, size=1000000)
w3 = np.random.weibull(a=5, size=1000000)
w4 = np.random.weibull(a=10, size=1000000)

plt.hist(w1, bins = 200)
plt.xlim((-1, 15))
plt.ylim((0, 200000))
plt.show()

plt.hist(w2, bins = 200)
plt.xlim((-1, 15))
plt.ylim((0, 200000))
plt.show()


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/01-Introduction to neural networks/06-Gradient-Descent-Algorithm/GradientDescent.ipynb".
The first task is:
Reading and plotting the data
Can you write Python code for it?

data = pd.read_csv('data.csv', header=None)
X = np.array(data[[0,1]])
y = np.array(data[2])
plot_points(X,y)
plt.show()


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/01-Introduction to neural networks/06-Gradient-Descent-Algorithm/GradientDescent.ipynb".
The first task is:
TODO: Implementing the basic functions
Here is your turn to shine. Implement the following formulas, as explained in the text.
- Sigmoid activation function
$$\sigma(x) = \frac{1}{1+e^{-x}}$$

Output (prediction) formula

$$\hat{y} = \sigma(w_1 x_1 + w_2 x_2 + b)$$

Error function

$$Error(y, \hat{y}) = - y \log(\hat{y}) - (1-y) \log(1-\hat{y})$$

The function that updates the weights

$$ w_i \longrightarrow w_i + \alpha (y - \hat{y}) x_i$$
$$ b \longrightarrow b + \alpha (y - \hat{y})$$
Can you write Python code for it?

# Implement the following functions

# Activation (sigmoid) function
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def output_formula(features, weights, bias):
    return sigmoid(np.dot(features, weights) + bias)

def error_formula(y, output):
    return - y*np.log(output) - (1 - y) * np.log(1-output)

def update_weights(x, y, weights, bias, learnrate):
    output = output_formula(x, weights, bias)
    d_error = -(y - output)
    weights -= learnrate * d_error * x
    bias -= learnrate * d_error
    return weights, bias


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/01-Introduction to neural networks/06-Gradient-Descent-Algorithm/GradientDescent.ipynb".
The first task is:
Time to train the algorithm!
When we run the function, we'll obtain the following:
- 10 updates with the current training loss and accuracy
- A plot of the data and some of the boundary lines obtained. The final one is in black. Notice how the lines get closer and closer to the best fit, as we go through more epochs.
- A plot of the error function. Notice how it decreases as we go through more epochs.
Can you write Python code for it?


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/01-Introduction to neural networks/07-Analysing-Student-Data-Example/StudentAdmissions.ipynb".
The first task is:
Splitting the data into Training and Testing
In order to test our algorithm, we'll split the data into a Training and a Testing set. The size of the testing set will be 10% of the total data.
Can you write Python code for it?

sample = np.random.choice(processed_data.index,
                          size=int(len(processed_data) * 0.9),
                          replace=False)
train_data, test_data = processed_data.iloc[sample], processed_data.drop(sample)

print("Number of training samples is", len(train_data))
print("Number of testing samples is", len(test_data))
print(train_data[:10])
print(test_data[:10])


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/01-Introduction to neural networks/07-Analysing-Student-Data-Example/StudentAdmissions.ipynb".
The first task is:
Splitting the data into features and targets (labels)
Now, as a final step before the training, we'll split the data into features (X) and targets (y).
Can you write Python code for it?

features = train_data.drop('admit',
                           axis=1)
targets = train_data['admit']
features_test = test_data.drop('admit',
                               axis=1)
targets_test = test_data['admit']

print(features[:10])
print(targets[:10])


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/01-Introduction to neural networks/07-Analysing-Student-Data-Example/StudentAdmissions.ipynb".
The first task is:
TODO: Backpropagate the error
Now it's your turn to shine. Write the error term. Remember that this is given by the equation $$ -(y-\hat{y}) \sigma'(x) $$
Can you write Python code for it?

# TODO: Write the error term formula
def error_term_formula(y, output):
    # Output is equal to sigmoid(x) so sigmoid_prime will be equal to (1 - output) * output
    return (y - output) * output * (1 - output)

# Neural Network hyperparameters
epochs = 1000
learnrate = 0.5

# Training function
def train_nn(features, targets, epochs, learnrate):

    # Use to same seed to make debugging easier
    np.random.seed(42)

    n_records, n_features = features.shape
    last_loss = None

    # Initialize weights
    weights = np.random.normal(scale=1 / n_features**.5, size=n_features)

    for e in range(epochs):
        del_w = np.zeros(weights.shape)
        for x, y in zip(features.values, targets):
            # Loop through all records, x is the input, y is the target

            # Activation of the output unit
            #   Notice we multiply the inputs and the weights here
            #   rather than storing h as a separate variable
            output = sigmoid(np.dot(x, weights))

            # The error, the target minus the network output
            error = error_formula(y, output)

            # The error term
            #   Notice we calulate f'(h) here instead of defining a separate
            #   sigmoid_prime function. This just makes it faster because we
            #   can re-use the result of the sigmoid function stored in
            #   the output variable
            error_term = error_term_formula(y, output)

            # The gradient descent step, the error times the gradient times the inputs
            del_w += error_term * x

        # Update the weights here. The learning rate times the
        # change in weights, divided by the number of records to average
        weights += learnrate * del_w / n_records

        # Printing out the mean square error on the training set
        if e % (epochs / 10) == 0:
            out = sigmoid(np.dot(features, weights))
            loss = np.mean((out - targets) ** 2)
            print("Epoch:", e)
            if last_loss and last_loss...


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/01-Introduction to neural networks/07-Analysing-Student-Data-Example/StudentAdmissions.ipynb".
The first task is:
TODO: Scaling the data
The next step is to scale the data. We notice that the range for grades is 1.0-4.0, whereas the range for test scores is roughly 200-800, which is much larger. This means our data is skewed, and that makes it hard for a neural network to handle. Let's fit our two features into a range of 0-1, by dividing the grades by 4.0, and the test score by 800.
Can you write Python code for it?

# Making a copy of our data
processed_data = one_hot_data[:]

# TODO: Scale the columns
processed_data = processed_data / processed_data.max()

# Printing the first 10 rows of our procesed data
processed_data[:10]


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/01-Introduction to neural networks/07-Analysing-Student-Data-Example/StudentAdmissions.ipynb".
The first task is:
This looks more promising, as it seems that the lower the rank, the higher the acceptance rate. Let's use the rank as one of our inputs. In order to do this, we should one-hot encode it.
TODO: One-hot encoding the rank
Use the get_dummies function in Pandas in order to one-hot encode the data.
Can you write Python code for it?

# TODO:  Make dummy variables for rank
one_hot_data = pd.concat(objs=[data, pd.get_dummies(data=data['rank'],
                                                    prefix='rank')],
                         axis=1)

# TODO: Drop the previous rank column
one_hot_data = one_hot_data.drop(labels='rank',
                                 axis=1)

# Print the first 10 rows of our data
one_hot_data[:10]


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/01-Introduction to neural networks/07-Analysing-Student-Data-Example/StudentAdmissions.ipynb".
The first task is:
Training the 2-layer Neural Network
The following function trains the 2-layer neural network. First, we'll write some helper functions.
Can you write Python code for it?

# Activation (sigmoid) function
def sigmoid(x):
    return 1 / (1 + np.exp(-x))
def sigmoid_prime(x):
    return sigmoid(x) * (1 - sigmoid(x))
def error_formula(y, output):
    return - y * np.log(output) - (1 - y) * np.log(1 - output)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 1 - Tensors in PyTorch.ipynb".
The first task is:
Above I generated data we can use to get the output of our simple network. This is all just random for now, going forward we'll start using normal data. Going through each relevant line:
features = torch.randn((1, 5)) creates a tensor with shape (1, 5), one row and five columns, that contains values randomly distributed according to the normal distribution with a mean of zero and standard deviation of one.
weights = torch.randn_like(features) creates another tensor with the same shape as features, again containing values from a normal distribution.
Finally, bias = torch.randn((1, 1)) creates a single value from a normal distribution.
PyTorch tensors can be added, multiplied, subtracted, etc, just like Numpy arrays. In general, you'll use PyTorch tensors pretty much the same way you'd use Numpy arrays. They come with some nice benefits though such as GPU acceleration which we'll get to later. For now, use the generated data to calculate the output of this simple single layer network.

Exercise: Calculate the output of the network with input features features, weights weights, and bias bias. Similar to Numpy, PyTorch has a torch.sum() function, as well as a .sum() method on tensors, for taking sums. Use the function activation defined above as the activation function.
Can you write Python code for it?

features

weights

features * weights

features.numpy()[0][0] * weights.numpy()[0][0]

features.numpy()[0][1] * weights.numpy()[0][1]

## Calculate the output of this network using the weights and bias tensors
### Element by element multiplication then summation
output = activation(torch.sum(features * weights) + bias)
output


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 1 - Tensors in PyTorch.ipynb".
The first task is:
Exercise: Calculate the output for this multi-layer network using the weights W1 & W2, and the biases, B1 & B2.
Can you write Python code for it?

print(W1)
print(W1.shape)

## Your solution here
h1 = torch.mm(features, W1) + B1
h1_activation = activation(h1)

h2 = torch.mm(h1_activation, W2) + B2
output = activation(h2)

output


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 1 - Tensors in PyTorch.ipynb".
The first task is:
Stack them up!
That's how you can calculate the output for a single neuron. The real power of this algorithm happens when you start stacking these individual units into layers and stacks of layers, into a network of neurons. The output of one layer of neurons becomes the input for the next layer. With multiple input units and output units, we now need to express the weights as a matrix.
<img src='assets/multilayer_diagram_weights.png' width=450px>
The first layer shown on the bottom here are the inputs, understandably called the input layer. The middle layer is called the hidden layer, and the final layer (on the right) is the output layer. We can express this network mathematically with matrices again and use matrix multiplication to get linear combinations for each unit in one operation. For example, the hidden layer ($h_1$ and $h_2$ here) can be calculated
$$
\vec{h} = [h_1 \, h_2] =
\begin{bmatrix}
x_1 \, x_2 \cdots \, x_n
\end{bmatrix}
\cdot
\begin{bmatrix}
           w_{11} & w_{12} \
           w_{21} &w_{22} \
           \vdots &\vdots \
           w_{n1} &w_{n2}
\end{bmatrix}
$$
The output for this small network is found by treating the hidden layer as inputs for the output unit. The network output is expressed simply
$$
y =  f_2 ! \left(\, f_1 ! \left(\vec{x} \, \mathbf{W_1}\right) \mathbf{W_2} \right)
$$
Can you write Python code for it?

### Generate some data
torch.manual_seed(7) # Set the random seed so things are predictable

# Features are 3 random normal variables
features = torch.randn((1, 3))

# Define the size of each layer in our network
n_input = features.shape[1]     # Number of input units, must match number of input features
n_hidden = 2                    # Number of hidden units
n_output = 1                    # Number of output units

# Weights for inputs to hidden layer
W1 = torch.randn(n_input, n_hidden)
# Weights for hidden layer to output layer
W2 = torch.randn(n_hidden, n_output)

# and bias terms for hidden and output layers
B1 = torch.randn((1, n_hidden))
B2 = torch.randn((1, n_output))


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 1 - Tensors in PyTorch.ipynb".
The first task is:
You can do the multiplication and sum in the same operation using a matrix multiplication. In general, you'll want to use matrix multiplications since they are more efficient and accelerated using modern libraries and high-performance computing on GPUs.
Here, we want to do a matrix multiplication of the features and the weights. For this we can use torch.mm() or torch.matmul() which is somewhat more complicated and supports broadcasting. If we try to do it with features and weights as they are, we'll get an error
```python

torch.mm(features, weights)

RuntimeError                              Traceback (most recent call last)
<ipython-input-13-15d592eb5279> in <module>()
----> 1 torch.mm(features, weights)
RuntimeError: size mismatch, m1: [1 x 5], m2: [1 x 5] at /Users/soumith/minicondabuild3/conda-bld/pytorch_1524590658547/work/aten/src/TH/generic/THTensorMath.c:2033
```
As you're building neural networks in any framework, you'll see this often. Really often. What's happening here is our tensors aren't the correct shapes to perform a matrix multiplication. Remember that for matrix multiplications, the number of columns in the first tensor must equal to the number of rows in the second column. Both features and weights have the same shape, (1, 5). This means we need to change the shape of weights to get the matrix multiplication to work.
Note: To see the shape of a tensor called tensor, use tensor.shape. If you're building neural networks, you'll be using this method often.
There are a few options here: weights.reshape(), weights.resize_(), and weights.view().

weights.reshape(a, b) will return a new tensor with the same data as weights with size (a, b) sometimes, and sometimes a clone, as in it copies the data to another part of memory.
weights.resize_(a, b) returns the same tensor with a different shape. However, if the new shape results in fewer elements than the original tensor, some elements will be removed from the tensor (but not from memory). If the new shape results in more elements than the...
Can you write Python code for it?

features.shape

## Calculate the output of this network using matrix multiplication
output = activation(torch.mm(features, weights.view([5, 1])) + bias)

output


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 2 - Neural Networks in PyTorch.ipynb".
The first task is:
Activation functions
So far we've only been looking at the softmax activation, but in general any function can be used as an activation function. The only requirement is that for a network to approximate a non-linear function, the activation functions must be non-linear. Here are a few more examples of common activation functions: Tanh (hyperbolic tangent), and ReLU (rectified linear unit).

In practice, the ReLU function is used almost exclusively as the activation function for hidden layers.
Your Turn to Build a Network

Exercise: Create a network with 784 input units, a hidden layer with 128 units and a ReLU activation, then a hidden layer with 64 units and a ReLU activation, and finally an output layer with a softmax activation as shown above. You can use a ReLU activation with the nn.ReLU module or F.relu function.
Can you write Python code for it?

## Your solution here
class MyNetwork(nn.Module):
    def __init__(self):
        super().__init__()

        # Network parameters
        self.h1 = nn.Linear(in_features=784,
                            out_features=128)
        self.h2 = nn.Linear(in_features=128,
                            out_features=64)
        self.out = nn.Linear(in_features=64,
                             out_features=10)
        # Define sigmoid activation and softmax output
        self.relu = nn.ReLU()
        self.softmax = nn.Softmax(dim=1)

    def forward(self, x: torch.tensor):
        x = self.h1(x)
        x = self.relu(x)
        x = self.h2(x)
        x = self.relu(x)
        x = self.out(x)
        x = self.softmax(x)
        return x

model = MyNetwork()
model


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 2 - Neural Networks in PyTorch.ipynb".
The first task is:
As you can see above, our network has basically no idea what this digit is. It's because we haven't trained it yet, all the weights are random!
Using nn.Sequential
PyTorch provides a convenient way to build networks like this where a tensor is passed sequentially through operations, nn.Sequential (documentation). Using this to build the equivalent network:
Can you write Python code for it?

# Hyperparameters for our network
input_size = 784
hidden_sizes = [128, 64]
output_size = 10

# Build a feed-forward network
model = nn.Sequential(nn.Linear(input_size, hidden_sizes[0]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[0], hidden_sizes[1]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[1], output_size),
                      nn.Softmax(dim=1))
print(model)

# Forward pass through the network and display output
images, labels = next(iter(trainloader))
images.resize_(images.shape[0], 1, 784)
ps = model.forward(images[0,:])
helper.view_classify(images[0].view(1, 28, 28), ps)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 2 - Neural Networks in PyTorch.ipynb".
The first task is:
Building networks with PyTorch
PyTorch provides a module nn that makes building networks much simpler. Here I'll show you how to build the same one as above with 784 inputs, 256 hidden units, 10 output units and a softmax output.
Can you write Python code for it?

from torch import nn

class Network(nn.Module):
    def __init__(self):
        # Important!
        # Without it, PyTorch will not be able to follow the setup!
        super().__init__()

        # Inputs to hidden layer linear transformation
        self.hidden = nn.Linear(784, 256)
        # Output layer, 10 units - one for each digit
        self.output = nn.Linear(256, 10)

        # Define sigmoid activation and softmax output
        self.sigmoid = nn.Sigmoid()
        self.softmax = nn.Softmax(dim=1)

    def forward(self, x):
        # Pass the input tensor through each of our operations
        x = self.hidden(x)
        x = self.sigmoid(x)
        x = self.output(x)
        x = self.softmax(x)

        return x


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 2 - Neural Networks in PyTorch.ipynb".
The first task is:
First, let's try to build a simple network for this dataset using weight matrices and matrix multiplications. Then, we'll see how to do it using PyTorch's nn module which provides a much more convenient and powerful method for defining network architectures.
The networks you've seen so far are called fully-connected or dense networks. Each unit in one layer is connected to each unit in the next layer. In fully-connected networks, the input to each layer must be a one-dimensional vector (which can be stacked into a 2D tensor as a batch of multiple examples). However, our images are 28x28 2D tensors, so we need to convert them into 1D vectors. Thinking about sizes, we need to convert the batch of images with shape (64, 1, 28, 28) to a have a shape of (64, 784), 784 is 28 times 28. This is typically called flattening, we flattened the 2D images into 1D vectors.
Previously you built a network with one output unit. Here we need 10 output units, one for each digit. We want our network to predict the digit shown in an image, so what we'll do is calculate probabilities that the image is of any one digit or class. This ends up being a discrete probability distribution over the classes (digits) that tells us the most likely class for the image. That means we need 10 output units for the 10 classes (digits). We'll see how to convert the network output into a probability distribution next.

Exercise: Flatten the batch of images images. Then build a multi-layer network with 784 input units, 256 hidden units, and 10 output units using random tensors for the weights and biases. For now, use a sigmoid activation for the hidden layer. Leave the output layer without an activation, we'll add one that gives us a probability distribution next.
Can you write Python code for it?

def sigmoid_activation(x: torch.tensor):
    return 1 / (1 + torch.exp(-x))

# Flattening the input image to have size 784 x 1
inputs = images.view(images.shape[0], -1)
inputs.shape

# Network parameters
num_hidden_units = 256
W1 = torch.randn(784, num_hidden_units)
b1 = torch.randn(num_hidden_units)

# 10 output units
W2 = torch.randn(num_hidden_units, 10)
b2 = torch.randn(10)

hidden = sigmoid_activation(torch.mm(inputs, W1) + b1)
out = torch.mm(hidden, W2) + b2

out.shape


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 2 - Neural Networks in PyTorch.ipynb".
The first task is:
For custom initialization, we want to modify these tensors in place. These are actually autograd Variables, so we need to get back the actual tensors with model.fc1.weight.data. Once we have the tensors, we can fill them with zeros (for biases) or random normal values.
Can you write Python code for it?

# Set biases to all zeros
model.h1.bias.data.fill_(0)

# sample from random normal with standard dev = 0.01
model.h1.weight.data.normal_(std=0.01)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 2 - Neural Networks in PyTorch.ipynb".
The first task is:
Forward pass
Now that we have a network, let's see what happens when we pass in an image.
Can you write Python code for it?

# Grab some data
dataiter = iter(trainloader)
images, labels = dataiter.next()

# Resize images into a 1D vector, new shape is (batch size, color channels, image pixels)
images.resize_(64, 1, 784)
# or images.resize_(images.shape[0], 1, 784) to automatically get batch size

# Forward pass through the network
img_idx = 0
ps = model.forward(images[img_idx,:])

img = images[img_idx]
helper.view_classify(img.view(1, 28, 28), ps)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 2 - Neural Networks in PyTorch.ipynb".
The first task is:
Initializing weights and biases
The weights and such are automatically initialized for you, but it's possible to customize how they are initialized. The weights and biases are tensors attached to the layer you defined, you can get them with model.fc1.weight for instance.
Can you write Python code for it?


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 2 - Neural Networks in PyTorch.ipynb".
The first task is:
Let's go through this bit by bit.
python
class Network(nn.Module):
Here we're inheriting from nn.Module. Combined with super().__init__() this creates a class that tracks the architecture and provides a lot of useful methods and attributes. It is mandatory to inherit from nn.Module when you're creating a class for your network. The name of the class itself can be anything.
python
self.hidden = nn.Linear(784, 256)
This line creates a module for a linear transformation, $x\mathbf{W} + b$, with 784 inputs and 256 outputs and assigns it to self.hidden. The module automatically creates the weight and bias tensors which we'll use in the forward method. You can access the weight and bias tensors once the network once it's create at net.hidden.weight and net.hidden.bias.
python
self.output = nn.Linear(256, 10)
Similarly, this creates another linear transformation with 256 inputs and 10 outputs.
python
self.sigmoid = nn.Sigmoid()
self.softmax = nn.Softmax(dim=1)
Here I defined operations for the sigmoid activation and softmax output. Setting dim=1 in nn.Softmax(dim=1) calculates softmax across the columns.
python
def forward(self, x):
PyTorch networks created with nn.Module must have a forward method defined. It takes in a tensor x and passes it through the operations you defined in the __init__ method.
python
x = self.hidden(x)
x = self.sigmoid(x)
x = self.output(x)
x = self.softmax(x)
Here the input tensor x is passed through each operation a reassigned to x. We can see that the input tensor goes through the hidden layer, then a sigmoid function, then the output layer, and finally the softmax function. It doesn't matter what you name the variables here, as long as the inputs and outputs of the operations match the network architecture you want to build. The order in which you define things in the __init__ method doesn't matter, but you'll need to sequence the operations correctly in the forward method.
Now we can create a Network object.
Can you write Python code for it?

# Create the network and look at it's text representation
model = Network()
model


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 2 - Neural Networks in PyTorch.ipynb".
The first task is:
Now we have 10 outputs for our network. We want to pass in an image to our network and get out a probability distribution over the classes that tells us the likely class(es) the image belongs to. Something that looks like this:
<img src='assets/image_distribution.png' width=500px>
Here we see that the probability for each class is roughly the same. This is representing an untrained network, it hasn't seen any data yet so it just returns a uniform distribution with equal probabilities for each class.
To calculate this probability distribution, we often use the softmax function. Mathematically this looks like
$$
\Large \sigma(x_i) = \cfrac{e^{x_i}}{\sum_k^K{e^{x_k}}}
$$
What this does is squish each input $x_i$ between 0 and 1 and normalizes the values to give you a proper probability distribution where the probabilites sum up to one.

Exercise: Implement a function softmax that performs the softmax calculation and returns probability distributions for each example in the batch. Note that you'll need to pay attention to the shapes when doing this. If you have a tensor a with shape (64, 10) and a tensor b with shape (64,), doing a/b will give you an error because PyTorch will try to do the division across the columns (called broadcasting) but you'll get a size mismatch. The way to think about this is for each of the 64 examples, you only want to divide by one value, the sum in the denominator. So you need b to have a shape of (64, 1). This way PyTorch will divide the 10 values in each row of a by the one value in each row of b. Pay attention to how you take the sum as well. You'll need to define the dim keyword in torch.sum. Setting dim=0 takes the sum across the rows while dim=1 takes the sum across the columns.
Can you write Python code for it?

def softmax(x):
    ## TODO: Implement the softmax function here
    # Using view to obtain one value per row
    # summation across the columns using dim=1
    return torch.exp(x) / torch.sum(torch.exp(x), dim=1).view(-1, 1)

# Here, out should be the output of the network in the previous excercise with shape (64,10)
probabilities = softmax(out)

# Does it have the right shape? Should be (64, 10)
print(probabilities.shape)
# Does it sum to 1?
print(probabilities.sum(dim=1))


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 2 - Neural Networks in PyTorch.ipynb".
The first task is:
Now you can access layers either by integer or the name
Can you write Python code for it?