I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 2 - Neural Networks in PyTorch.ipynb".
The first task is:
This is what one of the images looks like.
Can you write Python code for it?

plt.imshow(images[1].numpy().squeeze(),
           cmap='Greys_r');


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 2 - Neural Networks in PyTorch.ipynb".
The first task is:
We have the training data loaded into trainloader and we make that an iterator with iter(trainloader). Later, we'll use this to loop through the dataset for training, like
python
for image, label in trainloader:
    ## do things with images and labels
You'll notice I created the trainloader with a batch size of 64, and shuffle=True. The batch size is the number of images we get in one iteration from the data loader and pass through our network, often called a batch. And shuffle=True tells it to shuffle the dataset every time we start going through the data loader again. But here I'm just grabbing the first batch so we can check out the data. We can see below that images is just a tensor with size (64, 1, 28, 28). So, 64 images per batch, 1 color channel, and 28x28 images.
Can you write Python code for it?

dataiter = iter(trainloader)
images, labels = dataiter.next()
print(type(images))
print(images.shape)
print(labels.shape)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 2 - Neural Networks in PyTorch.ipynb".
The first task is:
You can define the network somewhat more concisely and clearly using the torch.nn.functional module. This is the most common way you'll see networks defined as many operations are simple element-wise functions. We normally import this module as F, import torch.nn.functional as F.
Can you write Python code for it?

import torch.nn.functional as F

class Network(nn.Module):
    def __init__(self):
        super().__init__()
        # Inputs to hidden layer linear transformation
        self.hidden = nn.Linear(784, 256)
        # Output layer, 10 units - one for each digit
        self.output = nn.Linear(256, 10)

    def forward(self, x):
        # Hidden layer with sigmoid activation
        x = F.sigmoid(self.hidden(x))
        # Output layer with softmax activation
        x = F.softmax(self.output(x), dim=1)

        return x


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 3 - Training Neural Networks.ipynb".
The first task is:
Autograd
Now that we know how to calculate a loss, how do we use it to perform backpropagation? Torch provides a module, autograd, for automatically calculating the gradients of tensors. We can use it to calculate the gradients of all our parameters with respect to the loss. Autograd works by keeping track of operations performed on tensors, then going backwards through those operations, calculating gradients along the way. To make sure PyTorch keeps track of operations on a tensor and calculates the gradients, you need to set requires_grad = True on a tensor. You can do this at creation with the requires_grad keyword, or at any time with x.requires_grad_(True).
You can turn off gradients for a block of code with the torch.no_grad() content:
```python
x = torch.zeros(1, requires_grad=True)

with torch.no_grad():
...     y = x * 2
y.requires_grad
False
```

Also, you can turn on or off gradients altogether with torch.set_grad_enabled(True|False).
The gradients are computed with respect to some variable z with z.backward(). This does a backward pass through the operations that created z.
Can you write Python code for it?

x = torch.randn(size=(2,2),
                requires_grad=True)
print(x)

y = x ** 2
print(y)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 3 - Training Neural Networks.ipynb".
The first task is:
Below we can see the operation that created y, a power operation PowBackward0.
Can you write Python code for it?

## grad_fn shows the function that generated this variable
print(y.grad_fn)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 3 - Training Neural Networks.ipynb".
The first task is:
In my experience it's more convenient to build the model with a log-softmax output using nn.LogSoftmax or F.log_softmax (documentation). Then you can get the actual probabilities by taking the exponential torch.exp(output). With a log-softmax output, you want to use the negative log likelihood loss, nn.NLLLoss (documentation).

Exercise: Build a model that returns the log-softmax as the output and calculate the loss using the negative log likelihood loss. Note that for nn.LogSoftmax and F.log_softmax you'll need to set the dim keyword argument appropriately. dim=0 calculates softmax across the rows, so each column sums to 1, while dim=1 calculates across the columns so each row sums to 1. Think about what you want the output to be and choose dim appropriately.
Can you write Python code for it?

# TODO: Build a feed-forward network
model = nn.Sequential(nn.Linear(in_features=784, out_features=128),
                      nn.ReLU(),
                      nn.Linear(in_features=128, out_features=64),
                      nn.ReLU(),
                      nn.Linear(in_features=64, out_features=10),
                      nn.LogSoftmax(dim=1))

# TODO: Define the loss
criterion = nn.NLLLoss()

### Run this to check your work
# Get our data
images, labels = next(iter(trainloader))
# Flatten images
images = images.view(images.shape[0], -1)

# Forward pass, get our logits
logits = model(images)
# Calculate the loss with the logits and the labels
loss = criterion(logits, labels)

print(loss)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 3 - Training Neural Networks.ipynb".
The first task is:
To calculate the gradients, you need to run the .backward method on a Variable, z for example. This will calculate the gradient for z with respect to x
$$
\frac{\partial z}{\partial x} = \frac{\partial}{\partial x}\left[\frac{1}{n}\sum_i^n x_i^2\right] = \frac{x}{2}
$$
Can you write Python code for it?


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 3 - Training Neural Networks.ipynb".
The first task is:
With the network trained, we can check out it's predictions.
Can you write Python code for it?

%matplotlib inline
import helper

images, labels = next(iter(trainloader))

img = images[0].view(1, 784)
# Turn off gradients to speed up this part
with torch.no_grad():
    logits = model.forward(img)

# Output of the network are logits, need to take softmax for probabilities
ps = F.softmax(logits,
               dim=1)
helper.view_classify(img.view(1, 28, 28),
                     ps)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 4 - Fashion-MNIST.ipynb".
The first task is:
Here we can see one of the images.
Can you write Python code for it?

image, label = next(iter(trainloader))
helper.imshow(image[0,:]);


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 5 - Inference and Validation.ipynb".
The first task is:
Now we can check if the predicted classes match the labels. This is simple to do by equating top_class and labels, but we have to be careful of the shapes. Here top_class is a 2D tensor with shape (64, 1) while labels is 1D with shape (64). To get the equality to work out the way we want, top_class and labels must have the same shape.
If we do
python
equals = top_class == labels
equals will have shape (64, 64), try it yourself. What it's doing is comparing the one element in each row of top_class with each element in labels which returns 64 True/False boolean values for each row.
Can you write Python code for it?

equals = top_class == labels.view(*top_class.shape)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 5 - Inference and Validation.ipynb".
The first task is:
Now we need to calculate the percentage of correct predictions. equals has binary values, either 0 or 1. This means that if we just sum up all the values and divide by the number of values, we get the percentage of correct predictions. This is the same operation as taking the mean, so we can get the accuracy with a call to torch.mean. If only it was that simple. If you try torch.mean(equals), you'll get an error
RuntimeError: mean is not implemented for type torch.ByteTensor
This happens because equals has type torch.ByteTensor but torch.mean isn't implement for tensors with that type. So we'll need to convert equals to a float tensor. Note that when we take torch.mean it returns a scalar tensor, to get the actual value as a float we'll need to do accuracy.item().
Can you write Python code for it?

accuracy = torch.mean(equals.type(torch.FloatTensor))
print(f'Accuracy: {accuracy.item()*100}%')


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 5 - Inference and Validation.ipynb".
The first task is:
The goal of validation is to measure the model's performance on data that isn't part of the training set. Performance here is up to the developer to define though. Typically this is just accuracy, the percentage of classes the network predicted correctly. Other options are precision and recall and top-5 error rate. We'll focus on accuracy here. First I'll do a forward pass with one batch from the test set.
Can you write Python code for it?

model = Classifier()

images, labels = next(iter(testloader))
# Get the class probabilities
ps = torch.exp(model(images))
# Make sure the shape is appropriate, we should get 10 class probabilities for 64 examples
print(ps.shape)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 5 - Inference and Validation.ipynb".
The first task is:
The network is untrained so it's making random guesses and we should see an accuracy around 10%. Now let's train our network and include our validation pass so we can measure how well the network is performing on the test set. Since we're not updating our parameters in the validation pass, we can speed up our code by turning off gradients using torch.no_grad():
```python
turn off gradients
with torch.no_grad():
    # validation pass here
    for images, labels in testloader:
        ...
```

Exercise: Implement the validation loop below and print out the total accuracy after the loop. You can largely copy and paste the code from above, but I suggest typing it in because writing it out yourself is essential for building the skill. In general you'll always learn more by typing it rather than copy-pasting. You should be able to get an accuracy above 80%.
Can you write Python code for it?

model = Classifier()
criterion = nn.NLLLoss()
optimizer = optim.Adam(params=model.parameters(),
                       lr=0.003)

epochs = 50
steps = 0

train_losses, test_losses = [], []
for e in range(epochs):
    running_loss = 0
    for images, labels in trainloader:

        optimizer.zero_grad()

        log_ps = model(images)
        loss = criterion(log_ps, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()

    else:
        ## TODO: Implement the validation pass and print out the validation accuracy
        ### Resetting the validation values each epoch
        test_loss = 0
        accuracy = 0
        with torch.no_grad():
            for images, labels in testloader:
                # Calculate the log probabilities
                log_probs = model(images)
                # Add the loss of the batch to the total loss
                test_loss += criterion(log_probs, labels)

                ps = torch.exp(log_probs)
                # Choose the predicted class with the highest probability
                top_p, top_class = ps.topk(k=1, dim=1)
                 # Calculate number of correct guesses for the batch
                equals = top_class == labels.view(*top_class.shape)
                accuracy += torch.mean(equals.type(torch.FloatTensor))
        train_losses.append(running_loss / len(trainloader))
        test_losses.append(test_loss / len(testloader))
        print("Epoch: {}/{}.. ".format(e + 1, epochs),
              "Training Loss: {:.3f}.. ".format(running_loss / len(trainloader)),
              "Test Loss: {:.3f}.. ".format(test_loss / len(testloader)),
              "Test Accuracy: {:.3f}".format(accuracy / len(testloader)))

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import matplotlib.pyplot as plt

plt.plot(train_losses, label='Training loss')
plt.plot(test_losses, label='Validation loss')
plt.legend(frameon=False)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 5 - Inference and Validation.ipynb".
The first task is:
With the probabilities, we can get the most likely class using the ps.topk method. This returns the $k$ highest values. Since we just want the most likely class, we can use ps.topk(1). This returns a tuple of the top-$k$ values and the top-$k$ indices. If the highest value is the fifth element, we'll get back 4 as the index.
Can you write Python code for it?

top_p, top_class = ps.topk(1, dim=1)
# Look at the most likely classes for the first 10 examples
print(top_class[:10,:])


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 6 - Saving and Loading Models.ipynb".
The first task is:
And to load the state dict in to the network, you do model.load_state_dict(state_dict).
Can you write Python code for it?


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 6 - Saving and Loading Models.ipynb".
The first task is:
Saving and loading networks
As you can imagine, it's impractical to train a network every time you need to use it. Instead, we can save trained networks then load them later to train more or use them for predictions.
The parameters for PyTorch networks are stored in a model's state_dict. We can see the state dict contains the weight and bias matrices for each of our layers.
Can you write Python code for it?

print("Our model: \n\n", model, '\n')
print("The state dict keys: \n\n", model.state_dict().keys())


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 6 - Saving and Loading Models.ipynb".
The first task is:
Seems pretty straightforward, but as usual it's a bit more complicated. Loading the state dict works only if the model architecture is exactly the same as the checkpoint architecture. If I create a model with a different architecture, this fails.
This means we need to rebuild the model exactly as it was when trained. Information about the model architecture needs to be saved in the checkpoint, along with the state dict.
Can you write Python code for it?

checkpoint = {'input_size': 784,
              'output_size': 10,
              'hidden_layers': [each.out_features for each in model.hidden_layers],
              'state_dict': model.state_dict()}

torch.save(checkpoint, './models/checkpoint.pth')


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 6 - Saving and Loading Models.ipynb".
The first task is:
The simplest thing to do is simply save the state dict with torch.save. For example, we can save it to a file 'checkpoint.pth'.
Can you write Python code for it?


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 6 - Saving and Loading Models.ipynb".
The first task is:
Then we can load the state dict with torch.load.
Can you write Python code for it?

state_dict = torch.load('./models/checkpoint.pth')
print(state_dict.keys())


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 6 - Saving and Loading Models.ipynb".
The first task is:
Train a network
To make things more concise here, I moved the model architecture and training code from the last part to a file called fc_model. Importing this, we can easily create a fully-connected network with fc_model.Network, and train the network using fc_model.train. I'll use this model (once it's trained) to demonstrate how we can save and load models.
Can you write Python code for it?

# Create the network, define the criterion and optimizer

model = fc_model.Network(784, 10, [512, 256, 128])
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

fc_model.train(model,
               trainloader,
               testloader,
               criterion,
               optimizer,
               epochs=2)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 8 - Transfer Learning.ipynb".
The first task is:
This model is built out of two main parts, the features and the classifier. The features part is a stack of convolutional layers and overall works as a feature detector that can be fed into a classifier. The classifier part is a single fully-connected layer (classifier): Linear(in_features=1024, out_features=1000). This layer was trained on the ImageNet dataset, so it won't work for our specific problem. That means we need to replace the classifier, but the features will work perfectly on their own. In general, I think about pre-trained networks as amazingly good feature detectors that can be used as the input for simple feed-forward classifiers.
Can you write Python code for it?

# Freeze parameters so we don't backprop through them
for param in model.parameters():
    param.requires_grad = False

from collections import OrderedDict
classifier = nn.Sequential(OrderedDict([
                          ('fc1', nn.Linear(1024, 500)),
                          ('relu', nn.ReLU()),
                          ('fc2', nn.Linear(500, 2)),
                          ('output', nn.LogSoftmax(dim=1))
                          ]))

model.classifier = classifier


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 8 - Transfer Learning.ipynb".
The first task is:
With our model built, we need to train the classifier. However, now we're using a really deep neural network. If you try to train this on a CPU like normal, it will take a long, long time. Instead, we're going to use the GPU to do the calculations. The linear algebra computations are done in parallel on the GPU leading to 100x increased training speeds. It's also possible to train on multiple GPUs, further decreasing training time.
PyTorch, along with pretty much every other deep learning framework, uses CUDA to efficiently compute the forward and backwards passes on the GPU. In PyTorch, you move your model parameters and other tensors to the GPU memory using model.to('cuda'). You can move them back from the GPU with model.to('cpu') which you'll commonly do when you need to operate on the network output outside of PyTorch. As a demonstration of the increased speed, I'll compare how long it takes to perform a forward and backward pass with and without a GPU.
Can you write Python code for it?

import time

for device in ['cpu', 'cuda']:

    criterion = nn.NLLLoss()
    # Only train the classifier parameters, feature parameters are frozen
    optimizer = optim.Adam(model.classifier.parameters(), lr=0.001)

    model.to(device)

    for ii, (inputs, labels) in enumerate(trainloader):

        # Move input and label tensors to the GPU
        inputs, labels = inputs.to(device), labels.to(device)

        start = time.time()

        outputs = model.forward(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        if ii==3:
            break

    print(f"Device = {device}; Time per batch: {(time.time() - start) / 3:.3f} seconds")


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/02-Introduction to PyTorch/Part 8 - Transfer Learning.ipynb".
The first task is:
You can write device agnostic code which will automatically use CUDA if it's enabled like so:
```python
at beginning of the script
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
...
then whenever you get a new Tensor or Module
this won't copy if they are already on the desired device
input = data.to(device)
model = MyModule(...).to(device)
```
From here, I'll let you finish training the model. The process is the same as before except now your model is much more powerful. You should get better than 95% accuracy easily.

Exercise: Train a pretrained models to classify the cat and dog images. Continue with the DenseNet model, or try ResNet, it's also a good model to try out first. Make sure you are only training the classifier and the parameters for the features part are frozen.
Can you write Python code for it?

# Use GPU is available
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

device

## TODO: Use a pretrained model to classify the cat and dog images
### The last layer of ResNet 50 is called fc.
model = models.densenet121(pretrained=True)

# Freezing the layers by turning off the gradients
for param in model.parameters():
    param.requires_grad = False

# Building the classifier part of the network
# It will replace the last part of the network
classifier = nn.Sequential(nn.Linear(in_features=1024, out_features=128),
                           nn.ReLU(),
                           nn.Dropout(p=0.2),
                           nn.Linear(in_features=128, out_features=2),
                           nn.LogSoftmax(dim=1))

model.classifier = classifier

# Defining the loss
criterion = nn.NLLLoss()

# Defining the optimizer
# Optimizing only the fc.part!
optimizer = optim.Adam(params=model.classifier.parameters(),
                       lr=0.003)

# Moving the model to the device
model.to(device)

# Hyperparameter
epochs = 1
steps = 0
running_loss = 0
print_every_steps = 5

for epoch in range(epochs):
    for images, labels in trainloader:
        steps += 1

        # Moving the data to the device
        images = images.to(device)
        labels = labels.to(device)

        # Resetting the gradient accumulator
        optimizer.zero_grad()
        # Calculating the log of the probabilities
        log_probs = model(images)
        # Calculating the loss
        loss = criterion(log_probs, labels)
        # Backward pass
        loss.backward()
        # Applying the backpropagation
        optimizer.step()

        running_loss += loss.item()

        if steps % print_every_steps == 0:
            # Changing mode to the evaluation mode
            # That results in turning of dropout
            model.eval()

            test_loss = 0
            accuracy = 0

            # Test data
            for images_test, labels_test in...


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 1 - Multi-layer Perceptron MNIST.ipynb".
The first task is:
Define the Network Architecture
The architecture will be responsible for seeing as input a 784-dim Tensor of pixel values for each image, and producing a Tensor of length 10 (our number of classes) that indicates the class scores for an input image. This particular example uses two hidden layers and dropout to avoid overfitting.
Can you write Python code for it?

import torch.nn as nn
import torch.nn.functional as F

## TODO: Define the NN architecture
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # Linear layer (784 -> 128 hidden nodes)
        self.fc1 = nn.Linear(in_features=(28 * 28),
                             out_features=128)
        self.fc2 = nn.Linear(in_features=128,
                             out_features=10)

    def forward(self, x):
        # flatten image input
        x = x.view(-1, 28 * 28)
        # add hidden layer, with relu activation function
        x = F.relu(self.fc1(x))
        x = F.softmax(self.fc2(x), dim=1)
        return x

# initialize the NN
model = Net()
print(model)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 1 - Multi-layer Perceptron MNIST.ipynb".
The first task is:
Load and Visualize the Data
Downloading may take a few moments, and you should see your progress as the data is loading. You may also choose to change the batch_size if you want to load more data at a time.
This cell will create DataLoaders for each of our datasets.
Can you write Python code for it?

from torchvision import datasets
import torchvision.transforms as transforms

# number of subprocesses to use for data loading
num_workers = 0
# how many samples per batch to load
batch_size = 20

# convert data to torch.FloatTensor
transform = transforms.ToTensor()

# choose the training and test datasets
train_data = datasets.MNIST(root='data',
                            train=True,
                            download=True,
                            transform=transform)
test_data = datasets.MNIST(root='data',
                           train=False,
                           download=True,
                           transform=transform)

# prepare data loaders
train_loader = torch.utils.data.DataLoader(dataset=train_data,
                                           batch_size=batch_size,
                                           num_workers=num_workers)
test_loader = torch.utils.data.DataLoader(dataset=test_data,
                                          batch_size=batch_size,
                                          num_workers=num_workers)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 1 - Multi-layer Perceptron MNIST.ipynb".
The first task is:
Train the Network
The steps for training/learning from a batch of data are described in the comments below:
1. Clear the gradients of all optimized variables
2. Forward pass: compute predicted outputs by passing inputs to the model
3. Calculate the loss
4. Backward pass: compute gradient of the loss with respect to model parameters
5. Perform a single optimization step (parameter update)
6. Update average training loss
The following loop trains for 30 epochs; feel free to change this number. For now, we suggest somewhere between 20-50 epochs. As you train, take a look at how the values for the training loss decrease over time. We want it to decrease while also avoiding overfitting the training data.
Can you write Python code for it?

# Number of epochs to train the model
n_epochs = 50  # suggest training between 20-50 epochs

# Prep model for training
model.train()

for epoch in range(n_epochs):
    # Monitor training loss
    train_loss = 0.0

    ###################
    # train the model #
    ###################
    for data, target in train_loader:
        # Clear the gradients of all optimized variables
        optimizer.zero_grad()
        # Forward pass: compute predicted outputs by passing inputs to the model
        output = model(data)
        # Calculate the loss
        loss = criterion(output, target)
        # Backward pass: compute gradient of the loss with respect to model parameters
        loss.backward()
        # Perform a single optimization step (parameter update)
        optimizer.step()
        # Update running training loss
        train_loss += loss.item() * data.size(0)

    # Print training statistics
    # Calculate average loss over an epoch
    train_loss = train_loss/len(train_loader.dataset)

    print('Epoch: {} \tTraining Loss: {:.6f}'.format(epoch+1,
                                                     train_loss))


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 1 - Multi-layer Perceptron MNIST.ipynb".
The first task is:
View an Image in More Detail
Can you write Python code for it?

img = np.squeeze(images[1])

fig = plt.figure(figsize = (12,12))
ax = fig.add_subplot(111)
ax.imshow(img, cmap='gray')
width, height = img.shape
thresh = img.max()/2.5
for x in range(width):
    for y in range(height):
        val = round(img[x][y],2) if img[x][y] !=0 else 0
        ax.annotate(str(val), xy=(y,x),
                    horizontalalignment='center',
                    verticalalignment='center',
                    color='white' if img[x][y]<thresh else 'black')


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 1 - Multi-layer Perceptron MNIST.ipynb".
The first task is:
Visualize Sample Test Results
This cell displays test images and their labels in this format: predicted (ground-truth). The text will be green for accurately classified examples and red for incorrect predictions.
Can you write Python code for it?

# obtain one batch of test images
dataiter = iter(test_loader)
images, labels = dataiter.next()

# get sample outputs
output = model(images)
# convert output probabilities to predicted class
_, preds = torch.max(output, 1)
# prep images for display
images = images.numpy()

# plot the images in the batch, along with predicted and true labels
fig = plt.figure(figsize=(25, 4))
for idx in np.arange(20):
    ax = fig.add_subplot(2, 20/2, idx+1, xticks=[], yticks=[])
    ax.imshow(np.squeeze(images[idx]), cmap='gray')
    ax.set_title("{} ({})".format(str(preds[idx].item()), str(labels[idx].item())),
                 color=("green" if preds[idx]==labels[idx] else "red"))


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 2 - Multi-layer Perceptron MNIST with validation.ipynb".
The first task is:
Specify Loss Function and Optimizer
It's recommended that you use cross-entropy loss for classification. If you look at the documentation (linked above), you can see that PyTorch's cross entropy function applies a softmax funtion to the output layer and then calculates the log loss.
Can you write Python code for it?

# Specify loss function (categorical cross-entropy)
criterion = nn.CrossEntropyLoss()

# Specify optimizer (stochastic gradient descent) and learning rate = 0.01
optimizer = torch.optim.SGD(params=model.parameters(),
                            lr=0.01)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 2 - Multi-layer Perceptron MNIST with validation.ipynb".
The first task is:
Train the Network
The steps for training/learning from a batch of data are described in the comments below:
1. Clear the gradients of all optimized variables
2. Forward pass: compute predicted outputs by passing inputs to the model
3. Calculate the loss
4. Backward pass: compute gradient of the loss with respect to model parameters
5. Perform a single optimization step (parameter update)
6. Update average training loss
The following loop trains for 50 epochs; take a look at how the values for the training loss decrease over time. We want it to decrease while also avoiding overfitting the training data.
Can you write Python code for it?

# Number of epochs to train the model
n_epochs = 50

# Initialize tracker for minimum validation loss
valid_loss_min = np.Inf # set initial "min" to infinity

for epoch in range(n_epochs):
    # Monitor training loss
    train_loss = 0.0
    valid_loss = 0.0

    # Prep model for training
    model.train()
    for data, target in train_loader:
        # Clear the gradients of all optimized variables
        optimizer.zero_grad()
        # Forward pass: compute predicted outputs by passing inputs to the model
        output = model(data)
        # Calculate the loss
        loss = criterion(output, target)
        # Backward pass: compute gradient of the loss with respect to model parameters
        loss.backward()
        # Perform a single optimization step (parameter update)
        optimizer.step()
        # Update running training loss
        train_loss += loss.item()*data.size(0)

    # Prep model for evaluation
    model.eval()
    for data, target in valid_loader:
        # Forward pass: compute predicted outputs by passing inputs to the model
        output = model(data)
        # Calculate the loss
        loss = criterion(output, target)
        # Update running validation loss
        valid_loss += loss.item()*data.size(0)

    # Print training/validation statistics
    # Calculate average loss over an epoch
    train_loss = train_loss/len(train_loader.dataset)
    valid_loss = valid_loss/len(valid_loader.dataset)

    print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f}'.format(
        epoch+1,
        train_loss,
        valid_loss
        ))

    # Save model if validation loss has decreased
    if valid_loss <= valid_loss_min:
        print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(
        valid_loss_min,
        valid_loss))
        torch.save(model.state_dict(), './models/model.pth')
        valid_loss_min = valid_loss


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 2 - Multi-layer Perceptron MNIST with validation.ipynb".
The first task is:
View an Image in More Detail
Can you write Python code for it?


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 2 - Multi-layer Perceptron MNIST with validation.ipynb".
The first task is:
Visualize Sample Test Results
This cell displays test images and their labels in this format: predicted (ground-truth). The text will be green for accurately classified examples and red for incorrect predictions.
Can you write Python code for it?

# Obtain one batch of test images
dataiter = iter(test_loader)
images, labels = dataiter.next()

# Get sample outputs
output = model(images)
# Convert output probabilities to predicted class
_, preds = torch.max(output, 1)
# Prep images for display
images = images.numpy()

# Plot the images in the batch, along with predicted and true labels
fig = plt.figure(figsize=(25, 4))
for idx in np.arange(20):
    ax = fig.add_subplot(2, 20/2, idx+1, xticks=[], yticks=[])
    ax.imshow(np.squeeze(images[idx]), cmap='gray')
    ax.set_title("{} ({})".format(str(preds[idx].item()),
                                  str(labels[idx].item())),
                 color=("green" if preds[idx]==labels[idx]
                        else "red"))


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 3 - Finding Edges and Custom Kernels.ipynb".
The first task is:
Convert the image to grayscale
Can you write Python code for it?

# Convert to grayscale for filtering
gray = cv2.cvtColor(src=image,
                    code=cv2.COLOR_RGB2GRAY)

plt.imshow(X=gray, cmap='gray')


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 3 - Finding Edges and Custom Kernels.ipynb".
The first task is:
TODO: Create a custom kernel
Below, you've been given one common type of edge detection filter: a Sobel operator.
The Sobel filter is very commonly used in edge detection and in finding patterns in intensity in an image. Applying a Sobel filter to an image is a way of taking (an approximation) of the derivative of the image in the x or y direction, separately. The operators look as follows.
<img src="images/sobel_ops.png" width=200 height=200>
It's up to you to create a Sobel x operator and apply it to the given image.
For a challenge, see if you can put the image through a series of filters: first one that blurs the image (takes an average of pixels), and then one that detects the edges.
Can you write Python code for it?

# Create a custom kernel

# 3x3 array for edge detection
sobel_y = np.array([[ -1, -2, -1],
                    [ 0, 0, 0],
                    [ 1, 2, 1]])

## TODO: Create and apply a Sobel x operator
sobel_x = np.array([[ -1, 0, 1],
                    [ -2, 0, 2],
                    [ -1, 0, 1]])

# Filter the image using filter2D, which has inputs: (grayscale image, bit-depth, kernel)
filtered_image = cv2.filter2D(src=gray,
                              ddepth=-1,
                              kernel=sobel_y)
filtered_image2 = cv2.filter2D(src=gray,
                               ddepth=-1,
                               kernel=sobel_x)

plt.imshow(X=filtered_image,
           cmap='gray')

plt.imshow(X=filtered_image2,
           cmap='gray')


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 3 - Finding Edges and Custom Kernels.ipynb".
The first task is:
Test out other filters!
You're encouraged to create other kinds of filters and apply them to see what happens! As an optional exercise, try the following:
* Create a filter with decimal value weights.
* Create a 5x5 filter
* Apply your filters to the other images in the images directory.
Can you write Python code for it?

image2 = mpimg.imread(fname='images/white_lines.jpg')
gray2 = cv2.cvtColor(src=image2,
                     code=cv2.COLOR_RGB2GRAY)

sample_filter = np.array([[ 0, 0, 2, 0, 0],
                          [ 0, 0, 1, 0, 0],
                          [ 2, -1, 1, -1, 2],
                          [ 0, 0, 1, 0, 0],
                          [ 0, 0, 2, 0, 0]])

filtered_image3 = cv2.filter2D(src=gray2,
                               ddepth=-1,
                               kernel=sample_filter)

plt.imshow(X=gray2,
           cmap='gray')

plt.imshow(X=filtered_image3,
           cmap='gray')


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 4 - CNN with CIFAR-10.ipynb".
The first task is:
Define the Network Architecture
This time, you'll define a CNN architecture. Instead of an MLP, which used linear, fully-connected layers, you'll use the following:
* Convolutional layers, which can be thought of as stack of filtered images.
* Maxpooling layers, which reduce the x-y size of an input, keeping only the most active pixels from the previous layer.
* The usual Linear + Dropout layers to avoid overfitting and produce a 10-dim output.
A network with 2 convolutional layers is shown in the image below and in the code, and you've been given starter code with one convolutional and one maxpooling layer.
<img src='images/2_layer_conv.png' height=50% width=50% />
TODO: Define a model with multiple convolutional layers, and define the feedforward network behavior.
The more convolutional layers you include, the more complex patterns in color and shape a model can detect. It's suggested that your final model include 2 or 3 convolutional layers as well as linear layers + dropout in between to avoid overfitting.
It's good practice to look at existing research and implementations of related models as a starting point for defining your own models. You may find it useful to look at this PyTorch classification example or this, more complex Keras example to help decide on a final structure.
Output volume for a convolutional layer
To compute the output size of a given convolutional layer we can perform the following calculation (taken from Stanford's cs231n course):

We can compute the spatial size of the output volume as a function of the input volume size (W), the kernel/filter size (F), the stride with which they are applied (S), and the amount of zero padding used (P) on the border. The correct formula for calculating how many neurons define the output_W is given by (W−F+2P)/S+1.

For example for a 7x7 input and a 3x3 filter with stride 1 and pad 0 we would get a 5x5 output. With stride 2 we would get a 3x3 output.
Can you write Python code for it?

import torch.nn as nn
import torch.nn.functional as F

# Define the CNN architecture
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # Convolutional layers
        # Input 32x32x3
        self.conv1 = nn.Conv2d(in_channels=3,
                               out_channels=16,
                               kernel_size=(3, 3),
                               stride=(1, 1),
                               padding=1)
        # Input 16x16x16
        self.conv2 = nn.Conv2d(in_channels=16,
                               out_channels=32,
                               kernel_size=(3, 3),
                               stride=(1, 1),
                               padding=1)
        # Input 8x8x32
        self.conv3 = nn.Conv2d(in_channels=32,
                               out_channels=64,
                               kernel_size=(3, 3),
                               stride=(1, 1),
                               padding=1)
        # Max pooling layer
        self.pool = nn.MaxPool2d(kernel_size=(2, 2),
                                 stride=(2, 2))
        # Dropout
        self.dropout = nn.Dropout(p=0.25)
        # Linear Layers
        # Input flatten 4x4x64 -> 500
        self.fc1 = nn.Linear(in_features=4 * 4 * 64,
                             out_features=128)
        # Input 500, Output 10
        self.fc2 = nn.Linear(in_features=128,
                             out_features=10)

    def forward(self, x):
        # Add sequence of convolutional and max pooling layers
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.pool(F.relu(self.conv3(x)))
        # Flatten the image
        x = x.view(-1, 64 * 4 * 4)
        # First fully-connected layer
        x = F.relu(self.fc1(x))
        # Adding dropout
        x = self.dropout(x)
        # Adding second layer
        x = self.fc2(x)
        return x

# Create a complete CNN
model = Net()
print(model)

# Move tensors to GPU if CUDA is available
if...


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 4 - CNN with CIFAR-10.ipynb".
The first task is:
Specify Loss Function and Optimizer
Decide on a loss and optimization function that is best suited for this classification task. The linked code examples from above, may be a good starting point; this PyTorch classification example or this, more complex Keras example. Pay close attention to the value for learning rate as this value determines how your model converges to a small error.
TODO: Define the loss and optimizer and see how these choices change the loss over time.
Can you write Python code for it?

import torch.optim as optim

# Specify loss function
criterion = nn.CrossEntropyLoss()

# Specify optimizer
optimizer = optim.Adam(params=model.parameters(),
                       lr=0.01)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 4 - CNN with CIFAR-10.ipynb".
The first task is:
Test the Trained Network
Test your trained model on previously unseen data! A "good" result will be a CNN that gets around 70% (or more, try your best!) accuracy on these test images.
Can you write Python code for it?

# track test loss
test_loss = 0.0
class_correct = list(0. for i in range(10))
class_total = list(0. for i in range(10))

model.eval()
# iterate over test data
for data, target in test_loader:
    # move tensors to GPU if CUDA is available
    if train_on_gpu:
        data, target = data.cuda(), target.cuda()
    # forward pass: compute predicted outputs by passing inputs to the model
    output = model(data)
    # calculate the batch loss
    loss = criterion(output, target)
    # update test loss
    test_loss += loss.item() * data.size(0)
    # convert output probabilities to predicted class
    _, pred = torch.max(output, 1)
    # compare predictions to true label
    correct_tensor = pred.eq(target.data.view_as(pred))
    correct = np.squeeze(correct_tensor.numpy()) if not train_on_gpu else np.squeeze(correct_tensor.cpu().numpy())
    # calculate test accuracy for each object class
    for i in range(batch_size):
        label = target.data[i]
        class_correct[label] += correct[i].item()
        class_total[label] += 1

# average test loss
test_loss = test_loss/len(test_loader.dataset)
print('Test Loss: {:.6f}\n'.format(test_loss))

for i in range(10):
    if class_total[i] > 0:
        print('Test Accuracy of %5s: %2d%% (%2d/%2d)' % (classes[i], 100 * class_correct[i] / class_total[i],
                                                         np.sum(class_correct[i]), np.sum(class_total[i])))
    else:
        print('Test Accuracy of %5s: N/A (no training examples)' % (classes[i]))

print('\nTest Accuracy (Overall): %2d%% (%2d/%2d)' % (100. * np.sum(class_correct) / np.sum(class_total),
                                                      np.sum(class_correct), np.sum(class_total)))


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 4 - CNN with CIFAR-10.ipynb".
The first task is:
Train the Network
Remember to look at how the training and validation loss decreases over time; if the validation loss ever increases it indicates possible overfitting.
Can you write Python code for it?

# Number of epochs to train the model
n_epochs = 50

# Initialize validation loss
valid_loss_min = np.Inf

for epoch in range(1, n_epochs+1):

    # Keep track of training and validation loss
    train_loss = 0.0
    valid_loss = 0.0

    model.train()
    for data, target in train_loader:
        # Move tensors to GPU if CUDA is available
        if train_on_gpu:
            data, target = data.cuda(), target.cuda()
        # Clear the gradients of all optimized variables
        optimizer.zero_grad()
        # Forward pass: compute predicted outputs by passing inputs to the model
        output = model(data)
        # Calculate the batch loss
        loss = criterion(output, target)
        # Backward pass: compute gradient of the loss with respect to model parameters
        loss.backward()
        # Perform a single optimization step (parameter update)
        optimizer.step()
        # Update training loss
        train_loss += loss.item() * data.size(0)

    # Validate the model
    model.eval()
    for data, target in valid_loader:
        # Move tensors to GPU if CUDA is available
        if train_on_gpu:
            data, target = data.cuda(), target.cuda()
        # Forward pass: compute predicted outputs by passing inputs to the model
        output = model(data)
        # Calculate the batch loss
        loss = criterion(output, target)
        # Update average validation loss
        valid_loss += loss.item()*data.size(0)

    # Calculate average losses
    train_loss = train_loss/len(train_loader.dataset)
    valid_loss = valid_loss/len(valid_loader.dataset)

    # Print training/validation statistics
    print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f}'.format(
        epoch, train_loss, valid_loss))

    # Save model if validation loss has decreased
    if valid_loss <= valid_loss_min:
        print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(
        valid_loss_min,
        valid_loss))
       ...


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 4 - CNN with CIFAR-10.ipynb".
The first task is:
Visualize a Batch of Training Data
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

# Helper function to un-normalize and display an image
def imshow(img):
    img = img / 2 + 0.5  # Unnormalize
    plt.imshow(np.transpose(img, (1, 2, 0)))  # Convert from Tensor image

# Obtain one batch of training images
dataiter = iter(train_loader)
images, labels = dataiter.next()
images = images.numpy() # convert images to numpy for display

# plot the images in the batch, along with the corresponding labels
fig = plt.figure(figsize=(25, 4))
# Display 20 images
for idx in np.arange(20):
    ax = fig.add_subplot(2, 20/2, idx+1, xticks=[], yticks=[])
    imshow(images[idx])
    ax.set_title(classes[labels[idx]])

images[0].shape


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 5 - Convolution visualisation.ipynb".
The first task is:
Define a convolutional layer
The various layers that make up any neural network are documented, here. For a convolutional neural network, we'll start by defining a:
* Convolutional layer
Initialize a single convolutional layer so that it contains all your created filters. Note that you are not training this network; you are initializing the weights in a convolutional layer so that you can visualize what happens after a forward pass through this network!
__init__ and forward
To define a neural network in PyTorch, you define the layers of a model in the function __init__ and define the forward behavior of a network that applyies those initialized layers to an input (x) in the function forward. In PyTorch we convert all inputs into the Tensor datatype, which is similar to a list data type in Python.
Below, I define the structure of a class called Net that has a convolutional layer that can contain four 3x3 grayscale filters.
Can you write Python code for it?

import torch
import torch.nn as nn
import torch.nn.functional as F

# define a neural network with a single convolutional layer with four filters
class Net(nn.Module):

    def __init__(self, weight):
        super(Net, self).__init__()
        # initializes the weights of the convolutional layer to be the weights of the 4 defined filters
        k_height, k_width = weight.shape[2:]
        # assumes there are 4 grayscale filters
        self.conv = nn.Conv2d(1, 4, kernel_size=(k_height, k_width), bias=False)
        self.conv.weight = torch.nn.Parameter(weight)

    def forward(self, x):
        # calculates the output of a convolutional layer
        # pre- and post-activation
        conv_x = self.conv(x)
        activated_x = F.relu(conv_x)

        # returns both layers
        return conv_x, activated_x

# instantiate the model and set the weights
weight = torch.from_numpy(filters).unsqueeze(1).type(torch.FloatTensor)
model = Net(weight)

# print out the layer in the network
print(model)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 5 - Convolution visualisation.ipynb".
The first task is:
Define and visualize the filters
Can you write Python code for it?

import numpy as np

## TODO: Feel free to modify the numbers here, to try out another filter!
filter_vals = np.array([[-1, -1, 1, 1], [-1, -1, 1, 1], [-1, -1, 1, 1], [-1, -1, 1, 1]])

print('Filter shape: ', filter_vals.shape)

# Defining four different filters,
# all of which are linear combinations of the `filter_vals` defined above

# define four filters
filter_1 = filter_vals
filter_2 = -filter_1
filter_3 = filter_1.T
filter_4 = -filter_3
filters = np.array([filter_1, filter_2, filter_3, filter_4])

# For an example, print out the values of filter 1
print('Filter 1: \n', filter_1)

# visualize all four filters
fig = plt.figure(figsize=(10, 5))
for i in range(4):
    ax = fig.add_subplot(1, 4, i+1, xticks=[], yticks=[])
    ax.imshow(filters[i], cmap='gray')
    ax.set_title('Filter %s' % str(i+1))
    width, height = filters[i].shape
    for x in range(width):
        for y in range(height):
            ax.annotate(str(filters[i][x][y]), xy=(y,x),
                        horizontalalignment='center',
                        verticalalignment='center',
                        color='white' if filters[i][x][y]<0 else 'black')


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 5 - Convolution visualisation.ipynb".
The first task is:
Let's look at the output of a convolutional layer, before and after a ReLu activation function is applied.
Can you write Python code for it?

# plot original image
plt.imshow(gray_img, cmap='gray')

# visualize all filters
fig = plt.figure(figsize=(12, 6))
fig.subplots_adjust(left=0, right=1.5, bottom=0.8, top=1, hspace=0.05, wspace=0.05)
for i in range(4):
    ax = fig.add_subplot(1, 4, i+1, xticks=[], yticks=[])
    ax.imshow(filters[i], cmap='gray')
    ax.set_title('Filter %s' % str(i+1))

# convert the image into an input Tensor
gray_img_tensor = torch.from_numpy(gray_img).unsqueeze(0).unsqueeze(1)

# get the convolutional layer (pre and post activation)
conv_layer, activated_layer = model(gray_img_tensor)

# visualize the output of a conv layer
viz_layer(conv_layer)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 5 - Convolution visualisation.ipynb".
The first task is:
ReLu activation
In this model, we've used an activation function that scales the output of the convolutional layer. We've chose a ReLu function to do this, and this function simply turns all negative pixel values in 0's (black). See the equation pictured below for input pixel values, x.
<img src='images/relu_ex.png' height=50% width=50% />
Can you write Python code for it?

# after a ReLu is applied
# visualize the output of an activated conv layer
viz_layer(activated_layer)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 5 - Convolution visualisation.ipynb".
The first task is:
Visualize the output of each filter
First, we'll define a helper function, viz_layer that takes in a specific layer and number of filters (optional argument), and displays the output of that layer once an image has been passed through.
Can you write Python code for it?

# helper function for visualizing the output of a given layer
# default number of filters is 4
def viz_layer(layer, n_filters= 4):
    fig = plt.figure(figsize=(20, 20))

    for i in range(n_filters):
        ax = fig.add_subplot(1, n_filters, i+1, xticks=[], yticks=[])
        # grab layer outputs
        ax.imshow(np.squeeze(layer[0,i].data.numpy()), cmap='gray')
        ax.set_title('Output %s' % str(i+1))


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 6 - Maxpooling visualization.ipynb".
The first task is:
Define and visualize the filters
Can you write Python code for it?

import numpy as np

## TODO: Feel free to modify the numbers here, to try out another filter!
filter_vals = np.array([[-1, -1, 1, 1], [-1, -1, 1, 1], [-1, -1, 1, 1], [-1, -1, 1, 1]])

print('Filter shape: ', filter_vals.shape)

# Defining four different filters,
# all of which are linear combinations of the `filter_vals` defined above

# define four filters
filter_1 = filter_vals
filter_2 = -filter_1
filter_3 = filter_1.T
filter_4 = -filter_3
filters = np.array([filter_1, filter_2, filter_3, filter_4])

# For an example, print out the values of filter 1
print('Filter 1: \n', filter_1)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 6 - Maxpooling visualization.ipynb".
The first task is:
Let's look at the output of a convolutional layer after a ReLu activation function is applied.
ReLu activation
A ReLu function turns all negative pixel values in 0's (black). See the equation pictured below for input pixel values, x.
<img src='images/relu_ex.png' height=50% width=50% />
Can you write Python code for it?

# plot original image
plt.imshow(gray_img, cmap='gray')

# visualize all filters
fig = plt.figure(figsize=(12, 6))
fig.subplots_adjust(left=0, right=1.5, bottom=0.8, top=1, hspace=0.05, wspace=0.05)
for i in range(4):
    ax = fig.add_subplot(1, 4, i+1, xticks=[], yticks=[])
    ax.imshow(filters[i], cmap='gray')
    ax.set_title('Filter %s' % str(i+1))

# convert the image into an input Tensor
gray_img_tensor = torch.from_numpy(gray_img).unsqueeze(0).unsqueeze(1)

# get all the layers
conv_layer, activated_layer, pooled_layer = model(gray_img_tensor)

# visualize the output of the activated conv layer
viz_layer(activated_layer)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 6 - Maxpooling visualization.ipynb".
The first task is:
Visualize the output of each filter
First, we'll define a helper function, viz_layer that takes in a specific layer and number of filters (optional argument), and displays the output of that layer once an image has been passed through.
Can you write Python code for it?

# helper function for visualizing the output of a given layer
# default number of filters is 4
def viz_layer(layer, n_filters= 4):
    fig = plt.figure(figsize=(20, 20))

    for i in range(n_filters):
        ax = fig.add_subplot(1, n_filters, i+1)
        # grab layer outputs
        ax.imshow(np.squeeze(layer[0,i].data.numpy()), cmap='gray')
        ax.set_title('Output %s' % str(i+1))


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 6 - Maxpooling visualization.ipynb".
The first task is:
Visualize the output of the pooling layer
Then, take a look at the output of a pooling layer. The pooling layer takes as input the feature maps pictured above and reduces the dimensionality of those maps, by some pooling factor, by constructing a new, smaller image of only the maximum (brightest) values in a given kernel area.
Take a look at the values on the x, y axes to see how the image has changed size.
Can you write Python code for it?

# visualize the output of the pooling layer
viz_layer(pooled_layer)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 7 - CIFAR - 10 with augmentation.ipynb".
The first task is:
Define the Network Architecture
This time, you'll define a CNN architecture. Instead of an MLP, which used linear, fully-connected layers, you'll use the following:
* Convolutional layers, which can be thought of as stack of filtered images.
* Maxpooling layers, which reduce the x-y size of an input, keeping only the most active pixels from the previous layer.
* The usual Linear + Dropout layers to avoid overfitting and produce a 10-dim output.
A network with 2 convolutional layers is shown in the image below and in the code, and you've been given starter code with one convolutional and one maxpooling layer.
<img src='images/2_layer_conv.png' height=50% width=50% />
TODO: Define a model with multiple convolutional layers, and define the feedforward metwork behavior.
The more convolutional layers you include, the more complex patterns in color and shape a model can detect. It's suggested that your final model include 2 or 3 convolutional layers as well as linear layers + dropout in between to avoid overfitting.
It's good practice to look at existing research and implementations of related models as a starting point for defining your own models. You may find it useful to look at this PyTorch classification example or this, more complex Keras example to help decide on a final structure.
Output volume for a convolutional layer
To compute the output size of a given convolutional layer we can perform the following calculation (taken from Stanford's cs231n course):

We can compute the spatial size of the output volume as a function of the input volume size (W), the kernel/filter size (F), the stride with which they are applied (S), and the amount of zero padding used (P) on the border. The correct formula for calculating how many neurons define the output_W is given by (W−F+2P)/S+1.

For example for a 7x7 input and a 3x3 filter with stride 1 and pad 0 we would get a 5x5 output. With stride 2 we would get a 3x3 output.
Can you write Python code for it?

import torch.nn as nn
import torch.nn.functional as F

# define the CNN architecture
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # Convolutional layer (sees 32x32x3 image tensor)
        self.conv1 = nn.Conv2d(3, 16, 3, padding=1)
        # Convolutional layer (sees 16x16x16 tensor)
        self.conv2 = nn.Conv2d(16, 32, 3, padding=1)
        # Convolutional layer (sees 8x8x32 tensor)
        self.conv3 = nn.Conv2d(32, 64, 3, padding=1)
        # Max pooling layer
        self.pool = nn.MaxPool2d(2, 2)
        # Linear layer (64 * 4 * 4 -> 500)
        self.fc1 = nn.Linear(64 * 4 * 4, 500)
        # Linear layer (500 -> 10)
        self.fc2 = nn.Linear(500, 10)
        # Dropout layer (p=0.25)
        self.dropout = nn.Dropout(0.25)

    def forward(self, x):
        # Add sequence of convolutional and max pooling layers
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.pool(F.relu(self.conv3(x)))
        # Flatten image input
        x = x.view(-1, 64 * 4 * 4)
        # Add dropout layer
        x = self.dropout(x)
        # Add 1st hidden layer, with relu activation function
        x = F.relu(self.fc1(x))
        # Add dropout layer
        x = self.dropout(x)
        # Add 2nd hidden layer, with relu activation function
        x = self.fc2(x)
        return x

# Create a complete CNN
model = Net()
print(model)

# Move tensors to GPU if CUDA is available
if train_on_gpu:
    model.cuda()


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 7 - CIFAR - 10 with augmentation.ipynb".
The first task is:
Load and Augment the Data
Downloading may take a minute. We load in the training and test data, split the training data into a training and validation set, then create DataLoaders for each of these sets of data.
Augmentation
In this cell, we perform some simple data augmentation by randomly flipping and rotating the given image data. We do this by defining a torchvision transform, and you can learn about all the transforms that are used to pre-process and augment data, here.
TODO: Look at the transformation documentation; add more augmentation transforms, and see how your model performs.
This type of data augmentation should add some positional variety to these images, so that when we train a model on this data, it will be robust in the face of geometric changes (i.e. it will recognize a ship, no matter which direction it is facing). It's recommended that you choose one or two transforms.
Can you write Python code for it?

from torchvision import datasets
import torchvision.transforms as transforms
from torch.utils.data.sampler import SubsetRandomSampler

# Number of subprocesses to use for data loading
num_workers = 0
# How many samples per batch to load
batch_size = 20
# Percentage of training set to use as validation
valid_size = 0.2

# Convert data to a normalized torch.FloatTensor
transform = transforms.Compose([transforms.RandomHorizontalFlip(), # randomly flip and rotate
                                transforms.RandomRotation(degrees=10),
                                transforms.ToTensor(),
                                transforms.Normalize(mean=(0.5, 0.5, 0.5),
                                                     std=(0.5, 0.5, 0.5))])

# choose the training and test datasets
train_data = datasets.CIFAR10(root='data',
                              train=True,
                              download=True,
                              transform=transform)
test_data = datasets.CIFAR10(root='data',
                             train=False,
                             download=True,
                             transform=transform)

# obtain training indices that will be used for validation
num_train = len(train_data)
indices = list(range(num_train))
np.random.shuffle(indices)
split = int(np.floor(valid_size * num_train))
train_idx, valid_idx = indices[split:], indices[:split]

# define samplers for obtaining training and validation batches
train_sampler = SubsetRandomSampler(indices=train_idx)
valid_sampler = SubsetRandomSampler(indices=valid_idx)

# prepare data loaders (combine dataset and sampler)
train_loader = torch.utils.data.DataLoader(dataset=train_data,
                                           batch_size=batch_size,
                                           sampler=train_sampler,
                                           num_workers=num_workers)
valid_loader = torch.utils.data.DataLoader(dataset=train_data,
                                           batch_size=batch_size,
                                ...


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 7 - CIFAR - 10 with augmentation.ipynb".
The first task is:
Specify Loss Function and Optimizer
Decide on a loss and optimization function that is best suited for this classification task. The linked code examples from above, may be a good starting point; this PyTorch classification example or this, more complex Keras example. Pay close attention to the value for learning rate as this value determines how your model converges to a small error.
TODO: Define the loss and optimizer and see how these choices change the loss over time.
Can you write Python code for it?

import torch.optim as optim

# Specify loss function (categorical cross-entropy)
criterion = nn.CrossEntropyLoss()

# Specify optimizer
optimizer = optim.SGD(model.parameters(), lr=0.001)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 7 - CIFAR - 10 with augmentation.ipynb".
The first task is:
Test the Trained Network
Test your trained model on previously unseen data! A "good" result will be a CNN that gets around 70% (or more, try your best!) accuracy on these test images.
Can you write Python code for it?

# Track test loss
test_loss = 0.0
class_correct = list(0. for i in range(10))
class_total = list(0. for i in range(10))

model.eval()
# Iterate over test data
for batch_idx, (data, target) in enumerate(test_loader):
    # Move tensors to GPU if CUDA is available
    if train_on_gpu:
        data, target = data.cuda(), target.cuda()
    # Forward pass: compute predicted outputs by passing inputs to the model
    output = model(data)
    # Calculate the batch loss
    loss = criterion(output, target)
    # Update test loss
    test_loss += loss.item()*data.size(0)
    # Convert output probabilities to predicted class
    _, pred = torch.max(output, 1)
    # Compare predictions to true label
    correct_tensor = pred.eq(target.data.view_as(pred))
    correct = np.squeeze(correct_tensor.numpy()) if not train_on_gpu else np.squeeze(correct_tensor.cpu().numpy())
    # Calculate test accuracy for each object class
    for i in range(batch_size):
        label = target.data[i]
        class_correct[label] += correct[i].item()
        class_total[label] += 1

# Average test loss
test_loss = test_loss/len(test_loader.dataset)
print('Test Loss: {:.6f}\n'.format(test_loss))

for i in range(10):
    if class_total[i] > 0:
        print('Test Accuracy of %5s: %2d%% (%2d/%2d)' % (classes[i], 100 * class_correct[i] / class_total[i],
                                                         np.sum(class_correct[i]), np.sum(class_total[i])))
    else:
        print('Test Accuracy of %5s: N/A (no training examples)' % (classes[i]))

print('\nTest Accuracy (Overall): %2d%% (%2d/%2d)' % (100. * np.sum(class_correct) / np.sum(class_total),
                                                      np.sum(class_correct), np.sum(class_total)))


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 7 - CIFAR - 10 with augmentation.ipynb".
The first task is:
Train the Network
Remember to look at how the training and validation loss decreases over time; if the validation loss ever increases it indicates possible overfitting.
Can you write Python code for it?

# Number of epochs to train the model
n_epochs = 30

valid_loss_min = np.Inf # track change in validation loss

for epoch in range(1, n_epochs + 1):

    # Keep track of training and validation loss
    train_loss = 0.0
    valid_loss = 0.0

    # train the model
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        # Move tensors to GPU if CUDA is available
        if train_on_gpu:
            data, target = data.cuda(), target.cuda()
        # Clear the gradients of all optimized variables
        optimizer.zero_grad()
        # Forward pass: compute predicted outputs by passing inputs to the model
        output = model(data)
        # Calculate the batch loss
        loss = criterion(output, target)
        # Backward pass: compute gradient of the loss with respect to model parameters
        loss.backward()
        # Perform a single optimization step (parameter update)
        optimizer.step()
        # Update training loss
        train_loss += loss.item()*data.size(0)

    # Validate the model
    model.eval()
    for batch_idx, (data, target) in enumerate(valid_loader):
        # Move tensors to GPU if CUDA is available
        if train_on_gpu:
            data, target = data.cuda(), target.cuda()
        # Forward pass: compute predicted outputs by passing inputs to the model
        output = model(data)
        # Calculate the batch loss
        loss = criterion(output, target)
        # Update average validation loss
        valid_loss += loss.item()*data.size(0)

    # Calculate average losses
    train_loss = train_loss/len(train_loader.dataset)
    valid_loss = valid_loss/len(valid_loader.dataset)

    # Print training / validation statistics
    print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f}'.format(
        epoch, train_loss, valid_loss))

    # Save model if validation loss has decreased
    if valid_loss <= valid_loss_min:
        print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model...


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 7 - CIFAR - 10 with augmentation.ipynb".
The first task is:
View an Image in More Detail
Here, we look at the normalized red, green, and blue (RGB) color channels as three separate, grayscale intensity images.
Can you write Python code for it?

rgb_img = np.squeeze(images[3])
channels = ['red channel', 'green channel', 'blue channel']

fig = plt.figure(figsize = (36, 36))
for idx in np.arange(rgb_img.shape[0]):
    ax = fig.add_subplot(1, 3, idx + 1)
    img = rgb_img[idx]
    ax.imshow(img, cmap='gray')
    ax.set_title(channels[idx])
    width, height = img.shape
    thresh = img.max()/2.5
    for x in range(width):
        for y in range(height):
            val = round(img[x][y],2) if img[x][y] !=0 else 0
            ax.annotate(str(val), xy=(y,x),
                    horizontalalignment='center',
                    verticalalignment='center', size=8,
                    color='white' if img[x][y]<thresh else 'black')


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 7 - CIFAR - 10 with augmentation.ipynb".
The first task is:
Visualize Sample Test Results
Can you write Python code for it?

# Obtain one batch of test images
dataiter = iter(test_loader)
images, labels = dataiter.next()
images.numpy()

# Move model inputs to cuda, if GPU available
if train_on_gpu:
    images = images.cuda()

# Get sample outputs
output = model(images)
# Convert output probabilities to predicted class
_, preds_tensor = torch.max(output, 1)
preds = np.squeeze(preds_tensor.numpy()) if not train_on_gpu else np.squeeze(preds_tensor.cpu().numpy())

# Plot the images in the batch, along with predicted and true labels
fig = plt.figure(figsize=(25, 4))
for idx in np.arange(20):
    ax = fig.add_subplot(2, 20/2, idx+1, xticks=[], yticks=[])
    imshow(images[idx])
    ax.set_title("{} ({})".format(classes[preds[idx]], classes[labels[idx]]),
                 color=("green" if preds[idx]==labels[idx].item() else "red"))


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/03-Convolutional Neural Networks/Part 7 - CIFAR - 10 with augmentation.ipynb".
The first task is:
Visualize a Batch of Training Data
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

# helper function to un-normalize and display an image
def imshow(img):
    img = img / 2 + 0.5  # unnormalize
    plt.imshow(np.transpose(img, (1, 2, 0)))  # convert from Tensor image

# obtain one batch of training images
dataiter = iter(train_loader)
images, labels = dataiter.next()
images = images.numpy() # convert images to numpy for display

# plot the images in the batch, along with the corresponding labels
fig = plt.figure(figsize=(25, 4))
# display 20 images
for idx in np.arange(20):
    ax = fig.add_subplot(2, 20/2, idx+1, xticks=[], yticks=[])
    imshow(images[idx])
    ax.set_title(classes[labels[idx]])


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/05-Recurrent Neural Networks/01 - Simple_RNN.ipynb".
The first task is:
Check the input and output dimensions
As a check that your model is working as expected, test out how it responds to input data.
Can you write Python code for it?

# Test that dimensions are as expected
test_rnn = RNN(input_size=1,
               output_size=1,
               hidden_dim=10,
               n_layers=2)

# Generate evenly spaced, test data pts
time_steps = np.linspace(0, np.pi, seq_length)
data = np.sin(time_steps)
data.resize((seq_length, 1))

# Give it a batch_size of 1 as first dimension
print('Input size before: ', torch.Tensor(data).size())
test_input = torch.Tensor(data).unsqueeze(0)
print('Input size after: ', test_input.size())

# Test out rnn sizes
test_out, test_h = test_rnn(test_input, None)
print('Output size: ', test_out.size())
print('Hidden state size: ', test_h.size())


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/05-Recurrent Neural Networks/01 - Simple_RNN.ipynb".
The first task is:
Define the RNN
Next, we define an RNN in PyTorch. We'll use nn.RNN to create an RNN layer, then we'll add a last, fully-connected layer to get the output size that we want. An RNN takes in a number of parameters:
* input_size - the size of the input
* hidden_dim - the number of features in the RNN output and in the hidden state
* n_layers - the number of layers that make up the RNN, typically 1-3; greater than 1 means that you'll create a stacked RNN
* batch_first - whether or not the input/output of the RNN will have the batch_size as the first dimension (batch_size, seq_length, hidden_dim)
Take a look at the RNN documentation to read more about recurrent layers.
Can you write Python code for it?

class RNN(nn.Module):
    def __init__(self, input_size, output_size, hidden_dim, n_layers):
        super(RNN, self).__init__()

        self.hidden_dim=hidden_dim

        # Define an RNN with specified parameters
        # Batch_first means that the first dim of the input and output will be the batch_size
        self.rnn = nn.RNN(input_size=input_size,
                          hidden_size=hidden_dim,
                          num_layers=n_layers,
                          batch_first=True)

        # last, fully-connected layer
        self.fc = nn.Linear(hidden_dim, output_size)

    def forward(self, x, hidden):
        # x (batch_size, seq_length, input_size)
        # hidden (n_layers, batch_size, hidden_dim)
        # r_out (batch_size, time_step, hidden_size)
        batch_size = x.size(0)

        # Get RNN outputs
        r_out, hidden = self.rnn(x, hidden)
        # Shape output to be (batch_size*seq_length, hidden_dim)
        r_out = r_out.view(-1, self.hidden_dim)

        # Get final output
        output = self.fc(r_out)

        return output, hidden


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/05-Recurrent Neural Networks/01 - Simple_RNN.ipynb".
The first task is:
Loss and Optimization
This is a regression problem: can we train an RNN to accurately predict the next data point, given a current data point?

The data points are coordinate values, so to compare a predicted and ground_truth point, we'll use a regression loss: the mean squared error.
It's typical to use an Adam optimizer for recurrent models.
Can you write Python code for it?

# MSE loss and Adam optimizer with a learning rate of 0.01
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(params=rnn.parameters(),
                             lr=0.01)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/05-Recurrent Neural Networks/02 - Character_Level_RNN.ipynb".
The first task is:
And we can see those same characters from above, encoded as integers.
Can you write Python code for it?


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/05-Recurrent Neural Networks/02 - Character_Level_RNN.ipynb".
The first task is:
Getting the best model
To set your hyperparameters to get the best performance, you'll want to watch the training and validation losses. If your training loss is much lower than the validation loss, you're overfitting. Increase regularization (more dropout) or use a smaller network. If the training and validation losses are close, you're underfitting so you can increase the size of the network.
Hyperparameters
Here are the hyperparameters for the network.
In defining the model:
* n_hidden - The number of units in the hidden layers.
* n_layers - Number of hidden LSTM layers to use.
We assume that dropout probability and learning rate will be kept at the default, in this example.
And in training:
* batch_size - Number of sequences running through the network in one pass.
* seq_length - Number of characters in the sequence the network is trained on. Larger is better typically, the network will learn more long range dependencies. But it takes longer to train. 100 is typically a good number here.
* lr - Learning rate for training
Here's some good advice from Andrej Karpathy on training the network. I'm going to copy it in here for your benefit, but also link to where it originally came from.

Tips and Tricks
Monitoring Validation Loss vs. Training Loss
If you're somewhat new to Machine Learning or Neural Networks it can take a bit of expertise to get good models. The most important quantity to keep track of is the difference between your training loss (printed during training) and the validation loss (printed once in a while when the RNN is run on the validation data (by default every 1000 iterations)). In particular:

If your training loss is much lower than validation loss then this means the network might be overfitting. Solutions to this are to decrease your network size, or to increase dropout. For example you could try dropout of 0.5 and so on.
If your training/validation loss are about equal then your model is underfitting. Increase the size of your model (either number of layers or the raw number of...
Can you write Python code for it?

# Change the name, for saving multiple files
model_name = './models/rnn_x_epoch.net'

checkpoint = {'n_hidden': net.n_hidden,
              'n_layers': net.n_layers,
              'state_dict': net.state_dict(),
              'tokens': net.chars}

with open(model_name, 'wb') as f:
    torch.save(checkpoint, f)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/05-Recurrent Neural Networks/02 - Character_Level_RNN.ipynb".
The first task is:
If you implemented get_batches correctly, the above output should look something like
```
x
 [[25  8 60 11 45 27 28 73  1  2]
 [17  7 20 73 45  8 60 45 73 60]
 [27 20 80 73  7 28 73 60 73 65]
 [17 73 45  8 27 73 66  8 46 27]
 [73 17 60 12 73  8 27 28 73 45]
 [66 64 17 17 46  7 20 73 60 20]
 [73 76 20 20 60 73  8 60 80 73]
 [47 35 43  7 20 17 24 50 37 73]]
y
 [[ 8 60 11 45 27 28 73  1  2  2]
 [ 7 20 73 45  8 60 45 73 60 45]
 [20 80 73  7 28 73 60 73 65  7]
 [73 45  8 27 73 66  8 46 27 65]
 [17 60 12 73  8 27 28 73 45 27]
 [64 17 17 46  7 20 73 60 20 80]
 [76 20 20 60 73  8 60 80 73 17]
 [35 43  7 20 17 24 50 37 73 36]]
 ``
 although the exact numbers may be different. Check to make sure the data is shifted over one step fory`.

Defining the network with PyTorch
Below is where you'll define the network.
<img src="images/charRNN.png" width=500px>
Next, you'll use PyTorch to define the architecture of the network. We start by defining the layers and operations we want. Then, define a method for the forward pass. You've also been given a method for predicting characters.
Model Structure
In __init__ the suggested structure is as follows:
* Create and store the necessary dictionaries (this has been done for you)
* Define an LSTM layer that takes as params: an input size (the number of characters), a hidden layer size n_hidden, a number of layers n_layers, a dropout probability drop_prob, and a batch_first boolean (True, since we are batching)
* Define a dropout layer with dropout_prob
* Define a fully-connected layer with params: input size n_hidden and output size (the number of characters)
* Finally, initialize the weights (again, this has been given)
Note that some parameters have been named and given in the __init__ function, and we use them and store them by doing something like self.drop_prob = drop_prob.

LSTM Inputs/Outputs
You can create a basic LSTM layer as follows
python
self.lstm = nn.LSTM(input_size, n_hidden, n_layers,
                            dropout=drop_prob, batch_first=True)
where input_size...
Can you write Python code for it?

# check if GPU is available
train_on_gpu = torch.cuda.is_available()
if(train_on_gpu):
    print('Training on GPU!')
else:
    print('No GPU available, training on CPU; consider making n_epochs very small.')

class CharRNN(nn.Module):

    def __init__(self,
                 tokens,
                 n_hidden=256,
                 n_layers=2,
                 drop_prob=0.5,
                 lr=0.001):

        super().__init__()
        self.drop_prob = drop_prob
        self.n_layers = n_layers
        self.n_hidden = n_hidden
        self.lr = lr

        # creating character dictionaries
        self.chars = tokens
        self.int2char = dict(enumerate(self.chars))
        self.char2int = {ch: ii for ii, ch in self.int2char.items()}

        ## TODO: define the layers of the model
        self.lstm = nn.LSTM(input_size=len(self.chars),
                            hidden_size=n_hidden,
                            num_layers=n_layers,
                            dropout=drop_prob,
                            batch_first=True)
        ## Define dropout
        self.dropout = nn.Dropout(drop_prob)
        ## Define the final fully-connected layer
        self.fc_out = nn.Linear(in_features=n_hidden,
                                out_features=len(self.chars))

    def forward(self, x, hidden):
        ''' Forward pass through the network.
            These inputs are x, and the hidden/cell state `hidden`. '''

        ## TODO: Get the outputs and the new hidden state from the lstm
        lstm_out, hidden = self.lstm(x, hidden)

        after_dropout = self.dropout(lstm_out)

        # Reshaping the data
        reshaped = after_dropout.contiguous().view(-1, self.n_hidden)

        # Return the final output and the hidden state
        out = self.fc_out(reshaped)
        return out, hidden

    def init_hidden(self, batch_size):
        ''' Initializes hidden state '''
        # Create two new tensors with sizes n_layers x...


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/05-Recurrent Neural Networks/02 - Character_Level_RNN.ipynb".
The first task is:
Instantiating the model
Now we can actually train the network. First we'll create the network itself, with some given hyperparameters. Then, define the mini-batches sizes, and start training!
Can you write Python code for it?

## TODO: set you model hyperparameters
# Define and print the net
n_hidden = 256
n_layers = 2

net = CharRNN(chars, n_hidden, n_layers)
print(net)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/05-Recurrent Neural Networks/02 - Character_Level_RNN.ipynb".
The first task is:
Making training mini-batches
To train on this data, we also want to create mini-batches for training. Remember that we want our batches to be multiple sequences of some desired number of sequence steps. Considering a simple example, our batches would look like this:

In this example, we'll take the encoded characters (passed in as the arr parameter) and split them into multiple sequences, given by batch_size. Each of our sequences will be seq_length long.
Creating Batches
1. The first thing we need to do is discard some of the text so we only have completely full mini-batches.
Each batch contains $N \times M$ characters, where $N$ is the batch size (the number of sequences in a batch) and $M$ is the seq_length or number of time steps in a sequence. Then, to get the total number of batches, $K$, that we can make from the array arr, you divide the length of arr by the number of characters per batch. Once you know the number of batches, you can get the total number of characters to keep from arr, $N * M * K$.
2. After that, we need to split arr into $N$ batches.
You can do this using arr.reshape(size) where size is a tuple containing the dimensions sizes of the reshaped array. We know we want $N$ sequences in a batch, so let's make that the size of the first dimension. For the second dimension, you can use -1 as a placeholder in the size, it'll fill up the array with the appropriate data for you. After this, you should have an array that is $N \times (M * K)$.
3. Now that we have this array, we can iterate through it to get our mini-batches.
The idea is each batch is a $N \times M$ window on the $N \times (M * K)$ array. For each subsequent batch, the window moves over by seq_length. We also want to create both the input and target arrays. Remember that the targets are just the inputs shifted over by one character. The way I like to do this window is use range to take steps of size n_steps from $0$ to arr.shape[1], the total number of tokens in each...
Can you write Python code for it?

def get_batches(arr, batch_size, seq_length):
    '''Create a generator that returns batches of size
       batch_size x seq_length from arr.

       Arguments
       ---------
       arr: Array you want to make batches from
       batch_size: Batch size, the number of sequences per batch
       seq_length: Number of encoded chars in a sequence
    '''

    ## TODO: Get the number of batches we can make
    n_batches = len(arr) // (batch_size * seq_length)

    ## TODO: Keep only enough characters to make full batches
    arr = arr[:(n_batches * batch_size * seq_length)]

    ## TODO: Reshape into batch_size rows
    arr = arr.reshape((batch_size, -1))

    ## TODO: Iterate over the batches using a window of size seq_length
    for n in range(0, arr.shape[1], seq_length):
        # The features
        x = arr[:, n:(n + seq_length)]
        # The targets, shifted by one
        y = np.zeros_like(x)
        try:
            y[:, :-1], y[:, -1] = x[:, 1:], arr[:, n + seq_length]
        except IndexError:
            y[:, :-1], y[:, -1] = x[:, 1:], arr[:, 0]
        yield x, y


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/05-Recurrent Neural Networks/02 - Character_Level_RNN.ipynb".
The first task is:
Priming and generating text
Typically you'll want to prime the network so you can build up a hidden state. Otherwise the network will start out generating characters at random. In general the first bunch of characters will be a little rough since it hasn't built up a long history of characters to predict from.
Can you write Python code for it?

def sample(net, size, prime='The', top_k=None):

    if(train_on_gpu):
        net.cuda()
    else:
        net.cpu()
    # Eval mode
    net.eval()

    # First off, run through the prime characters
    chars = [ch for ch in prime]
    h = net.init_hidden(1)
    for ch in prime:
        char, h = predict(net, ch, h, top_k=top_k)

    chars.append(char)

    # Now pass in the previous character and get a new one
    for ii in range(size):
        char, h = predict(net, chars[-1], h, top_k=top_k)
        chars.append(char)

    return ''.join(chars)

print(sample(net, 1000, prime='Anna', top_k=5))


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/05-Recurrent Neural Networks/02 - Character_Level_RNN.ipynb".
The first task is:
Test Your Implementation
Now I'll make some data sets and we can check out what's going on as we batch data. Here, as an example, I'm going to use a batch size of 8 and 50 sequence steps.
Can you write Python code for it?

batches = get_batches(encoded, 8, 50)
x, y = next(batches)

# printing out the first 10 items in a sequence
print('x\n', x[:10, :10])
print('\ny\n', y[:10, :10])


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/05-Recurrent Neural Networks/02 - Character_Level_RNN.ipynb".
The first task is:
Time to train
The train function gives us the ability to set the number of epochs, the learning rate, and other parameters.
Below we're using an Adam optimizer and cross entropy loss since we are looking at character class scores as output. We calculate the loss and perform backpropagation, as usual!
A couple of details about training:

Within the batch loop, we detach the hidden state from its history; this time setting it equal to a new tuple variable because an LSTM has a hidden state that is a tuple of the hidden and cell states.
We use clip_grad_norm_ to help prevent exploding gradients.
Can you write Python code for it?

def train(net, data, epochs=10, batch_size=10, seq_length=50, lr=0.001, clip=5, val_frac=0.1, print_every=10):
    ''' Training a network

        Arguments
        ---------

        net: CharRNN network
        data: text data to train the network
        epochs: Number of epochs to train
        batch_size: Number of mini-sequences per mini-batch, aka batch size
        seq_length: Number of character steps per mini-batch
        lr: learning rate
        clip: gradient clipping
        val_frac: Fraction of data to hold out for validation
        print_every: Number of steps for printing training and validation loss

    '''
    net.train()

    opt = torch.optim.Adam(params=net.parameters(),
                           lr=lr)
    criterion = nn.CrossEntropyLoss()

    # Create training and validation data
    val_idx = int(len(data)*(1-val_frac))
    data, val_data = data[:val_idx], data[val_idx:]

    if(train_on_gpu):
        net.cuda()

    counter = 0
    n_chars = len(net.chars)
    for e in range(epochs):
        # Initialize hidden state
        h = net.init_hidden(batch_size)

        for x, y in get_batches(data, batch_size, seq_length):
            counter += 1

            # One-hot encode our data and make them Torch tensors
            x = one_hot_encode(x, n_chars)
            inputs, targets = torch.from_numpy(x), torch.from_numpy(y).type(torch.LongTensor)

            if(train_on_gpu):
                inputs, targets = inputs.cuda(), targets.cuda()

            # Creating new variables for the hidden state, otherwise
            # we'd backprop through the entire training history
            h = tuple([each.data for each in h])

            # Zero accumulated gradients
            net.zero_grad()

            # Get the output from the model
            output, h = net(inputs, h)

            # Calculate the loss and perform backprop
            loss = criterion(output, targets.view(batch_size *...


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/06-Sentiment Prediction with RNNs/Sentiment_analysis_with_RNNs.ipynb".
The first task is:
Check your work
With train, validation, and test fractions equal to 0.8, 0.1, 0.1, respectively, the final, feature data shapes should look like:
Feature Shapes:
Train set:       (20000, 200)
Validation set:     (2500, 200)
Test set:         (2500, 200)

DataLoaders and Batching
After creating training, test, and validation data, we can create DataLoaders for this data by following two steps:
1. Create a known format for accessing our data, using TensorDataset which takes in an input set of data and a target set of data with the same first dimension, and creates a dataset.
2. Create DataLoaders and batch our training, validation, and test Tensor datasets.
train_data = TensorDataset(torch.from_numpy(train_x), torch.from_numpy(train_y))
train_loader = DataLoader(train_data, batch_size=batch_size)
This is an alternative to creating a generator function for batching our data into full batches.
Can you write Python code for it?

import torch
from torch.utils.data import TensorDataset, DataLoader

# Create Tensor datasets
train_data = TensorDataset(torch.from_numpy(train_x),
                           torch.from_numpy(train_y))
valid_data = TensorDataset(torch.from_numpy(val_x),
                           torch.from_numpy(val_y))
test_data = TensorDataset(torch.from_numpy(test_x),
                          torch.from_numpy(test_y))

# Dataloaders
batch_size = 50

# Make sure the SHUFFLE your training data
train_loader = DataLoader(dataset=train_data,
                          shuffle=True,
                          batch_size=batch_size)
valid_loader = DataLoader(dataset=valid_data,
                          shuffle=True,
                          batch_size=batch_size)
test_loader = DataLoader(dataset=test_data,
                         shuffle=True,
                         batch_size=batch_size)

# Obtain one batch of training data
dataiter = iter(train_loader)
sample_x, sample_y = dataiter.next()

# batch_size, seq_length
print('Sample input size: ', sample_x.size())
print('Sample input: \n', sample_x)
print()
# batch_size
print('Sample label size: ', sample_y.size())
print('Sample label: \n', sample_y)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/06-Sentiment Prediction with RNNs/Sentiment_analysis_with_RNNs.ipynb".
The first task is:
Data pre-processing
The first step when building a neural network model is getting your data into the proper form to feed into the network. Since we're using embedding layers, we'll need to encode each word with an integer. We'll also want to clean it up a bit.
You can see an example of the reviews data above. Here are the processing steps, we'll want to take:

We'll want to get rid of periods and extraneous punctuation.
Also, you might notice that the reviews are delimited with newline characters \n. To deal with those, I'm going to split the text into each review using \n as the delimiter.
Then I can combined all the reviews back together into one big string.

First, let's remove all punctuation. Then get all the text without the newlines and split it into individual words.
Can you write Python code for it?

from string import punctuation

# get rid of punctuation
reviews = reviews.lower() # lowercase, standardize
all_text = ''.join([c for c in reviews if c not in punctuation])

# split by new lines and spaces
reviews_split = all_text.split('\n')
all_text = ' '.join(reviews_split)

# create a list of words
words = all_text.split()

words[:30]


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/06-Sentiment Prediction with RNNs/Sentiment_analysis_with_RNNs.ipynb".
The first task is:
Encoding the words
The embedding lookup requires that we pass in integers to our network. The easiest way to do this is to create dictionaries that map the words in the vocabulary to integers. Then we can convert each of our reviews into integers so they can be passed into the network.

Exercise: Now you're going to encode the words with integers. Build a dictionary that maps words to integers. Later we're going to pad our input vectors with zeros, so make sure the integers start at 1, not 0.
Also, convert the reviews to integers and store the reviews in a new list called reviews_ints.
Can you write Python code for it?

# feel free to use this import
from collections import Counter

## Build a dictionary that maps words to integers
counts = Counter(words)
vocab = sorted(counts, key=counts.get, reverse=True)
vocab_to_int = {word: ii for ii, word in enumerate(vocab, 1)}

## use the dict to tokenize each review in reviews_split
## store the tokenized reviews in reviews_ints
reviews_ints = []
for review in reviews_split:
    reviews_ints.append([vocab_to_int[word] for word in review.split()])


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/06-Sentiment Prediction with RNNs/Sentiment_analysis_with_RNNs.ipynb".
The first task is:
Instantiate the network
Here, we'll instantiate the network. First up, defining the hyperparameters.

vocab_size: Size of our vocabulary or the range of values for our input, word tokens.
output_size: Size of our desired output; the number of class scores we want to output (pos/neg).
embedding_dim: Number of columns in the embedding lookup table; size of our embeddings.
hidden_dim: Number of units in the hidden layers of our LSTM cells. Usually larger is better performance wise. Common values are 128, 256, 512, etc.
n_layers: Number of LSTM layers in the network. Typically between 1-3

Exercise: Define the model  hyperparameters.
Can you write Python code for it?

# Instantiate the model w/ hyperparams
vocab_size = len(vocab_to_int) + 1 # +1 for the 0 padding + our word tokens
output_size = 1
embedding_dim = 200
hidden_dim = 32
n_layers = 2

net = SentimentRNN(vocab_size, output_size, embedding_dim, hidden_dim, n_layers)

print(net)


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/06-Sentiment Prediction with RNNs/Sentiment_analysis_with_RNNs.ipynb".
The first task is:
Okay, a couple issues here. We seem to have one review with zero length. And, the maximum review length is way too many steps for our RNN. We'll have to remove any super short reviews and truncate super long reviews. This removes outliers and should allow our model to train more efficiently.

Exercise: First, remove any reviews with zero length from the reviews_ints list and their corresponding label in encoded_labels.
Can you write Python code for it?

print('Number of reviews before removing outliers: ', len(reviews_ints))

## Remove any reviews/labels with zero length from the reviews_ints list.

# Get indices of any reviews with length 0
non_zero_idx = [ii for ii, review in enumerate(reviews_ints) if len(review) != 0]

# Remove 0-length reviews and their labels
reviews_ints = [reviews_ints[ii] for ii in non_zero_idx]
encoded_labels = np.array([encoded_labels[ii] for ii in non_zero_idx])

print('Number of reviews after removing outliers: ', len(reviews_ints))