I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/Autoencoders/convolutional-autoencoder/Upsampling_Solution.ipynb".
The first task is:
Visualize the Data
Can you write Python code for it?


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/Autoencoders/denoising-autoencoder/Denoising_Autoencoder_Exercise.ipynb".
The first task is:
Checking out the results
Here I'm adding noise to the test images and passing them through the autoencoder. It does a suprising great job of removing the noise, even though it's sometimes difficult to tell what the original number is.
Can you write Python code for it?

# obtain one batch of test images
dataiter = iter(test_loader)
images, labels = dataiter.next()

# add noise to the test images
noisy_imgs = images + noise_factor * torch.randn(*images.shape)
noisy_imgs = np.clip(noisy_imgs, 0., 1.)

# get sample outputs
output = model(noisy_imgs)
# prep images for display
noisy_imgs = noisy_imgs.numpy()

# output is resized into a batch of iages
output = output.view(batch_size, 1, 28, 28)
# use detach when it's an output that requires_grad
output = output.detach().numpy()

# plot the first ten input images and then reconstructed images
fig, axes = plt.subplots(nrows=2, ncols=10, sharex=True, sharey=True, figsize=(25,4))

# input images on top row, reconstructions on bottom
for noisy_imgs, row in zip([noisy_imgs, output], axes):
    for img, ax in zip(noisy_imgs, row):
        ax.imshow(np.squeeze(img), cmap='gray')
        ax.get_xaxis().set_visible(False)
        ax.get_yaxis().set_visible(False)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/Autoencoders/denoising-autoencoder/Denoising_Autoencoder_Exercise.ipynb".
The first task is:
Denoising
As I've mentioned before, autoencoders like the ones you've built so far aren't too useful in practive. However, they can be used to denoise images quite successfully just by training the network on noisy images. We can create the noisy images ourselves by adding Gaussian noise to the training images, then clipping the values to be between 0 and 1.

We'll use noisy images as input and the original, clean images as targets.

Below is an example of some of the noisy images I generated and the associated, denoised images.
<img src='notebook_ims/denoising.png' />
Since this is a harder problem for the network, we'll want to use deeper convolutional layers here; layers with more feature maps. You might also consider adding additional layers. I suggest starting with a depth of 32 for the convolutional layers in the encoder, and the same depths going backward through the decoder.
TODO: Build the network for the denoising autoencoder. Add deeper and/or additional layers compared to the model above.
Can you write Python code for it?

import torch.nn as nn
import torch.nn.functional as F

# define the NN architecture
class ConvDenoiser(nn.Module):
    def __init__(self):
        super(ConvDenoiser, self).__init__()
        ## encoder layers ##

        ## decoder layers ##
        ## a kernel of 2 and a stride of 2 will increase the spatial dims by 2

    def forward(self, x):
        ## encode ##

        ## decode ##

        return x

# initialize the NN
model = ConvDenoiser()
print(model)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/Autoencoders/denoising-autoencoder/Denoising_Autoencoder_Exercise.ipynb".
The first task is:
Visualize the Data
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

# obtain one batch of training images
dataiter = iter(train_loader)
images, labels = dataiter.next()
images = images.numpy()

# get one image from the batch
img = np.squeeze(images[0])

fig = plt.figure(figsize = (5,5))
ax = fig.add_subplot(111)
ax.imshow(img, cmap='gray')


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/Autoencoders/denoising-autoencoder/Denoising_Autoencoder_Solution.ipynb".
The first task is:
Checking out the results
Here I'm adding noise to the test images and passing them through the autoencoder. It does a suprising great job of removing the noise, even though it's sometimes difficult to tell what the original number is.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/Autoencoders/denoising-autoencoder/Denoising_Autoencoder_Solution.ipynb".
The first task is:
Visualize the Data
Can you write Python code for it?


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/Autoencoders/linear-autoencoder/Simple_Autoencoder_Exercise.ipynb".
The first task is:
Checking out the results
Below I've plotted some of the test images along with their reconstructions. For the most part these look pretty good except for some blurriness in some parts.
Can you write Python code for it?

# obtain one batch of test images
dataiter = iter(test_loader)
images, labels = dataiter.next()

images_flatten = images.view(images.size(0), -1)
# get sample outputs
output = model(images_flatten)
# prep images for display
images = images.numpy()

# output is resized into a batch of images
output = output.view(batch_size, 1, 28, 28)
# use detach when it's an output that requires_grad
output = output.detach().numpy()

# plot the first ten input images and then reconstructed images
fig, axes = plt.subplots(nrows=2, ncols=10, sharex=True, sharey=True, figsize=(25,4))

# input images on top row, reconstructions on bottom
for images, row in zip([images, output], axes):
    for img, ax in zip(images, row):
        ax.imshow(np.squeeze(img), cmap='gray')
        ax.get_xaxis().set_visible(False)
        ax.get_yaxis().set_visible(False)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/Autoencoders/linear-autoencoder/Simple_Autoencoder_Exercise.ipynb".
The first task is:
Visualize the Data
Can you write Python code for it?


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/Autoencoders/linear-autoencoder/Simple_Autoencoder_Solution.ipynb".
The first task is:
Checking out the results
Below I've plotted some of the test images along with their reconstructions. For the most part these look pretty good except for some blurriness in some parts.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/Autoencoders/linear-autoencoder/Simple_Autoencoder_Solution.ipynb".
The first task is:
Visualize the Data
Can you write Python code for it?


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/cycle-gan/CycleGAN_Exercise.ipynb".
The first task is:
Check that you've implemented this correctly
The function create_model should return the two generator and two discriminator networks. After you've defined these discriminator and generator components, it's good practice to check your work. The easiest way to do this is to print out your model architecture and read through it to make sure the parameters are what you expected. The next cell will print out their architectures.
Can you write Python code for it?

# helper function for printing the model architecture
def print_models(G_XtoY, G_YtoX, D_X, D_Y):
    """Prints model information for the generators and discriminators.
    """
    print("                     G_XtoY                    ")
    print("-----------------------------------------------")
    print(G_XtoY)
    print()

    print("                     G_YtoX                    ")
    print("-----------------------------------------------")
    print(G_YtoX)
    print()

    print("                      D_X                      ")
    print("-----------------------------------------------")
    print(D_X)
    print()

    print("                      D_Y                      ")
    print("-----------------------------------------------")
    print(D_Y)
    print()

# print all of the models
print_models(G_XtoY, G_YtoX, D_X, D_Y)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/cycle-gan/CycleGAN_Exercise.ipynb".
The first task is:
Create the complete network
Using the classes you defined earlier, you can define the discriminators and generators necessary to create a complete CycleGAN. The given parameters should work for training.
First, create two discriminators, one for checking if $X$ sample images are real, and one for checking if $Y$ sample images are real. Then the generators. Instantiate two of them, one for transforming a painting into a realistic photo and one for transforming a photo into  into a painting.
Can you write Python code for it?

def create_model(g_conv_dim=64, d_conv_dim=64, n_res_blocks=6):
    """Builds the generators and discriminators."""

    # Instantiate generators
    G_XtoY =
    G_YtoX =
    # Instantiate discriminators
    D_X =
    D_Y =

    # move models to GPU, if available
    if torch.cuda.is_available():
        device = torch.device("cuda:0")
        G_XtoY.to(device)
        G_YtoX.to(device)
        D_X.to(device)
        D_Y.to(device)
        print('Models moved to GPU.')
    else:
        print('Only CPU available.')

    return G_XtoY, G_YtoX, D_X, D_Y

# call the function to get models
G_XtoY, G_YtoX, D_X, D_Y = create_model()


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/cycle-gan/CycleGAN_Exercise.ipynb".
The first task is:
DataLoaders
The get_data_loader function returns training and test DataLoaders that can load data efficiently and in specified batches. The function has the following parameters:
* image_type: summer or winter,  the names of the directories where the X and Y images are stored
* image_dir: name of the main image directory, which holds all training and test images
* image_size: resized, square image dimension (all images will be resized to this dim)
* batch_size: number of images in one batch of data
The test data is strictly for feeding to our generators, later on, so we can visualize some generated samples on fixed, test data.
You can see that this function is also responsible for making sure our images are of the right, square size (128x128x3) and converted into Tensor image types.
It's suggested that you use the default values of these parameters.
Note: If you are trying this code on a different set of data, you may get better results with larger image_size and batch_size parameters. If you change the batch_size, make sure that you create complete batches in the training loop otherwise you may get an error when trying to save sample data.
Can you write Python code for it?

def get_data_loader(image_type, image_dir='summer2winter-yosemite',
                    image_size=128, batch_size=16, num_workers=0):
    """Returns training and test data loaders for a given image type, either 'summer' or 'winter'.
       These images will be resized to 128x128x3, by default, converted into Tensors, and normalized.
    """

    # resize and normalize the images
    transform = transforms.Compose([transforms.Resize(image_size), # resize to 128x128
                                    transforms.ToTensor()])

    # get training and test directories
    image_path = './' + image_dir
    train_path = os.path.join(image_path, image_type)
    test_path = os.path.join(image_path, 'test_{}'.format(image_type))

    # define datasets using ImageFolder
    train_dataset = datasets.ImageFolder(train_path, transform)
    test_dataset = datasets.ImageFolder(test_path, transform)

    # create and return DataLoaders
    train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True, num_workers=num_workers)
    test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=False, num_workers=num_workers)

    return train_loader, test_loader

# Create train and test dataloaders for images from the two domains X and Y
# image_type = directory names for our data
dataloader_X, test_dataloader_X = get_data_loader(image_type='summer')
dataloader_Y, test_dataloader_Y = get_data_loader(image_type='winter')


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/cycle-gan/CycleGAN_Exercise.ipynb".
The first task is:
Define the Discriminator Architecture
Your task is to fill in the __init__ function with the specified 5 layer conv net architecture. Both $D_X$ and $D_Y$ have the same architecture, so we only need to define one class, and later instantiate two discriminators.

It's recommended that you use a kernel size of 4x4 and use that to determine the correct stride and padding size for each layer. This Stanford resource may also help in determining stride and padding sizes.

Define your convolutional layers in __init__
Then fill in the forward behavior of the network

The forward function defines how an input image moves through the discriminator, and the most important thing is to pass it through your convolutional layers in order, with a ReLu activation function applied to all but the last layer.
You should not apply a sigmoid activation function to the output, here, and that is because we are planning on using a squared error loss for training. And you can read more about this loss function, later in the notebook.
Can you write Python code for it?

class Discriminator(nn.Module):

    def __init__(self, conv_dim=64):
        super(Discriminator, self).__init__()

        # Define all convolutional layers
        # Should accept an RGB image as input and output a single value

    def forward(self, x):
        # define feedforward behavior

        return x


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/cycle-gan/CycleGAN_Exercise.ipynb".
The first task is:
Define the Generator Architecture

Complete the __init__ function with the specified 3 layer encoder convolutional net, a series of residual blocks (the number of which is given by n_res_blocks), and then a 3 layer decoder transpose convolutional net.
Then complete the forward function to define the forward behavior of the generators. Recall that the last layer has a tanh activation function.

Both $G_{XtoY}$ and $G_{YtoX}$ have the same architecture, so we only need to define one class, and later instantiate two generators.
Can you write Python code for it?

class CycleGenerator(nn.Module):

    def __init__(self, conv_dim=64, n_res_blocks=6):
        super(CycleGenerator, self).__init__()

        # 1. Define the encoder part of the generator

        # 2. Define the resnet part of the generator

        # 3. Define the decoder part of the generator

    def forward(self, x):
        """Given an image x, returns a transformed image."""
        # define feedforward behavior, applying activations as necessary

        return x


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/cycle-gan/CycleGAN_Exercise.ipynb".
The first task is:
Discriminator and Generator Losses
Computing the discriminator and the generator losses are key to getting a CycleGAN to train.
<img src='notebook_images/CycleGAN_loss.png' width=90% height=90% />
Image from original paper by Jun-Yan Zhu et. al.

The CycleGAN contains two mapping functions $G: X \rightarrow Y$ and $F: Y \rightarrow X$, and associated adversarial discriminators $D_Y$ and $D_X$. (a) $D_Y$ encourages $G$ to translate $X$ into outputs indistinguishable from domain $Y$, and vice versa for $D_X$ and $F$.

To further regularize the mappings, we introduce two cycle consistency losses that capture the intuition that if
we translate from one domain to the other and back again we should arrive at where we started. (b) Forward cycle-consistency loss and (c) backward cycle-consistency loss.

Least Squares GANs
We've seen that regular GANs treat the discriminator as a classifier with the sigmoid cross entropy loss function. However, this loss function may lead to the vanishing gradients problem during the learning process. To overcome such a problem, we'll use a least squares loss function for the discriminator. This structure is also referred to as a least squares GAN or LSGAN, and you can read the original paper on LSGANs, here. The authors show that LSGANs are able to generate higher quality images than regular GANs and that this loss type is a bit more stable during training!
Discriminator Losses
The discriminator losses will be mean squared errors between the output of the discriminator, given an image, and the target value, 0 or 1, depending on whether it should classify that image as fake or real. For example, for a real image, x, we can train $D_X$ by looking at how close it is to recognizing and image x as real using the mean squared error:
out_x = D_X(x)
real_err = torch.mean((out_x-1)**2)
Generator Losses
Calculating the generator losses will look somewhat similar to calculating the discriminator loss; there will still be steps in which you generate fake images that look like they belong to...
Can you write Python code for it?

def real_mse_loss(D_out):
    # how close is the produced output from being "real"?

def fake_mse_loss(D_out):
    # how close is the produced output from being "false"?

def cycle_consistency_loss(real_im, reconstructed_im, lambda_weight):
    # calculate reconstruction loss
    # return weighted loss


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/cycle-gan/CycleGAN_Exercise.ipynb".
The first task is:
Evaluate the Result!
As you trained this model, you may have chosen to sample and save the results of your generated images after a certain number of training iterations. This gives you a way to see whether or not your Generators are creating good fake images. For example, the image below depicts real images in the $Y$ set, and the corresponding generated images during different points in the training process. You can see that the generator starts out creating very noisy, fake images, but begins to converge to better representations as it trains (though, not perfect).
<img src='notebook_images/sample-004000-summer2winter.png' width=50% />
Below, you've been given a helper function for displaying generated samples based on the passed in training iteration.
Can you write Python code for it?

import matplotlib.image as mpimg

# helper visualization code
def view_samples(iteration, sample_dir='samples_cyclegan'):

    # samples are named by iteration
    path_XtoY = os.path.join(sample_dir, 'sample-{:06d}-X-Y.png'.format(iteration))
    path_YtoX = os.path.join(sample_dir, 'sample-{:06d}-Y-X.png'.format(iteration))

    # read in those samples
    try:
        x2y = mpimg.imread(path_XtoY)
        y2x = mpimg.imread(path_YtoX)
    except:
        print('Invalid number of iterations.')

    fig, (ax1, ax2) = plt.subplots(figsize=(18,20), nrows=2, ncols=1, sharey=True, sharex=True)
    ax1.imshow(x2y)
    ax1.set_title('X to Y')
    ax2.imshow(y2x)
    ax2.set_title('Y to X')

# view samples at iteration 100
view_samples(100, 'samples_cyclegan')

# view samples at iteration 1000
view_samples(1000, 'samples_cyclegan')


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/cycle-gan/CycleGAN_Exercise.ipynb".
The first task is:
Pre-processing: scaling from -1 to 1
We need to do a bit of pre-processing; we know that the output of our tanh activated generator will contain pixel values in a range from -1 to 1, and so, we need to rescale our training images to a range of -1 to 1. (Right now, they are in a range from 0-1.)
Can you write Python code for it?

# current range
img = images[0]

print('Min: ', img.min())
print('Max: ', img.max())

# helper scale function
def scale(x, feature_range=(-1, 1)):
    ''' Scale takes in an image x and returns that image, scaled
       with a feature_range of pixel values from -1 to 1.
       This function assumes that the input x is already scaled from 0-255.'''

    # scale from 0-1 to feature_range
    min, max = feature_range
    x = x * (max - min) + min
    return x

# scaled range
scaled_img = scale(img)

print('Scaled min: ', scaled_img.min())
print('Scaled max: ', scaled_img.max())


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/cycle-gan/CycleGAN_Exercise.ipynb".
The first task is:
Tips on Training and Loss Patterns
A lot of experimentation goes into finding the best hyperparameters such that the generators and discriminators don't overpower each other. It's often a good starting point to look at existing papers to find what has worked in previous experiments, I'd recommend this DCGAN paper in addition to the original CycleGAN paper to see what worked for them. Then, you can try your own experiments based off of a good foundation.
Discriminator Losses
When you display the generator and discriminator losses you should see that there is always some discriminator loss; recall that we are trying to design a model that can generate good "fake" images. So, the ideal discriminator will not be able to tell the difference between real and fake images and, as such, will always have some loss. You should also see that $D_X$ and $D_Y$ are roughly at the same loss levels; if they are not, this indicates that your training is favoring one type of discriminator over the and you may need to look at biases in your models or data.
Generator Loss
The generator's loss should start significantly higher than the discriminator losses because it is accounting for the loss of both generators and weighted reconstruction errors. You should see this loss decrease a lot at the start of training because initial, generated images are often far-off from being good fakes. After some time it may level off; this is normal since the generator and discriminator are both improving as they train. If you see that the loss is jumping around a lot, over time, you may want to try decreasing your learning rates or changing your cycle consistency loss to be a little more/less weighted.
Can you write Python code for it?

fig, ax = plt.subplots(figsize=(12,8))
losses = np.array(losses)
plt.plot(losses.T[0], label='Discriminator, X', alpha=0.5)
plt.plot(losses.T[1], label='Discriminator, Y', alpha=0.5)
plt.plot(losses.T[2], label='Generators', alpha=0.5)
plt.title("Training Losses")
plt.legend()


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/dcgan-svhn/DCGAN_Exercise.ipynb".
The first task is:
Define the Model
A GAN is comprised of two adversarial networks, a discriminator and a generator.
Discriminator
Here you'll build the discriminator. This is a convolutional classifier like you've built before, only without any maxpooling layers.
* The inputs to the discriminator are 32x32x3 tensor images
* You'll want a few convolutional, hidden layers
* Then a fully connected layer for the output; as before, we want a sigmoid output, but we'll add that in the loss function, BCEWithLogitsLoss, later
<img src='assets/conv_discriminator.png' width=80%/>
For the depths of the convolutional layers I suggest starting with 32 filters in the first layer, then double that depth as you add layers (to 64, 128, etc.). Note that in the DCGAN paper, they did all the downsampling using only strided convolutional layers with no maxpooling layers.
You'll also want to use batch normalization with nn.BatchNorm2d on each layer except the first convolutional layer and final, linear output layer.
Helper conv function
In general, each layer should look something like convolution > batch norm > leaky ReLU, and so we'll define a function to put these layers together. This function will create a sequential series of a convolutional + an optional batch norm layer. We'll create these using PyTorch's Sequential container, which takes in a list of layers and creates layers according to the order that they are passed in to the Sequential constructor.
Note: It is also suggested that you use a kernel_size of 4 and a stride of 2 for strided convolutions.
Can you write Python code for it?

import torch.nn as nn
import torch.nn.functional as F

# helper conv function
def conv(in_channels, out_channels, kernel_size, stride=2, padding=1, batch_norm=True):
    """Creates a convolutional layer, with optional batch normalization.
    """
    layers = []
    conv_layer = nn.Conv2d(in_channels, out_channels,
                           kernel_size, stride, padding, bias=False)

    # append conv layer
    layers.append(conv_layer)

    if batch_norm:
        # append batchnorm layer
        layers.append(nn.BatchNorm2d(out_channels))

    # using Sequential container
    return nn.Sequential(*layers)

class Discriminator(nn.Module):

    def __init__(self, conv_dim=32):
        super(Discriminator, self).__init__()

        # complete init function

    def forward(self, x):
        # complete forward function

        return x


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/dcgan-svhn/DCGAN_Exercise.ipynb".
The first task is:
Generator
Next, you'll build the generator network. The input will be our noise vector z, as before. And, the output will be a $tanh$ output, but this time with size 32x32 which is the size of our SVHN images.
<img src='assets/conv_generator.png' width=80% />
What's new here is we'll use transpose convolutional layers to create our new images.
* The first layer is a fully connected layer which is reshaped into a deep and narrow layer, something like 4x4x512.
* Then, we use batch normalization and a leaky ReLU activation.
* Next is a series of transpose convolutional layers, where you typically halve the depth and double the width and height of the previous layer.
* And, we'll apply batch normalization and ReLU to all but the last of these hidden layers. Where we will just apply a tanh activation.
Helper deconv function
For each of these layers, the general scheme is transpose convolution > batch norm > ReLU, and so we'll define a function to put these layers together. This function will create a sequential series of a transpose convolutional + an optional batch norm layer. We'll create these using PyTorch's Sequential container, which takes in a list of layers and creates layers according to the order that they are passed in to the Sequential constructor.
Note: It is also suggested that you use a kernel_size of 4 and a stride of 2 for transpose convolutions.
Can you write Python code for it?

# helper deconv function
def deconv(in_channels, out_channels, kernel_size, stride=2, padding=1, batch_norm=True):
    """Creates a transposed-convolutional layer, with optional batch normalization.
    """
    ## TODO: Complete this function
    ## create a sequence of transpose + optional batch norm layers

    return None

class Generator(nn.Module):

    def __init__(self, z_size, conv_dim=32):
        super(Generator, self).__init__()

        # complete init function

    def forward(self, x):
        # complete forward function

        return x


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/dcgan-svhn/DCGAN_Exercise.ipynb".
The first task is:
Generator samples from training
Here we can view samples of images from the generator. We'll look at the images we saved during training.
Can you write Python code for it?

# helper function for viewing a list of passed in sample images
def view_samples(epoch, samples):
    fig, axes = plt.subplots(figsize=(16,4), nrows=2, ncols=8, sharey=True, sharex=True)
    for ax, img in zip(axes.flatten(), samples[epoch]):
        img = img.detach().cpu().numpy()
        img = np.transpose(img, (1, 2, 0))
        img = ((img +1)*255 / (2)).astype(np.uint8) # rescale to pixel range (0-255)
        ax.xaxis.set_visible(False)
        ax.yaxis.set_visible(False)
        im = ax.imshow(img.reshape((32,32,3)))

_ = view_samples(-1, samples)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/dcgan-svhn/DCGAN_Exercise.ipynb".
The first task is:
Optimizers
Not much new here, but notice how I am using a small learning rate and custom parameters for the Adam optimizers, This is based on some research into DCGAN model convergence.
Hyperparameters
GANs are very sensitive to hyperparameters. A lot of experimentation goes into finding the best hyperparameters such that the generator and discriminator don't overpower each other. Try out your own hyperparameters or read the DCGAN paper to see what worked for them.
Can you write Python code for it?

import torch.optim as optim

# params
lr =
beta1=
beta2=

# Create optimizers for the discriminator and generator
d_optimizer = optim.Adam(D.parameters(), lr, [beta1, beta2])
g_optimizer = optim.Adam(G.parameters(), lr, [beta1, beta2])


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/dcgan-svhn/DCGAN_Exercise.ipynb".
The first task is:
Pre-processing: scaling from -1 to 1
We need to do a bit of pre-processing; we know that the output of our tanh activated generator will contain pixel values in a range from -1 to 1, and so, we need to rescale our training images to a range of -1 to 1. (Right now, they are in a range from 0-1.)
Can you write Python code for it?

# current range
img = images[0]

print('Min: ', img.min())
print('Max: ', img.max())

# helper scale function
def scale(x, feature_range=(-1, 1)):
    ''' Scale takes in an image x and returns that image, scaled
       with a feature_range of pixel values from -1 to 1.
       This function assumes that the input x is already scaled from 0-1.'''
    # assume x is scaled to (0, 1)
    # scale to feature_range and return scaled x

    return x

# scaled range
scaled_img = scale(img)

print('Scaled min: ', scaled_img.min())
print('Scaled max: ', scaled_img.max())


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/dcgan-svhn/DCGAN_Exercise.ipynb".
The first task is:
Training
Training will involve alternating between training the discriminator and the generator. We'll use our functions real_loss and fake_loss to help us calculate the discriminator losses in all of the following cases.
Discriminator training

Compute the discriminator loss on real, training images
Generate fake images
Compute the discriminator loss on fake, generated images
Add up real and fake loss
Perform backpropagation + an optimization step to update the discriminator's weights

Generator training

Generate fake images
Compute the discriminator loss on fake images, using flipped labels!
Perform backpropagation + an optimization step to update the generator's weights

Saving Samples
As we train, we'll also print out some loss statistics and save some generated "fake" samples.
Evaluation mode
Notice that, when we call our generator to create the samples to display, we set our model to evaluation mode: G.eval(). That's so the batch normalization layers will use the population statistics rather than the batch statistics (as they do during training), and so dropout layers will operate in eval() mode; not turning off any nodes for generating samples.
Can you write Python code for it?

import pickle as pkl

# training hyperparams
num_epochs = 30

# keep track of loss and generated, "fake" samples
samples = []
losses = []

print_every = 300

# Get some fixed data for sampling. These are images that are held
# constant throughout training, and allow us to inspect the model's performance
sample_size=16
fixed_z = np.random.uniform(-1, 1, size=(sample_size, z_size))
fixed_z = torch.from_numpy(fixed_z).float()

# train the network
for epoch in range(num_epochs):

    for batch_i, (real_images, _) in enumerate(train_loader):

        batch_size = real_images.size(0)

        # important rescaling step
        real_images = scale(real_images)

        # ============================================
        #            TRAIN THE DISCRIMINATOR
        # ============================================

        d_optimizer.zero_grad()

        # 1. Train with real images

        # Compute the discriminator losses on real images
        if train_on_gpu:
            real_images = real_images.cuda()

        D_real = D(real_images)
        d_real_loss = real_loss(D_real)

        # 2. Train with fake images

        # Generate fake images
        z = np.random.uniform(-1, 1, size=(batch_size, z_size))
        z = torch.from_numpy(z).float()
        # move x to GPU, if available
        if train_on_gpu:
            z = z.cuda()
        fake_images = G(z)

        # Compute the discriminator losses on fake images
        D_fake = D(fake_images)
        d_fake_loss = fake_loss(D_fake)

        # add up loss and perform backprop
        d_loss = d_real_loss + d_fake_loss
        d_loss.backward()
        d_optimizer.step()

        # =========================================
        #            TRAIN THE GENERATOR
        # =========================================
        g_optimizer.zero_grad()

        # 1. Train with fake images and flipped labels

        # Generate...


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/dcgan-svhn/DCGAN_Solution.ipynb".
The first task is:
Build complete network
Define your models' hyperparameters and instantiate the discriminator and generator from the classes defined above. Make sure you've passed in the correct input arguments.
Can you write Python code for it?

# define hyperparams
conv_dim = 32
z_size = 100

# define discriminator and generator
D = Discriminator(conv_dim)
G = Generator(z_size=z_size, conv_dim=conv_dim)

print(D)
print()
print(G)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/dcgan-svhn/DCGAN_Solution.ipynb".
The first task is:
Define the Model
A GAN is comprised of two adversarial networks, a discriminator and a generator.
Discriminator
Here you'll build the discriminator. This is a convolutional classifier like you've built before, only without any maxpooling layers.
* The inputs to the discriminator are 32x32x3 tensor images
* You'll want a few convolutional, hidden layers
* Then a fully connected layer for the output; as before, we want a sigmoid output, but we'll add that in the loss function, BCEWithLogitsLoss, later
<img src='assets/conv_discriminator.png' width=80%/>
For the depths of the convolutional layers I suggest starting with 32 filters in the first layer, then double that depth as you add layers (to 64, 128, etc.). Note that in the DCGAN paper, they did all the downsampling using only strided convolutional layers with no maxpooling layers.
You'll also want to use batch normalization with nn.BatchNorm2d on each layer except the first convolutional layer and final, linear output layer.
Helper conv function
In general, each layer should look something like convolution > batch norm > leaky ReLU, and so we'll define a function to put these layers together. This function will create a sequential series of a convolutional + an optional batch norm layer. We'll create these using PyTorch's Sequential container, which takes in a list of layers and creates layers according to the order that they are passed in to the Sequential constructor.
Note: It is also suggested that you use a kernel_size of 4 and a stride of 2 for strided convolutions.
Can you write Python code for it?

import torch.nn as nn
import torch.nn.functional as F

# helper conv function
def conv(in_channels, out_channels, kernel_size, stride=2, padding=1, batch_norm=True):
    """Creates a convolutional layer, with optional batch normalization.
    """
    layers = []
    conv_layer = nn.Conv2d(in_channels, out_channels,
                           kernel_size, stride, padding, bias=False)

    # append conv layer
    layers.append(conv_layer)

    if batch_norm:
        # append batchnorm layer
        layers.append(nn.BatchNorm2d(out_channels))

    # using Sequential container
    return nn.Sequential(*layers)

class Discriminator(nn.Module):

    def __init__(self, conv_dim=32):
        super(Discriminator, self).__init__()

        # complete init function
        self.conv_dim = conv_dim

        # 32x32 input
        self.conv1 = conv(3, conv_dim, 4, batch_norm=False) # first layer, no batch_norm
        # 16x16 out
        self.conv2 = conv(conv_dim, conv_dim*2, 4)
        # 8x8 out
        self.conv3 = conv(conv_dim*2, conv_dim*4, 4)
        # 4x4 out

        # final, fully-connected layer
        self.fc = nn.Linear(conv_dim*4*4*4, 1)

    def forward(self, x):
        # all hidden layers + leaky relu activation
        out = F.leaky_relu(self.conv1(x), 0.2)
        out = F.leaky_relu(self.conv2(out), 0.2)
        out = F.leaky_relu(self.conv3(out), 0.2)

        # flatten
        out = out.view(-1, self.conv_dim*4*4*4)

        # final output layer
        out = self.fc(out)
        return out


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/dcgan-svhn/DCGAN_Solution.ipynb".
The first task is:
Discriminator and Generator Losses
Now we need to calculate the losses. And this will be exactly the same as before.
Discriminator Losses

For the discriminator, the total loss is the sum of the losses for real and fake images, d_loss = d_real_loss + d_fake_loss.
Remember that we want the discriminator to output 1 for real images and 0 for fake images, so we need to set up the losses to reflect that.

The losses will by binary cross entropy loss with logits, which we can get with BCEWithLogitsLoss. This combines a sigmoid activation function and and binary cross entropy loss in one function.
For the real images, we want D(real_images) = 1. That is, we want the discriminator to classify the the real images with a label = 1, indicating that these are real. The discriminator loss for the fake data is similar. We want D(fake_images) = 0, where the fake images are the generator output, fake_images = G(z).
Generator Loss
The generator loss will look similar only with flipped labels. The generator's goal is to get D(fake_images) = 1. In this case, the labels are flipped to represent that the generator is trying to fool the discriminator into thinking that the images it generates (fakes) are real!
Can you write Python code for it?

def real_loss(D_out, smooth=False):
    batch_size = D_out.size(0)
    # label smoothing
    if smooth:
        # smooth, real labels = 0.9
        labels = torch.ones(batch_size)*0.9
    else:
        labels = torch.ones(batch_size) # real labels = 1
    # move labels to GPU if available
    if train_on_gpu:
        labels = labels.cuda()
    # binary cross entropy with logits loss
    criterion = nn.BCEWithLogitsLoss()
    # calculate loss
    loss = criterion(D_out.squeeze(), labels)
    return loss

def fake_loss(D_out):
    batch_size = D_out.size(0)
    labels = torch.zeros(batch_size) # fake labels = 0
    if train_on_gpu:
        labels = labels.cuda()
    criterion = nn.BCEWithLogitsLoss()
    # calculate loss
    loss = criterion(D_out.squeeze(), labels)
    return loss


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/dcgan-svhn/DCGAN_Solution.ipynb".
The first task is:
Generator
Next, you'll build the generator network. The input will be our noise vector z, as before. And, the output will be a $tanh$ output, but this time with size 32x32 which is the size of our SVHN images.
<img src='assets/conv_generator.png' width=80% />
What's new here is we'll use transpose convolutional layers to create our new images.
* The first layer is a fully connected layer which is reshaped into a deep and narrow layer, something like 4x4x512.
* Then, we use batch normalization and a leaky ReLU activation.
* Next is a series of transpose convolutional layers, where you typically halve the depth and double the width and height of the previous layer.
* And, we'll apply batch normalization and ReLU to all but the last of these hidden layers. Where we will just apply a tanh activation.
Helper deconv function
For each of these layers, the general scheme is transpose convolution > batch norm > ReLU, and so we'll define a function to put these layers together. This function will create a sequential series of a transpose convolutional + an optional batch norm layer. We'll create these using PyTorch's Sequential container, which takes in a list of layers and creates layers according to the order that they are passed in to the Sequential constructor.
Note: It is also suggested that you use a kernel_size of 4 and a stride of 2 for transpose convolutions.
Can you write Python code for it?

# helper deconv function
def deconv(in_channels, out_channels, kernel_size, stride=2, padding=1, batch_norm=True):
    """Creates a transposed-convolutional layer, with optional batch normalization.
    """
    # create a sequence of transpose + optional batch norm layers
    layers = []
    transpose_conv_layer = nn.ConvTranspose2d(in_channels, out_channels,
                                              kernel_size, stride, padding, bias=False)
    # append transpose convolutional layer
    layers.append(transpose_conv_layer)

    if batch_norm:
        # append batchnorm layer
        layers.append(nn.BatchNorm2d(out_channels))

    return nn.Sequential(*layers)

class Generator(nn.Module):

    def __init__(self, z_size, conv_dim=32):
        super(Generator, self).__init__()

        # complete init function

        self.conv_dim = conv_dim

        # first, fully-connected layer
        self.fc = nn.Linear(z_size, conv_dim*4*4*4)

        # transpose conv layers
        self.t_conv1 = deconv(conv_dim*4, conv_dim*2, 4)
        self.t_conv2 = deconv(conv_dim*2, conv_dim, 4)
        self.t_conv3 = deconv(conv_dim, 3, 4, batch_norm=False)

    def forward(self, x):
        # fully-connected + reshape
        out = self.fc(x)
        out = out.view(-1, self.conv_dim*4, 4, 4) # (batch_size, depth, 4, 4)

        # hidden transpose conv layers + relu
        out = F.relu(self.t_conv1(out))
        out = F.relu(self.t_conv2(out))

        # last layer + tanh activation
        out = self.t_conv3(out)
        out = F.tanh(out)

        return out


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/dcgan-svhn/DCGAN_Solution.ipynb".
The first task is:
Optimizers
Not much new here, but notice how I am using a small learning rate and custom parameters for the Adam optimizers, This is based on some research into DCGAN model convergence.
Hyperparameters
GANs are very sensitive to hyperparameters. A lot of experimentation goes into finding the best hyperparameters such that the generator and discriminator don't overpower each other. Try out your own hyperparameters or read the DCGAN paper to see what worked for them.
Can you write Python code for it?

import torch.optim as optim

# params
lr = 0.0002
beta1=0.5
beta2=0.999 # default value

# Create optimizers for the discriminator and generator
d_optimizer = optim.Adam(D.parameters(), lr, [beta1, beta2])
g_optimizer = optim.Adam(G.parameters(), lr, [beta1, beta2])


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/dcgan-svhn/DCGAN_Solution.ipynb".
The first task is:
Pre-processing: scaling from -1 to 1
We need to do a bit of pre-processing; we know that the output of our tanh activated generator will contain pixel values in a range from -1 to 1, and so, we need to rescale our training images to a range of -1 to 1. (Right now, they are in a range from 0-1.)
Can you write Python code for it?

# current range
img = images[0]

print('Min: ', img.min())
print('Max: ', img.max())

# helper scale function
def scale(x, feature_range=(-1, 1)):
    ''' Scale takes in an image x and returns that image, scaled
       with a feature_range of pixel values from -1 to 1.
       This function assumes that the input x is already scaled from 0-1.'''
    # assume x is scaled to (0, 1)
    # scale to feature_range and return scaled x
    min, max = feature_range
    x = x * (max - min) + min
    return x

# scaled range
scaled_img = scale(img)

print('Scaled min: ', scaled_img.min())
print('Scaled max: ', scaled_img.max())


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/dcgan-svhn/DCGAN_Solution.ipynb".
The first task is:
Training
Training will involve alternating between training the discriminator and the generator. We'll use our functions real_loss and fake_loss to help us calculate the discriminator losses in all of the following cases.
Discriminator training

Compute the discriminator loss on real, training images
Generate fake images
Compute the discriminator loss on fake, generated images
Add up real and fake loss
Perform backpropagation + an optimization step to update the discriminator's weights

Generator training

Generate fake images
Compute the discriminator loss on fake images, using flipped labels!
Perform backpropagation + an optimization step to update the generator's weights

Saving Samples
As we train, we'll also print out some loss statistics and save some generated "fake" samples.
Evaluation mode
Notice that, when we call our generator to create the samples to display, we set our model to evaluation mode: G.eval(). That's so the batch normalization layers will use the population statistics rather than the batch statistics (as they do during training), and so dropout layers will operate in eval() mode; not turning off any nodes for generating samples.
Can you write Python code for it?

import pickle as pkl

# training hyperparams
num_epochs = 50

# keep track of loss and generated, "fake" samples
samples = []
losses = []

print_every = 300

# Get some fixed data for sampling. These are images that are held
# constant throughout training, and allow us to inspect the model's performance
sample_size=16
fixed_z = np.random.uniform(-1, 1, size=(sample_size, z_size))
fixed_z = torch.from_numpy(fixed_z).float()

# train the network
for epoch in range(num_epochs):

    for batch_i, (real_images, _) in enumerate(train_loader):

        batch_size = real_images.size(0)

        # important rescaling step
        real_images = scale(real_images)

        # ============================================
        #            TRAIN THE DISCRIMINATOR
        # ============================================

        d_optimizer.zero_grad()

        # 1. Train with real images

        # Compute the discriminator losses on real images
        if train_on_gpu:
            real_images = real_images.cuda()

        D_real = D(real_images)
        d_real_loss = real_loss(D_real)

        # 2. Train with fake images

        # Generate fake images
        z = np.random.uniform(-1, 1, size=(batch_size, z_size))
        z = torch.from_numpy(z).float()
        # move x to GPU, if available
        if train_on_gpu:
            z = z.cuda()
        fake_images = G(z)

        # Compute the discriminator losses on fake images
        D_fake = D(fake_images)
        d_fake_loss = fake_loss(D_fake)

        # add up loss and perform backprop
        d_loss = d_real_loss + d_fake_loss
        d_loss.backward()
        d_optimizer.step()

        # =========================================
        #            TRAIN THE GENERATOR
        # =========================================
        g_optimizer.zero_grad()

        # 1. Train with fake images and flipped labels

        # Generate...


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/dcgan-svhn/DCGAN_Solution.ipynb".
The first task is:
Training on GPU
Check if you can train on GPU. If you can, set this as a variable and move your models to GPU.

Later, we'll also move any inputs our models and loss functions see (real_images, z, and ground truth labels) to GPU as well.
Can you write Python code for it?

train_on_gpu = torch.cuda.is_available()

if train_on_gpu:
    # move models to GPU
    G.cuda()
    D.cuda()
    print('GPU available for training. Models moved to GPU')
else:
    print('Training on CPU.')


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/gan-mnist/MNIST_GAN_Exercise.ipynb".
The first task is:
Below I'm showing the generated images as the network was training, every 10 epochs.
Can you write Python code for it?

rows = 10 # split epochs into 10, so 100/10 = every 10 epochs
cols = 6
fig, axes = plt.subplots(figsize=(7,12), nrows=rows, ncols=cols, sharex=True, sharey=True)

for sample, ax_row in zip(samples[::int(len(samples)/rows)], axes):
    for img, ax in zip(sample[::int(len(sample)/cols)], ax_row):
        img = img.detach()
        ax.imshow(img.reshape((28,28)), cmap='Greys_r')
        ax.xaxis.set_visible(False)
        ax.yaxis.set_visible(False)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/gan-mnist/MNIST_GAN_Exercise.ipynb".
The first task is:
Discriminator and Generator Losses
Now we need to calculate the losses.
Discriminator Losses

For the discriminator, the total loss is the sum of the losses for real and fake images, d_loss = d_real_loss + d_fake_loss.
Remember that we want the discriminator to output 1 for real images and 0 for fake images, so we need to set up the losses to reflect that.

<img src='assets/gan_pipeline.png' width=70% />
The losses will by binary cross entropy loss with logits, which we can get with BCEWithLogitsLoss. This combines a sigmoid activation function and and binary cross entropy loss in one function.
For the real images, we want D(real_images) = 1. That is, we want the discriminator to classify the the real images with a label = 1, indicating that these are real. To help the discriminator generalize better, the labels are reduced a bit from 1.0 to 0.9. For this, we'll use the parameter smooth; if True, then we should smooth our labels. In PyTorch, this looks like labels = torch.ones(size) * 0.9
The discriminator loss for the fake data is similar. We want D(fake_images) = 0, where the fake images are the generator output, fake_images = G(z).
Generator Loss
The generator loss will look similar only with flipped labels. The generator's goal is to get D(fake_images) = 1. In this case, the labels are flipped to represent that the generator is trying to fool the discriminator into thinking that the images it generates (fakes) are real!
Can you write Python code for it?

# Calculate losses
def real_loss(D_out, smooth=False):
    # compare logits to real labels
    # smooth labels if smooth=True
    loss =
    return loss

def fake_loss(D_out):
    # compare logits to fake labels
    loss =
    return loss


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/gan-mnist/MNIST_GAN_Exercise.ipynb".
The first task is:
Generator samples from training
Here we can view samples of images from the generator. First we'll look at the images we saved during training.
Can you write Python code for it?

# helper function for viewing a list of passed in sample images
def view_samples(epoch, samples):
    fig, axes = plt.subplots(figsize=(7,7), nrows=4, ncols=4, sharey=True, sharex=True)
    for ax, img in zip(axes.flatten(), samples[epoch]):
        img = img.detach()
        ax.xaxis.set_visible(False)
        ax.yaxis.set_visible(False)
        im = ax.imshow(img.reshape((28,28)), cmap='Greys_r')

# Load samples from generator, taken while training
with open('train_samples.pkl', 'rb') as f:
    samples = pkl.load(f)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/gan-mnist/MNIST_GAN_Exercise.ipynb".
The first task is:
It starts out as all noise. Then it learns to make only the center white and the rest black. You can start to see some number like structures appear out of the noise like 1s and 9s.
Sampling from the generator
We can also get completely new images from the generator by using the checkpoint we saved after training. We just need to pass in a new latent vector $z$ and we'll get new samples!
Can you write Python code for it?

# randomly generated, new latent vectors
sample_size=16
rand_z = np.random.uniform(-1, 1, size=(sample_size, z_size))
rand_z = torch.from_numpy(rand_z).float()

G.eval() # eval mode
# generated samples
rand_images = G(rand_z)

# 0 indicates the first set of samples in the passed in list
# and we only have one batch of samples, here
view_samples(0, [rand_images])


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/gan-mnist/MNIST_GAN_Exercise.ipynb".
The first task is:
Model hyperparameters
Can you write Python code for it?

# Discriminator hyperparams

# Size of input image to discriminator (28*28)
input_size =
# Size of discriminator output (real or fake)
d_output_size =
# Size of *last* hidden layer in the discriminator
d_hidden_size =

# Generator hyperparams

# Size of latent vector to give to generator
z_size =
# Size of discriminator output (generated image)
g_output_size =
# Size of *first* hidden layer in the generator
g_hidden_size =


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/gan-mnist/MNIST_GAN_Exercise.ipynb".
The first task is:
Optimizers
We want to update the generator and discriminator variables separately. So, we'll define two separate Adam optimizers.
Can you write Python code for it?

import torch.optim as optim

# learning rate for optimizers
lr = 0.002

# Create optimizers for the discriminator and generator
d_optimizer =
g_optimizer =


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/gan-mnist/MNIST_GAN_Exercise.ipynb".
The first task is:
Training
Training will involve alternating between training the discriminator and the generator. We'll use our functions real_loss and fake_loss to help us calculate the discriminator losses in all of the following cases.
Discriminator training

Compute the discriminator loss on real, training images
Generate fake images
Compute the discriminator loss on fake, generated images
Add up real and fake loss
Perform backpropagation + an optimization step to update the discriminator's weights

Generator training

Generate fake images
Compute the discriminator loss on fake images, using flipped labels!
Perform backpropagation + an optimization step to update the generator's weights

Saving Samples
As we train, we'll also print out some loss statistics and save some generated "fake" samples.
Can you write Python code for it?

import pickle as pkl

# training hyperparams
num_epochs = 40

# keep track of loss and generated, "fake" samples
samples = []
losses = []

print_every = 400

# Get some fixed data for sampling. These are images that are held
# constant throughout training, and allow us to inspect the model's performance
sample_size=16
fixed_z = np.random.uniform(-1, 1, size=(sample_size, z_size))
fixed_z = torch.from_numpy(fixed_z).float()

# train the network
D.train()
G.train()
for epoch in range(num_epochs):

    for batch_i, (real_images, _) in enumerate(train_loader):

        batch_size = real_images.size(0)

        ## Important rescaling step ##
        real_images = real_images*2 - 1  # rescale input images from [0,1) to [-1, 1)

        # ============================================
        #            TRAIN THE DISCRIMINATOR
        # ============================================

        # 1. Train with real images

        # Compute the discriminator losses on real images
        # use smoothed labels

        # 2. Train with fake images

        # Generate fake images
        z = np.random.uniform(-1, 1, size=(batch_size, z_size))
        z = torch.from_numpy(z).float()
        fake_images = G(z)

        # Compute the discriminator losses on fake images

        # add up real and fake losses and perform backprop
        d_loss =

        # =========================================
        #            TRAIN THE GENERATOR
        # =========================================

        # 1. Train with fake images and flipped labels

        # Generate fake images

        # Compute the discriminator losses on fake images
        # using flipped labels!

        # perform backprop
        g_loss =

        # Print some loss stats
        if batch_i % print_every == 0:
            # print discriminator and generator loss
            print('Epoch [{:5d}/{:5d}] |...


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/gan-mnist/MNIST_GAN_Exercise.ipynb".
The first task is:
Training loss
Here we'll plot the training losses for the generator and discriminator, recorded after each epoch.
Can you write Python code for it?

fig, ax = plt.subplots()
losses = np.array(losses)
plt.plot(losses.T[0], label='Discriminator')
plt.plot(losses.T[1], label='Generator')
plt.title("Training Losses")
plt.legend()


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/gan-mnist/MNIST_GAN_Solution.ipynb".
The first task is:
Below I'm showing the generated images as the network was training, every 10 epochs.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/gan-mnist/MNIST_GAN_Solution.ipynb".
The first task is:
Define the Model
A GAN is comprised of two adversarial networks, a discriminator and a generator.
Discriminator
The discriminator network is going to be a pretty typical linear classifier. To make this network a universal function approximator, we'll need at least one hidden layer, and these hidden layers should have one key attribute:

All hidden layers will have a Leaky ReLu activation function applied to their outputs.

Leaky ReLu
We should use a leaky ReLU to allow gradients to flow backwards through the layer unimpeded. A leaky ReLU is like a normal ReLU, except that there is a small non-zero output for negative input values.

Sigmoid Output
We'll also take the approach of using a more numerically stable loss function on the outputs. Recall that we want the discriminator to output a value 0-1 indicating whether an image is real or fake.

We will ultimately use BCEWithLogitsLoss, which combines a sigmoid activation function and and binary cross entropy loss in one function.

So, our final output layer should not have any activation function applied to it.
Can you write Python code for it?

import torch.nn as nn
import torch.nn.functional as F

class Discriminator(nn.Module):

    def __init__(self, input_size, hidden_dim, output_size):
        super(Discriminator, self).__init__()

        # define hidden linear layers
        self.fc1 = nn.Linear(input_size, hidden_dim*4)
        self.fc2 = nn.Linear(hidden_dim*4, hidden_dim*2)
        self.fc3 = nn.Linear(hidden_dim*2, hidden_dim)

        # final fully-connected layer
        self.fc4 = nn.Linear(hidden_dim, output_size)

        # dropout layer
        self.dropout = nn.Dropout(0.3)

    def forward(self, x):
        # flatten image
        x = x.view(-1, 28*28)
        # all hidden layers
        x = F.leaky_relu(self.fc1(x), 0.2) # (input, negative_slope=0.2)
        x = self.dropout(x)
        x = F.leaky_relu(self.fc2(x), 0.2)
        x = self.dropout(x)
        x = F.leaky_relu(self.fc3(x), 0.2)
        x = self.dropout(x)
        # final layer
        out = self.fc4(x)

        return out


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/gan-mnist/MNIST_GAN_Solution.ipynb".
The first task is:
Generator
The generator network will be almost exactly the same as the discriminator network, except that we're applying a tanh activation function to our output layer.
tanh Output
The generator has been found to perform the best with $tanh$ for the generator output, which scales the output to be between -1 and 1, instead of 0 and 1.
<img src='assets/tanh_fn.png' width=40% />
Recall that we also want these outputs to be comparable to the real input pixel values, which are read in as normalized values between 0 and 1.

So, we'll also have to scale our real input images to have pixel values between -1 and 1 when we train the discriminator.

I'll do this in the training loop, later on.
Can you write Python code for it?

class Generator(nn.Module):

    def __init__(self, input_size, hidden_dim, output_size):
        super(Generator, self).__init__()

        # define hidden linear layers
        self.fc1 = nn.Linear(input_size, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, hidden_dim*2)
        self.fc3 = nn.Linear(hidden_dim*2, hidden_dim*4)

        # final fully-connected layer
        self.fc4 = nn.Linear(hidden_dim*4, output_size)

        # dropout layer
        self.dropout = nn.Dropout(0.3)

    def forward(self, x):
        # all hidden layers
        x = F.leaky_relu(self.fc1(x), 0.2) # (input, negative_slope=0.2)
        x = self.dropout(x)
        x = F.leaky_relu(self.fc2(x), 0.2)
        x = self.dropout(x)
        x = F.leaky_relu(self.fc3(x), 0.2)
        x = self.dropout(x)
        # final layer with tanh applied
        out = F.tanh(self.fc4(x))

        return out


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/gan-mnist/MNIST_GAN_Solution.ipynb".
The first task is:
Generator samples from training
Here we can view samples of images from the generator. First we'll look at the images we saved during training.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/gan-mnist/MNIST_GAN_Solution.ipynb".
The first task is:
It starts out as all noise. Then it learns to make only the center white and the rest black. You can start to see some number like structures appear out of the noise like 1s and 9s.
Sampling from the generator
We can also get completely new images from the generator by using the checkpoint we saved after training. We just need to pass in a new latent vector $z$ and we'll get new samples!
Can you write Python code for it?


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/gan-mnist/MNIST_GAN_Solution.ipynb".
The first task is:
Model hyperparameters
Can you write Python code for it?

# Discriminator hyperparams

# Size of input image to discriminator (28*28)
input_size = 784
# Size of discriminator output (real or fake)
d_output_size = 1
# Size of last hidden layer in the discriminator
d_hidden_size = 32

# Generator hyperparams

# Size of latent vector to give to generator
z_size = 100
# Size of discriminator output (generated image)
g_output_size = 784
# Size of first hidden layer in the generator
g_hidden_size = 32


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/gan-mnist/MNIST_GAN_Solution.ipynb".
The first task is:
Optimizers
We want to update the generator and discriminator variables separately. So, we'll define two separate Adam optimizers.
Can you write Python code for it?

import torch.optim as optim

# Optimizers
lr = 0.002

# Create optimizers for the discriminator and generator
d_optimizer = optim.Adam(D.parameters(), lr)
g_optimizer = optim.Adam(G.parameters(), lr)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/gan-mnist/MNIST_GAN_Solution.ipynb".
The first task is:
Training
Training will involve alternating between training the discriminator and the generator. We'll use our functions real_loss and fake_loss to help us calculate the discriminator losses in all of the following cases.
Discriminator training

Compute the discriminator loss on real, training images
Generate fake images
Compute the discriminator loss on fake, generated images
Add up real and fake loss
Perform backpropagation + an optimization step to update the discriminator's weights

Generator training

Generate fake images
Compute the discriminator loss on fake images, using flipped labels!
Perform backpropagation + an optimization step to update the generator's weights

Saving Samples
As we train, we'll also print out some loss statistics and save some generated "fake" samples.
Can you write Python code for it?

import pickle as pkl

# training hyperparams
num_epochs = 100

# keep track of loss and generated, "fake" samples
samples = []
losses = []

print_every = 400

# Get some fixed data for sampling. These are images that are held
# constant throughout training, and allow us to inspect the model's performance
sample_size=16
fixed_z = np.random.uniform(-1, 1, size=(sample_size, z_size))
fixed_z = torch.from_numpy(fixed_z).float()

# train the network
D.train()
G.train()
for epoch in range(num_epochs):

    for batch_i, (real_images, _) in enumerate(train_loader):

        batch_size = real_images.size(0)

        ## Important rescaling step ##
        real_images = real_images*2 - 1  # rescale input images from [0,1) to [-1, 1)

        # ============================================
        #            TRAIN THE DISCRIMINATOR
        # ============================================

        d_optimizer.zero_grad()

        # 1. Train with real images

        # Compute the discriminator losses on real images
        # smooth the real labels
        D_real = D(real_images)
        d_real_loss = real_loss(D_real, smooth=True)

        # 2. Train with fake images

        # Generate fake images
        z = np.random.uniform(-1, 1, size=(batch_size, z_size))
        z = torch.from_numpy(z).float()
        fake_images = G(z)

        # Compute the discriminator losses on fake images
        D_fake = D(fake_images)
        d_fake_loss = fake_loss(D_fake)

        # add up loss and perform backprop
        d_loss = d_real_loss + d_fake_loss
        d_loss.backward()
        d_optimizer.step()

        # =========================================
        #            TRAIN THE GENERATOR
        # =========================================
        g_optimizer.zero_grad()

        # 1. Train with fake images and flipped labels

        # Generate fake images
        z = np.random.uniform(-1, 1,...


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/gan-mnist/MNIST_GAN_Solution.ipynb".
The first task is:
Training loss
Here we'll plot the training losses for the generator and discriminator, recorded after each epoch.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/gan-mnist/MNIST_GAN_Solution.ipynb".
The first task is:
Visualize the data
Can you write Python code for it?

# obtain one batch of training images
dataiter = iter(train_loader)
images, labels = dataiter.next()
images = images.numpy()

# get one image from the batch
img = np.squeeze(images[0])

fig = plt.figure(figsize = (3,3))
ax = fig.add_subplot(111)
ax.imshow(img, cmap='gray')


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/project-face-generation/dlnd_face_generation.ipynb".
The first task is:
Create a DataLoader
Exercise: Create a DataLoader celeba_train_loader with appropriate hyperparameters.
Call the above function and create a dataloader to view images.
* You can decide on any reasonable batch_size parameter
* Your image_size must be 32. Resizing the data to a smaller size will make for faster training, while still creating convincing images of faces!
Can you write Python code for it?

# Define function hyperparameters
batch_size =
img_size =

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
# Call your function and get a dataloader
celeba_train_loader = get_dataloader(batch_size, img_size)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/project-face-generation/dlnd_face_generation.ipynb".
The first task is:
Define the Model
A GAN is comprised of two adversarial networks, a discriminator and a generator.
Discriminator
Your first task will be to define the discriminator. This is a convolutional classifier like you've built before, only without any maxpooling layers. To deal with this complex data, it's suggested you use a deep network with normalization. You are also allowed to create any helper functions that may be useful.
Exercise: Complete the Discriminator class

The inputs to the discriminator are 32x32x3 tensor images
The output should be a single value that will indicate whether a given image is real or fake
Can you write Python code for it?

import torch.nn as nn
import torch.nn.functional as F

class Discriminator(nn.Module):

    def __init__(self, conv_dim):
        """
        Initialize the Discriminator Module
        :param conv_dim: The depth of the first convolutional layer
        """
        super(Discriminator, self).__init__()

        # complete init function

    def forward(self, x):
        """
        Forward propagation of the neural network
        :param x: The input to the neural network
        :return: Discriminator logits; the output of the neural network
        """
        # define feedforward behavior

        return x

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_discriminator(Discriminator)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/project-face-generation/dlnd_face_generation.ipynb".
The first task is:
Discriminator and Generator Losses
Now we need to calculate the losses for both types of adversarial networks.
Discriminator Losses

For the discriminator, the total loss is the sum of the losses for real and fake images, d_loss = d_real_loss + d_fake_loss.
Remember that we want the discriminator to output 1 for real images and 0 for fake images, so we need to set up the losses to reflect that.

Generator Loss
The generator loss will look similar only with flipped labels. The generator's goal is to get the discriminator to think its generated images are real.
Exercise: Complete real and fake loss functions
You may choose to use either cross entropy or a least squares error loss to complete the following real_loss and fake_loss functions.
Can you write Python code for it?

def real_loss(D_out):
    '''Calculates how close discriminator outputs are to being real.
       param, D_out: discriminator logits
       return: real loss'''
    loss =
    return loss

def fake_loss(D_out):
    '''Calculates how close discriminator outputs are to being fake.
       param, D_out: discriminator logits
       return: fake loss'''
    loss =
    return loss


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/project-face-generation/dlnd_face_generation.ipynb".
The first task is:
Exercise: Define model hyperparameters
Can you write Python code for it?

# Define model hyperparams
d_conv_dim =
g_conv_dim =
z_size =

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
D, G = build_network(d_conv_dim, g_conv_dim, z_size)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/project-face-generation/dlnd_face_generation.ipynb".
The first task is:
Exercise: Pre-process your image data and scale it to a pixel range of -1 to 1
You need to do a bit of pre-processing; you know that the output of a tanh activated generator will contain pixel values in a range from -1 to 1, and so, we need to rescale our training images to a range of -1 to 1. (Right now, they are in a range from 0-1.)
Can you write Python code for it?

# TODO: Complete the scale function
def scale(x, feature_range=(-1, 1)):
    ''' Scale takes in an image x and returns that image, scaled
       with a feature_range of pixel values from -1 to 1.
       This function assumes that the input x is already scaled from 0-1.'''
    # assume x is scaled to (0, 1)
    # scale to feature_range and return scaled x

    return x

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
# check scaled range
# should be close to -1 to 1
img = images[0]
scaled_img = scale(img)

print('Min: ', scaled_img.min())
print('Max: ', scaled_img.max())


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/project-face-generation/dlnd_face_generation.ipynb".
The first task is:
Generator
The generator should upsample an input and generate a new image of the same size as our training data 32x32x3. This should be mostly transpose convolutional layers with normalization applied to the outputs.
Exercise: Complete the Generator class

The inputs to the generator are vectors of some length z_size
The output should be a image of shape 32x32x3
Can you write Python code for it?

class Generator(nn.Module):

    def __init__(self, z_size, conv_dim):
        """
        Initialize the Generator Module
        :param z_size: The length of the input latent vector, z
        :param conv_dim: The depth of the inputs to the *last* transpose convolutional layer
        """
        super(Generator, self).__init__()

        # complete init function

    def forward(self, x):
        """
        Forward propagation of the neural network
        :param x: The input to the neural network
        :return: A 32x32x3 Tensor image as output
        """
        # define feedforward behavior

        return x

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_generator(Generator)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/project-face-generation/dlnd_face_generation.ipynb".
The first task is:
Generator samples from training
View samples of images from the generator, and answer a question about the strengths and weaknesses of your trained models.
Can you write Python code for it?

# helper function for viewing a list of passed in sample images
def view_samples(epoch, samples):
    fig, axes = plt.subplots(figsize=(16,4), nrows=2, ncols=8, sharey=True, sharex=True)
    for ax, img in zip(axes.flatten(), samples[epoch]):
        img = img.detach().cpu().numpy()
        img = np.transpose(img, (1, 2, 0))
        img = ((img + 1)*255 / (2)).astype(np.uint8)
        ax.xaxis.set_visible(False)
        ax.yaxis.set_visible(False)
        im = ax.imshow(img.reshape((32,32,3)))

# Load samples from generator, taken while training
with open('train_samples.pkl', 'rb') as f:
    samples = pkl.load(f)

_ = view_samples(-1, samples)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/project-face-generation/dlnd_face_generation.ipynb".
The first task is:
Initialize the weights of your networks
To help your models converge, you should initialize the weights of the convolutional and linear layers in your model. From reading the original DCGAN paper, they say:

All weights were initialized from a zero-centered Normal distribution with standard deviation 0.02.

So, your next task will be to define a weight initialization function that does just this!
You can refer back to the lesson on weight initialization or even consult existing model code, such as that from the networks.py file in CycleGAN Github repository to help you complete this function.
Exercise: Complete the weight initialization function

This should initialize only convolutional and linear layers
Initialize the weights to a normal distribution, centered around 0, with a standard deviation of 0.02.
The bias terms, if they exist, may be left alone or set to 0.
Can you write Python code for it?

def weights_init_normal(m):
    """
    Applies initial weights to certain layers in a model .
    The weights are taken from a normal distribution
    with mean = 0, std dev = 0.02.
    :param m: A module or layer in a network
    """
    # classname will be something like:
    # `Conv`, `BatchNorm2d`, `Linear`, etc.
    classname = m.__class__.__name__

    # TODO: Apply initial weights to convolutional and linear layers


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/project-face-generation/dlnd_face_generation.ipynb".
The first task is:
Next, you can view some images! You should seen square images of somewhat-centered faces.
Note: You'll need to convert the Tensor images into a NumPy type and transpose the dimensions to correctly display an image, suggested imshow code is below, but it may not be perfect.
Can you write Python code for it?

# helper display function
def imshow(img):
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
# obtain one batch of training images
dataiter = iter(celeba_train_loader)
images, _ = dataiter.next() # _ for no labels

# plot the images in the batch, along with the corresponding labels
fig = plt.figure(figsize=(20, 4))
plot_size=20
for idx in np.arange(plot_size):
    ax = fig.add_subplot(2, plot_size/2, idx+1, xticks=[], yticks=[])
    imshow(images[idx])


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/project-face-generation/dlnd_face_generation.ipynb".
The first task is:
Optimizers
Exercise: Define optimizers for your Discriminator (D) and Generator (G)
Define optimizers for your models with appropriate hyperparameters.
Can you write Python code for it?

import torch.optim as optim

# Create optimizers for the discriminator D and generator G
d_optimizer =
g_optimizer =


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/project-face-generation/dlnd_face_generation.ipynb".
The first task is:
Training loss
Plot the training losses for the generator and discriminator, recorded after each epoch.
Can you write Python code for it?

fig, ax = plt.subplots()
losses = np.array(losses)
plt.plot(losses.T[0], label='Discriminator', alpha=0.5)
plt.plot(losses.T[1], label='Generator', alpha=0.5)
plt.title("Training Losses")
plt.legend()


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/project-face-generation/dlnd_face_generation.ipynb".
The first task is:
Training on GPU
Check if you can train on GPU. Here, we'll set this as a boolean variable train_on_gpu. Later, you'll be responsible for making sure that

Models,
Model inputs, and
Loss function arguments

Are moved to GPU, where appropriate.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import torch

# Check for a GPU
train_on_gpu = torch.cuda.is_available()
if not train_on_gpu:
    print('No GPU found. Please use a GPU to train your neural network.')
else:
    print('Training on GPU!')


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/GAN/project-face-generation/dlnd_face_generation.ipynb".
The first task is:
Visualize the CelebA Data
The CelebA dataset contains over 200,000 celebrity images with annotations. Since you're going to be generating faces, you won't need the annotations, you'll only need the images. Note that these are color images with 3 color channels (RGB) each.
Pre-process and Load the Data
Since the project's main focus is on building the GANs, we've done some of the pre-processing for you. Each of the CelebA images has been cropped to remove parts of the image that don't include a face, then resized down to 64x64x3 NumPy images. This pre-processed dataset is a smaller subset of the very large CelebA data.

There are a few other steps that you'll need to transform this data and create a DataLoader.

Exercise: Complete the following get_dataloader function, such that it satisfies these requirements:

Your images should be square, Tensor images of size image_size x image_size in the x and y dimension.
Your function should return a DataLoader that shuffles and batches these Tensor images.

ImageFolder
To create a dataset given a directory of images, it's recommended that you use PyTorch's ImageFolder wrapper, with a root directory processed_celeba_small/ and data transformation passed in.
Can you write Python code for it?

# necessary imports
import torch
from torchvision import datasets
from torchvision import transforms

def get_dataloader(batch_size, image_size, data_dir='processed_celeba_small/'):
    """
    Batch the neural network data using DataLoader
    :param batch_size: The size of each batch; the number of images in a batch
    :param img_size: The square size of the image data (x, y)
    :param data_dir: Directory where image data is located
    :return: DataLoader with batched data
    """

    # TODO: Implement function and return a dataloader

    return None


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/attention/Attention_Basics.ipynb".
The first task is:
Annotations Matrix
Let's now look at scoring all the annotations at once. To do that, here's our annotation matrix:
Can you write Python code for it?


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/attention/Attention_Basics.ipynb".
The first task is:
Even when knowing which annotation will get the most focus, it's interesting to see how drastic softmax makes the end score become. The first and last annotation had the respective scores of 927 and 929. But after softmax, the attention they'll get is 0.12 and 0.88 respectively.
Applying the scores back on the annotations
Now that we have our scores, let's multiply each annotation by its score to proceed closer to the attention context vector. This is the multiplication part of this formula (we'll tackle the summation part in the latter cells)
<img src="images/Context_vector.png" />
Can you write Python code for it?

def apply_attention_scores(attention_weights, annotations):
    # TODO: Multiple the annotations by their weights
    return

applied_attention = apply_attention_scores(attention_weights, annotations)
applied_attention


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/attention/Attention_Basics.ipynb".
The first task is:
IMPLEMENT: Scoring All Annotations at Once
Let's calculate the scores of all the annotations in one step using matrix multiplication. Let's continue to us the dot scoring method
<img src="images/scoring_functions.png" />
To do that, we'll have to transpose dec_hidden_state and matrix multiply it with annotations.
Can you write Python code for it?

def dot_attention_score(dec_hidden_state, annotations):
    # TODO: return the product of dec_hidden_state transpose and enc_hidden_states
    return

attention_weights_raw = dot_attention_score(dec_hidden_state, annotations)
attention_weights_raw


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/attention/Attention_Basics.ipynb".
The first task is:
IMPLEMENT: Scoring a Single Annotation
Let's calculate the dot product of a single annotation. NumPy's dot() is a good candidate for this operation
Can you write Python code for it?

def single_dot_attention_score(dec_hidden_state, enc_hidden_state):
    # TODO: return the dot product of the two vectors
    return

single_dot_attention_score(dec_hidden_state, annotation)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/attention/Attention_Basics.ipynb".
The first task is:
Let's visualize how the context vector looks now that we've applied the attention scores back on it:
Can you write Python code for it?

# Let's visualize our annotations after applying attention to them
ax = sns.heatmap(applied_attention, annot=True, cmap=sns.light_palette("orange", as_cmap=True), linewidths=1)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/attention/Attention_Basics.ipynb".
The first task is:
Let's visualize this vector:
Can you write Python code for it?

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Let's visualize our decoder hidden state
plt.figure(figsize=(1.5, 4.5))
sns.heatmap(np.transpose(np.matrix(dec_hidden_state)), annot=True, cmap=sns.light_palette("purple", as_cmap=True), linewidths=1)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/attention/Attention_Basics.ipynb".
The first task is:
Our first scoring function will score a single annotation (encoder hidden state), which looks like this:
Can you write Python code for it?

annotation = [3,12,45] #e.g. Encoder hidden state

# Let's visualize the single annotation
plt.figure(figsize=(1.5, 4.5))
sns.heatmap(np.transpose(np.matrix(annotation)), annot=True, cmap=sns.light_palette("orange", as_cmap=True), linewidths=1)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/attention/Attention_Basics_Solution.ipynb".
The first task is:
And it can be visualized like this (each column is a hidden state of an encoder time step):
Can you write Python code for it?

# Let's visualize our annotation (each column is an annotation)
ax = sns.heatmap(annotations, annot=True, cmap=sns.light_palette("orange", as_cmap=True), linewidths=1)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/attention/Attention_Basics_Solution.ipynb".
The first task is:
Annotations Matrix
Let's now look at scoring all the annotations at once. To do that, here's our annotation matrix:
Can you write Python code for it?


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/attention/Attention_Basics_Solution.ipynb".
The first task is:
Contrast this with the raw annotations visualized earlier in the notebook, and we can see that the second and third annotations (columns) have been nearly wiped out. The first annotation maintains some of its value, and the fourth annotation is the most pronounced.
Calculating the Attention Context Vector
All that remains to produce our attention context vector now is to sum up the four columns to produce a single attention context vector
Can you write Python code for it?

def calculate_attention_vector(applied_attention):
    return np.sum(applied_attention, axis=1)

attention_vector = calculate_attention_vector(applied_attention)
attention_vector

# Let's visualize the attention context vector
plt.figure(figsize=(1.5, 4.5))
sns.heatmap(np.transpose(np.matrix(attention_vector)), annot=True, cmap=sns.light_palette("Blue", as_cmap=True), linewidths=1)


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/attention/Attention_Basics_Solution.ipynb".
The first task is:
Let's visualize how the context vector looks now that we've applied the attention scores back on it:
Can you write Python code for it?


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/attention/Attention_Basics_Solution.ipynb".
The first task is:
Looking at these scores, can you guess which of the four vectors will get the most attention from the decoder at this time step?
Softmax
Now that we have our scores, let's apply softmax:
<img src="images/softmax.png" />
Can you write Python code for it?

def softmax(x):
    x = np.array(x, dtype=np.float128)
    e_x = np.exp(x)
    return e_x / e_x.sum(axis=0)

attention_weights = softmax(attention_weights_raw)
attention_weights


I am working on the file "DEEP LEARNING/Pytorch from scratch/TODO/attention/Attention_Basics_Solution.ipynb".
The first task is:
Our first scoring function will score a single annotation (encoder hidden state), which looks like this:
Can you write Python code for it?


I am working on the file "DEEP LEARNING/Pytorch from scratch/word2vec-embeddings/Negative_Sampling.ipynb".
The first task is:
Dictionaries
Next, I'm creating two dictionaries to convert words to integers and back again (integers to words). This is again done with a function in the utils.py file. create_lookup_tables takes in a list of words in a text and returns two dictionaries.

The integers are assigned in descending frequency order, so the most frequent word ("the") is given the integer 0 and the next most frequent is 1, and so on.

Once we have our dictionaries, the words are converted to integers and stored in the list int_words.
Can you write Python code for it?

vocab_to_int, int_to_vocab = utils.create_lookup_tables(words)
int_words = [vocab_to_int[word] for word in words]

print(int_words[:30])


I am working on the file "DEEP LEARNING/Pytorch from scratch/word2vec-embeddings/Negative_Sampling.ipynb".
The first task is:
Generating Batches
Here's a generator function that returns batches of input and target data for our model, using the get_target function from above. The idea is that it grabs batch_size words from a words list. Then for each of those batches, it gets the target words in a window.
Can you write Python code for it?

def get_batches(words, batch_size, window_size=5):
    ''' Create a generator of word batches as a tuple (inputs, targets) '''

    n_batches = len(words)//batch_size

    # only full batches
    words = words[:n_batches*batch_size]

    for idx in range(0, len(words), batch_size):
        x, y = [], []
        batch = words[idx:idx+batch_size]
        for ii in range(len(batch)):
            batch_x = batch[ii]
            batch_y = get_target(batch, ii, window_size)
            y.extend(batch_y)
            x.extend([batch_x]*len(batch_y))
        yield x, y

int_text = [i for i in range(20)]
x,y = next(get_batches(int_text, batch_size=4, window_size=5))

print('x\n', x)
print('y\n', y)


I am working on the file "DEEP LEARNING/Pytorch from scratch/word2vec-embeddings/Negative_Sampling.ipynb".
The first task is:
Making batches
Now that our data is in good shape, we need to get it into the proper form to pass it into our network. With the skip-gram architecture, for each word in the text, we want to define a surrounding context and grab all the words in a window around that word, with size $C$.
From Mikolov et al.:
"Since the more distant words are usually less related to the current word than those close to it, we give less weight to the distant words by sampling less from those words in our training examples... If we choose $C = 5$, for each training word we will select randomly a number $R$ in range $[ 1: C ]$, and then use $R$ words from history and $R$ words from the future of the current word as correct labels."

Exercise: Implement a function get_target that receives a list of words, an index, and a window size, then returns a list of words in the window around the index. Make sure to use the algorithm described above, where you chose a random number of words to from the window.

Say, we have an input and we're interested in the idx=2 token, 741:
[5233, 58, 741, 10571, 27349, 0, 15067, 58112, 3580, 58, 10712]
For R=2, get_target should return a list of four values:
[5233, 58, 10571, 27349]
Can you write Python code for it?

def get_target(words, idx, window_size=5):
    ''' Get a list of words in a window around an index. '''

    R = np.random.randint(1, window_size+1)
    start = idx - R if (idx - R) > 0 else 0
    stop = idx + R
    target_words = words[start:idx] + words[idx+1:stop+1]

    return list(target_words)

# test your code!

# run this cell multiple times to check for random window selection
int_text = [i for i in range(10)]
print('Input: ', int_text)
idx=5 # word index of interest

target = get_target(int_text, idx=idx, window_size=5)
print('Target: ', target)  # you should get some indices around the idx


I am working on the file "DEEP LEARNING/Pytorch from scratch/word2vec-embeddings/Negative_Sampling.ipynb".
The first task is:
Pre-processing
Here I'm fixing up the text to make training easier. This comes from the utils.py file. The preprocess function does a few things:

It converts any punctuation into tokens, so a period is changed to <PERIOD>. In this data set, there aren't any periods, but it will help in other NLP problems.
It removes all words that show up five or fewer times in the dataset. This will greatly reduce issues due to noise in the data and improve the quality of the vector representations.
It returns a list of words in the text.

This may take a few seconds to run, since our text file is quite large. If you want to write your own functions for this stuff, go for it!
Can you write Python code for it?

import utils

# get list of words
words = utils.preprocess(text)
print(words[:30])

# print some stats about this word data
print("Total words in text: {}".format(len(words)))
print("Unique words: {}".format(len(set(words)))) # `set` removes any duplicate words


I am working on the file "DEEP LEARNING/Pytorch from scratch/word2vec-embeddings/Negative_Sampling.ipynb".
The first task is:
Subsampling
Words that show up often such as "the", "of", and "for" don't provide much context to the nearby words. If we discard some of them, we can remove some of the noise from our data and in return get faster training and better representations. This process is called subsampling by Mikolov. For each word $w_i$ in the training set, we'll discard it with probability given by
$$ P(w_i) = 1 - \sqrt{\frac{t}{f(w_i)}} $$
where $t$ is a threshold parameter and $f(w_i)$ is the frequency of word $w_i$ in the total dataset.

Implement subsampling for the words in int_words. That is, go through int_words and discard each word given the probablility $P(w_i)$ shown above. Note that $P(w_i)$ is the probability that a word is discarded. Assign the subsampled data to train_words.
Can you write Python code for it?

from collections import Counter
import random
import numpy as np

threshold = 1e-5
word_counts = Counter(int_words)
#print(list(word_counts.items())[0])  # dictionary of int_words, how many times they appear

total_count = len(int_words)
freqs = {word: count/total_count for word, count in word_counts.items()}
p_drop = {word: 1 - np.sqrt(threshold/freqs[word]) for word in word_counts}
# discard some frequent words, according to the subsampling equation
# create a new list of words for training
train_words = [word for word in int_words if random.random() < (1 - p_drop[word])]

print(train_words[:30])


I am working on the file "DEEP LEARNING/Pytorch from scratch/word2vec-embeddings/Negative_Sampling.ipynb".
The first task is:
Training
Below is our training loop, and I recommend that you train on GPU, if available.
Can you write Python code for it?

device = 'cuda' if torch.cuda.is_available() else 'cpu'

# Get our noise distribution
# Using word frequencies calculated earlier in the notebook
word_freqs = np.array(sorted(freqs.values(), reverse=True))
unigram_dist = word_freqs/word_freqs.sum()
noise_dist = torch.from_numpy(unigram_dist**(0.75)/np.sum(unigram_dist**(0.75)))

# instantiating the model
embedding_dim = 300
model = SkipGramNeg(len(vocab_to_int), embedding_dim, noise_dist=noise_dist).to(device)

# using the loss that we defined
criterion = NegativeSamplingLoss()
optimizer = optim.Adam(model.parameters(), lr=0.003)

print_every = 1500
steps = 0
epochs = 5

# train for some number of epochs
for e in range(epochs):

    # get our input, target batches
    for input_words, target_words in get_batches(train_words, 512):
        steps += 1
        inputs, targets = torch.LongTensor(input_words), torch.LongTensor(target_words)
        inputs, targets = inputs.to(device), targets.to(device)

        # input, output, and noise vectors
        input_vectors = model.forward_input(inputs)
        output_vectors = model.forward_output(targets)
        noise_vectors = model.forward_noise(inputs.shape[0], 5)

        # negative sampling loss
        loss = criterion(input_vectors, output_vectors, noise_vectors)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # loss stats
        if steps % print_every == 0:
            print("Epoch: {}/{}".format(e+1, epochs))
            print("Loss: ", loss.item()) # avg batch loss at this point in training
            valid_examples, valid_similarities = cosine_similarity(model.in_embed, device=device)
            _, closest_idxs = valid_similarities.topk(6)

            valid_examples, closest_idxs = valid_examples.to('cpu'), closest_idxs.to('cpu')
            for ii, valid_idx in enumerate(valid_examples):
                closest_words = [int_to_vocab[idx.item()] for idx in closest_idxs[ii]][1:]
                print(int_to_vocab[valid_idx.item()] + " | " + ',...


I am working on the file "DEEP LEARNING/Pytorch from scratch/word2vec-embeddings/Negative_Sampling.ipynb".
The first task is:
Visualizing the word vectors
Below we'll use T-SNE to visualize how our high-dimensional word vectors cluster together. T-SNE is used to project these vectors into two dimensions while preserving local stucture. Check out this post from Christopher Olah to learn more about T-SNE and other ways to visualize high-dimensional data.
Can you write Python code for it?

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import matplotlib.pyplot as plt
from sklearn.manifold import TSNE

# getting embeddings from the embedding layer of our model, by name
embeddings = model.in_embed.weight.to('cpu').data.numpy()

viz_words = 380
tsne = TSNE()
embed_tsne = tsne.fit_transform(embeddings[:viz_words, :])

fig, ax = plt.subplots(figsize=(16, 16))
for idx in range(viz_words):
    plt.scatter(*embed_tsne[idx, :], color='steelblue')
    plt.annotate(int_to_vocab[idx], (embed_tsne[idx, 0], embed_tsne[idx, 1]), alpha=0.7)


I am working on the file "DEEP LEARNING/fastai kaggle rossman - tabular regression.ipynb".
The first task is:
All
Can you write Python code for it?

m = md.get_learner(emb_szs, len(df.columns)-len(cat_vars),
                   0.04, 1, [1000,500], [0.001,0.01], y_range=y_range)
lr = 1e-3

m.fit(lr, 1, metrics=[exp_rmspe])

m.fit(lr, 3, metrics=[exp_rmspe])

m.fit(lr, 3, metrics=[exp_rmspe], cycle_len=1)


I am working on the file "DEEP LEARNING/fastai kaggle rossman - tabular regression.ipynb".
The first task is:
An even better option for picking a validation set is using the exact same length of time period as the test set uses - this is implemented here:
Can you write Python code for it?

val_idx = np.flatnonzero(
    (df.index<=datetime.datetime(2014,9,17)) & (df.index>=datetime.datetime(2014,8,1)))

val_idx=[0]


I am working on the file "DEEP LEARNING/fastai kaggle rossman - tabular regression.ipynb".
The first task is:
Create datasets
In addition to the provided data, we will be using external datasets put together by participants in the Kaggle competition. You can download all of them here.
For completeness, the implementation used to put them together is included below.
Can you write Python code for it?

def concat_csvs(dirname):
    path = f'{PATH}{dirname}'
    filenames=glob(f"{PATH}/*.csv")

    wrote_header = False
    with open(f"{path}.csv","w") as outputfile:
        for filename in filenames:
            name = filename.split(".")[0]
            with open(filename) as f:
                line = f.readline()
                if not wrote_header:
                    wrote_header = True
                    outputfile.write("file,"+line)
                for line in f:
                     outputfile.write(name + "," + line)
                outputfile.write("\n")

# concat_csvs('googletrend')
# concat_csvs('weather')


I am working on the file "DEEP LEARNING/fastai kaggle rossman - tabular regression.ipynb".
The first task is:
Data Cleaning / Feature Engineering
As a structured data problem, we necessarily have to go through all the cleaning and feature engineering, even though we're using a neural network.
Can you write Python code for it?

train, store, store_states, state_names, googletrend, weather, test = tables

len(train),len(test)


I am working on the file "DEEP LEARNING/fastai kaggle rossman - tabular regression.ipynb".
The first task is:
Durations
It is common when working with time series data to extract data that explains relationships across rows as opposed to columns, e.g.:
* Running averages
* Time until next event
* Time since last event
This is often difficult to do with most table manipulation frameworks, since they are designed to work with relationships across columns. As such, we've created a class to handle this type of data.
We'll define a function get_elapsed for cumulative counting across a sorted dataframe. Given a particular field fld to monitor, this function will start tracking time since the last occurrence of that field. When the field is seen again, the counter is set to zero.
Upon initialization, this will result in datetime na's until the field is encountered. This is reset every time a new store is seen. We'll see how to use this shortly.
Can you write Python code for it?

def get_elapsed(fld, pre):
    day1 = np.timedelta64(1, 'D')
    last_date = np.datetime64()
    last_store = 0
    res = []

    for s,v,d in zip(df.Store.values,df[fld].values, df.Date.values):
        if s != last_store:
            last_date = np.datetime64()
            last_store = s
        if v: last_date = d
        res.append(((d-last_date).astype('timedelta64[D]') / day1))
    df[pre+fld] = res