I am working on the file "FAI_old/lesson1/dogs_cats_redux.ipynb".
The first task is:
Finetuning and Training
Can you write Python code for it?

%cd $DATA_HOME_DIR

#Set path to sample/ path if desired
path = DATA_HOME_DIR + '/' #'/sample/'
test_path = DATA_HOME_DIR + '/test/' #We use all the test data
results_path=DATA_HOME_DIR + '/results/'
train_path=path + '/train/'
valid_path=path + '/valid/'

#import Vgg16 helper class
vgg = Vgg16()

#Set constants. You can experiment with no_of_epochs to improve the model
batch_size=64
no_of_epochs=3

#Finetune the model
batches = vgg.get_batches(train_path, batch_size=batch_size)
val_batches = vgg.get_batches(valid_path, batch_size=batch_size*2)
vgg.finetune(batches)

#Not sure if we set this for all fits
vgg.model.optimizer.lr = 0.01

#Notice we are passing in the validation dataset to the fit() method
#For each epoch we test our model against the validation set
latest_weights_filename = None
for epoch in range(no_of_epochs):
    print "Running epoch: %d" % epoch
    vgg.fit(batches, val_batches, nb_epoch=1)
#     latest_weights_filename = 'ft%d.h5' % epoch
#     vgg.model.save_weights(results_path+latest_weights_filename)
print "Completed %s fit operations" % no_of_epochs


I am working on the file "FAI_old/lesson1/dogs_cats_redux.ipynb".
The first task is:
Generate Predictions
Let's use our new model to make predictions on the test dataset
Can you write Python code for it?

batches, preds = vgg.test(test_path, batch_size = batch_size*2)

#For every image, vgg.test() generates two probabilities
#based on how we've ordered the cats/dogs directories.
#It looks like column one is cats and column two is dogs
print preds[:5]

filenames = batches.filenames
print filenames[:5]

#You can verify the column ordering by viewing some images
from PIL import Image
Image.open(test_path + filenames[2])

#Save our test results arrays so we can use them again later
save_array(results_path + 'test_preds.dat', preds)
save_array(results_path + 'filenames.dat', filenames)


I am working on the file "FAI_old/lesson1/dogs_cats_redux.ipynb".
The first task is:
Here we join the two columns into an array of [imageId, isDog]
Can you write Python code for it?

subm = np.stack([ids,isdog], axis=1)
subm[:5]

%cd $DATA_HOME_DIR
submission_file_name = 'submission1.csv'
np.savetxt(submission_file_name, subm, fmt='%d,%.5f', header='id,label', comments='')

from IPython.display import FileLink
%cd $LESSON_HOME_DIR
FileLink('data/redux/'+submission_file_name)


I am working on the file "FAI_old/lesson1/dogs_cats_redux.ipynb".
The first task is:
Log Loss doesn't support probability values of 0 or 1--they are undefined (and we have many). Fortunately, Kaggle helps us by offsetting our 0s and 1s by a very small value. So if we upload our submission now we will have lots of .99999999 and .000000001 values. This seems good, right?
Not so. There is an additional twist due to how log loss is calculated--log loss rewards predictions that are confident and correct (p=.9999,label=1), but it punishes predictions that are confident and wrong far more (p=.0001,label=1). See visualization below.
Can you write Python code for it?

#Visualize Log Loss when True value = 1
#y-axis is log loss, x-axis is probabilty that label = 1
#As you can see Log Loss increases rapidly as we approach 0
#But increases slowly as our predicted probability gets closer to 1
import matplotlib.pyplot as plt
import numpy as np
from sklearn.metrics import log_loss

x = [i*.0001 for i in range(1,10000)]
y = [log_loss([1],[[i*.0001,1-(i*.0001)]],eps=1e-15) for i in range(1,10000,1)]

plt.plot(x, y)
plt.axis([-.05, 1.1, -.8, 10])
plt.title("Log Loss when true label = 1")
plt.xlabel("predicted probability")
plt.ylabel("log loss")

plt.show()

#So to play it safe, we use a sneaky trick to round down our edge predictions
#Swap all ones with .95 and all zeros with .05
isdog = isdog.clip(min=0.05, max=0.95)

#Extract imageIds from the filenames in our test/unknown directory
filenames = batches.filenames
ids = np.array([int(f[8:f.find('.')]) for f in filenames])


I am working on the file "FAI_old/lesson1/dogs_cats_redux.ipynb".
The first task is:
Perhaps the most common way to analyze the result of a classification model is to use a confusion matrix. Scikit-learn has a convenient function we can use for this purpose:
Can you write Python code for it?

from sklearn.metrics import confusion_matrix
cm = confusion_matrix(expected_labels, our_labels)


I am working on the file "FAI_old/lesson1/dogs_cats_redux.ipynb".
The first task is:
Rearrange image files into their respective directories
Can you write Python code for it?

#Divide cat/dog images into separate directories

%cd $DATA_HOME_DIR/sample/train
%mkdir cats
%mkdir dogs
%mv cat.*.jpg cats/
%mv dog.*.jpg dogs/

%cd $DATA_HOME_DIR/sample/valid
%mkdir cats
%mkdir dogs
%mv cat.*.jpg cats/
%mv dog.*.jpg dogs/

%cd $DATA_HOME_DIR/valid
%mkdir cats
%mkdir dogs
%mv cat.*.jpg cats/
%mv dog.*.jpg dogs/

%cd $DATA_HOME_DIR/train
%mkdir cats
%mkdir dogs
%mv cat.*.jpg cats/
%mv dog.*.jpg dogs/

# Create single 'unknown' class for test set
%cd $DATA_HOME_DIR/test
%mv *.jpg unknown/


I am working on the file "FAI_old/lesson1/dogs_cats_redux.ipynb".
The first task is:
Validate Predictions
Keras' fit() function conveniently shows us the value of the loss function, and the accuracy, after every epoch ("epoch" refers to one full run through all training examples). The most important metrics for us to look at are for the validation set, since we want to check for over-fitting.

Tip: with our first model we should try to overfit before we start worrying about how to reduce over-fitting - there's no point even thinking about regularization, data augmentation, etc if you're still under-fitting! (We'll be looking at these techniques shortly).

As well as looking at the overall metrics, it's also a good idea to look at examples of each of:
1. A few correct labels at random
2. A few incorrect labels at random
3. The most correct labels of each class (ie those with highest probability that are correct)
4. The most incorrect labels of each class (ie those with highest probability that are incorrect)
5. The most uncertain labels (ie those with probability closest to 0.5).
Let's see what we can learn from these examples. (In general, this is a particularly useful technique for debugging problems in the model. However, since this model is so simple, there may not be too much to learn at this stage.)
Calculate predictions on validation set, so we can find correct and incorrect examples:
Can you write Python code for it?

vgg.model.load_weights(results_path+latest_weights_filename)

val_batches, probs = vgg.test(valid_path, batch_size = batch_size)

filenames = val_batches.filenames
expected_labels = val_batches.classes #0 or 1

#Round our predictions to 0/1 to generate labels
our_predictions = probs[:,0]
our_labels = np.round(1-our_predictions)

from keras.preprocessing import image

#Helper function to plot images by index in the validation set
#Plots is a helper function in utils.py
def plots_idx(idx, titles=None):
    plots([image.load_img(valid_path + filenames[i]) for i in idx], titles=titles)

#Number of images to view for each visualization task
n_view = 4

#1. A few correct labels at random
correct = np.where(our_labels==expected_labels)[0]
print "Found %d correct labels" % len(correct)
idx = permutation(correct)[:n_view]
plots_idx(idx, our_predictions[idx])

#2. A few incorrect labels at random
incorrect = np.where(our_labels!=expected_labels)[0]
print "Found %d incorrect labels" % len(incorrect)
idx = permutation(incorrect)[:n_view]
plots_idx(idx, our_predictions[idx])

#3a. The images we most confident were cats, and are actually cats
correct_cats = np.where((our_labels==0) & (our_labels==expected_labels))[0]
print "Found %d confident correct cats labels" % len(correct_cats)
most_correct_cats = np.argsort(our_predictions[correct_cats])[::-1][:n_view]
plots_idx(correct_cats[most_correct_cats], our_predictions[correct_cats][most_correct_cats])

#3b. The images we most confident were dogs, and are actually dogs
correct_dogs = np.where((our_labels==1) & (our_labels==expected_labels))[0]
print "Found %d confident correct dogs labels" % len(correct_dogs)
most_correct_dogs = np.argsort(our_predictions[correct_dogs])[:n_view]
plots_idx(correct_dogs[most_correct_dogs], our_predictions[correct_dogs][most_correct_dogs])

#4a. The images we were most confident were cats, but are actually dogs
incorrect_cats = np.where((our_labels==0) & (our_labels!=expected_labels))[0]
print "Found %d incorrect cats" %...


I am working on the file "FAI_old/lesson1/dogs_cats_redux.ipynb".
The first task is:
We can just print out the confusion matrix, or we can show a graphical view (which is mainly useful for dependents with a larger number of categories).
Can you write Python code for it?


I am working on the file "FAI_old/lesson1/lesson1.ipynb".
The first task is:
As well as the architecture, we need the weights that the VGG creators trained. The weights are the part of the model that is learnt from the data, whereas the architecture is pre-defined based on the nature of the problem.
Downloading pre-trained weights is much preferred to training the model ourselves, since otherwise we would have to download the entire Imagenet archive, and train the model for many days! It's very helpful when researchers release their weights, as they did here.
Can you write Python code for it?

fpath = get_file('vgg16.h5', FILES_PATH+'vgg16.h5', cache_subdir='models')
model.load_weights(fpath)


I am working on the file "FAI_old/lesson1/lesson1.ipynb".
The first task is:
Calling finetune() modifies the model such that it will be trained based on the data in the batches provided - in this case, to predict either 'dog' or 'cat'.
Can you write Python code for it?


I am working on the file "FAI_old/lesson1/lesson1.ipynb".
The first task is:
Define path to data: (It's a good idea to put it in a subdirectory of your notebooks folder, and then exclude that directory from git control by adding it to .gitignore.)
Can you write Python code for it?

# path = "data/dogscats/"
path = "data/"
#path = "data/dogscats/sample/"


I am working on the file "FAI_old/lesson1/lesson1.ipynb".
The first task is:
Finally, we fit() the parameters of the model using the training data, reporting the accuracy on the validation set after every epoch. (An epoch is one full pass through the training data.)
Can you write Python code for it?


I am working on the file "FAI_old/lesson1/lesson1.ipynb".
The first task is:
From here we can use exactly the same steps as before to look at predictions from the model.
Can you write Python code for it?

batches = get_batches('train', batch_size=batch_size)
val_batches = get_batches('valid', batch_size=batch_size)
imgs,labels = next(batches)

# This shows the 'ground truth'
plots(imgs, titles=labels)


I am working on the file "FAI_old/lesson1/lesson1.ipynb".
The first task is:
Getting imagenet predictions
The setup of the imagenet model is now complete, so all we have to do is grab a batch of images and call predict() on them.
Can you write Python code for it?


I am working on the file "FAI_old/lesson1/lesson1.ipynb".
The first task is:
Here's a few examples of the categories we just imported:
Can you write Python code for it?


I am working on the file "FAI_old/lesson1/lesson1.ipynb".
The first task is:
Keras provides functionality to create batches of data from directories containing images; all we have to do is to define the size to resize the images to, what type of labels to create, whether to randomly shuffle the images, and how many images to include in each batch. We use this little wrapper to define some helpful defaults appropriate for imagenet data:
Can you write Python code for it?

def get_batches(dirname, gen=image.ImageDataGenerator(), shuffle=True,
                batch_size=batch_size, class_mode='categorical'):
    return gen.flow_from_directory(path+dirname, target_size=(224,224),
                class_mode=class_mode, shuffle=shuffle, batch_size=batch_size)


I am working on the file "FAI_old/lesson1/lesson1.ipynb".
The first task is:
Model creation
Creating the model involves creating the model architecture, and then loading the model weights into that architecture. We will start by defining the basic pieces of the VGG architecture.
VGG has just one type of convolutional block, and one type of fully connected ('dense') block. Here's the convolutional block definition:
Can you write Python code for it?

def ConvBlock(layers, model, filters):
    for i in range(layers):
        model.add(ZeroPadding2D((1,1)))
        model.add(Convolution2D(filters, 3, 3, activation='relu'))
    model.add(MaxPooling2D((2,2), strides=(2,2)))


I am working on the file "FAI_old/lesson1/lesson1.ipynb".
The first task is:
Now we're ready to define the VGG model architecture - look at how simple it is, now that we have the basic blocks defined!
Can you write Python code for it?

def VGG_16():
    model = Sequential()
    model.add(Lambda(vgg_preprocess, input_shape=(3,224,224)))

    ConvBlock(2, model, 64)
    ConvBlock(2, model, 128)
    ConvBlock(3, model, 256)
    ConvBlock(3, model, 512)
    ConvBlock(3, model, 512)

    model.add(Flatten())
    FCBlock(model)
    FCBlock(model)
    model.add(Dense(1000, activation='softmax'))
    return model


I am working on the file "FAI_old/lesson1/lesson1.ipynb".
The first task is:
That shows all of the steps involved in using the Vgg16 class to create an image recognition model using whatever labels you are interested in. For instance, this process could classify paintings by style, or leaves by type of disease, or satellite photos by type of crop, and so forth.
Next up, we'll dig one level deeper to see what's going on in the Vgg16 class.
Create a VGG model from scratch in Keras
For the rest of this tutorial, we will not be using the Vgg16 class at all. Instead, we will recreate from scratch the functionality we just used. This is not necessary if all you want to do is use the existing model - but if you want to create your own models, you'll need to understand these details. It will also help you in the future when you debug any problems with your models, since you'll understand what's going on behind the scenes.
Model setup
We need to import all the modules we'll be using from numpy, scipy, and keras:
Can you write Python code for it?

from numpy.random import random, permutation
from scipy import misc, ndimage
from scipy.ndimage.interpolation import zoom

import keras
from keras import backend as K
from keras.utils.data_utils import get_file
from keras.models import Sequential, Model
from keras.layers.core import Flatten, Dense, Dropout, Lambda
from keras.layers import Input
from keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D
from keras.optimizers import SGD, RMSprop
from keras.preprocessing import image


I am working on the file "FAI_old/lesson1/lesson1.ipynb".
The first task is:
The VGG model returns 1,000 probabilities for each image, representing the probability that the model assigns to each possible imagenet category for each image. By finding the index with the largest probability (with np.argmax()) we can find the predicted label.
Can you write Python code for it?

def pred_batch(imgs):
    preds = model.predict(imgs)
    idxs = np.argmax(preds, axis=1)

    print('Shape: {}'.format(preds.shape))
    print('First 5 classes: {}'.format(classes[:5]))
    print('First 5 probabilities: {}\n'.format(preds[0, :5]))
    print('Predictions prob/class: ')

    for i in range(len(idxs)):
        idx = idxs[i]
        print ('  {:.4f}/{}'.format(preds[i, idx], classes[idx]))

pred_batch(imgs)


I am working on the file "FAI_old/lesson1/lesson1.ipynb".
The first task is:
The code above will work for any image recognition task, with any number of categories! All you have to do is to put your images into one folder per category, and run the code above.
Let's take a look at how this works, step by step...
Use Vgg16 for basic image recognition
Let's start off by using the Vgg16 class to recognise the main imagenet category for each image.
We won't be able to enter the Cats vs Dogs competition with an Imagenet model alone, since 'cat' and 'dog' are not categories in Imagenet - instead each individual breed is a separate category. However, we can use it to see how well it can recognise the images, which is a good first step.
First, create a Vgg16 object:
Can you write Python code for it?


I am working on the file "FAI_old/lesson1/lesson1.ipynb".
The first task is:
Vgg16 is built on top of Keras (which we will be learning much more about shortly!), a flexible, easy to use deep learning library that sits on top of Theano or Tensorflow. Keras reads groups of images and labels in batches, using a fixed directory structure, where images from each category for training must be placed in a separate folder.
Let's grab batches of data from our training folder:
Can you write Python code for it?

batches = vgg.get_batches(path+'train', batch_size=4)


I am working on the file "FAI_old/lesson1/lesson1.ipynb".
The first task is:
We can now pass the images to Vgg16's predict() function to get back probabilities, category indexes, and category names for each image's VGG prediction.
Can you write Python code for it?


I am working on the file "FAI_old/lesson1/lesson1.ipynb".
The first task is:
We have created a file most imaginatively called 'utils.py' to store any little convenience functions we'll want to use. We will discuss these as we use them.
Can you write Python code for it?

LESSON_HOME_DIR = os.getcwd()
import sys
sys.path.insert(1, os.path.join(LESSON_HOME_DIR, '../utils'))
import utils; reload(utils)
from utils import plots


I am working on the file "FAI_old/lesson1/lesson1.ipynb".
The first task is:
We'll learn about what these different blocks do later in the course. For now, it's enough to know that:

Convolution layers are for finding patterns in images
Dense (fully connected) layers are for combining patterns across an image

Now that we've defined the architecture, we can create the model like any python object:
Can you write Python code for it?


I am working on the file "FAI_old/lesson1/lesson1.ipynb".
The first task is:
When the VGG model was trained in 2014, the creators subtracted the average of each of the three (R,G,B) channels first, so that the data for each channel had a mean of zero. Furthermore, their software that expected the channels to be in B,G,R order, whereas Python by default uses R,G,B. We need to preprocess our data to make these two changes, so that it is compatible with the VGG model:
Can you write Python code for it?

# Mean of each channel as provided by VGG researchers
vgg_mean = np.array([123.68, 116.779, 103.939]).reshape((3,1,1))

def vgg_preprocess(x):
    x = x - vgg_mean     # subtract mean
    return x[:, ::-1]    # reverse axis bgr->rgb


I am working on the file "FAI_old/lesson1/lesson1_HW.ipynb".
The first task is:
Calling fineunte() modifies the model so that it'll be trained based on the data in the batches provided: dog vs cat. The model's parameters are then fitted using the training data, and accuracy is reported using teh validation set. An epoch is 1 full pass through the training data.
Can you write Python code for it?

vgg.finetune(batches)
vgg.fit(batches, val_batches, nb_epoch=1)


I am working on the file "FAI_old/lesson1/lesson1_HW.ipynb".
The first task is:
Get some batches from the sample training folder. vgg.get_batches(..) is an iterator that returns a batch of images and their labels, indefinitely.
Can you write Python code for it?

batches = vgg.get_batches(path+'train',batch_size=4)

imgs, labels = next(batches)
plots(imgs, titles=labels)


I am working on the file "FAI_old/lesson1/lesson1_HW.ipynb".
The first task is:
The model is ready to classify dogs vs cats. In order to complete this assignment, we'll have to create a csv (comma-separated-values) file of IDs and labels of the full 25,000 set of images and upload it to Kaggle for scoring.
First modify the path to the full dataset:
Can you write Python code for it?


I am working on the file "FAI_old/lesson1/lesson1_HW.ipynb".
The first task is:
The syntax for getting the IDs below is: make an array of the index of the period plus one, to fourth-from-end character, of each item f in filenames. The file extensions are all ".jpg" which is 4 characters long, and the ID number of each file comes after a period as such: "dog.0010.jpg" or "cat.1001.jpg" & etc.
Can you write Python code for it?

vgg.model.load_weights(path+'ft1.hd5')

# ids = np.array([int(f[f.find('.')+1:-4]) for f in filenames])
ids = np.array([int(f[8:f.find('.')]) for f in filenames])
dog_predictions = predictions[:,1]

# NOTE: this is important due to logloss error: clip the probabilities
dog_predictions = dog_predictions.clip(min=0.05, max=0.95)
# dog_predictions = np.clip(dog_predictions, 0.05, 0.95) # in case .clip doesn't work
# you know what's funny, as I'm writing this I have no clue where the clip function is coming from..
# Python? NumPy? Utils?

submission = np.stack([ids, dog_predictions], axis=1)
submission_file_name = 'submission_HW1_.csv'
np.savetxt(submission_file_name, submission, fmt='%d,%.5f',header='id,label',comments='')

%pwd


I am working on the file "FAI_old/lesson1/lesson1_HW.ipynb".
The first task is:
To get an idea of the categories Vgg16 usually works with, here're the first 5:
Can you write Python code for it?


I am working on the file "FAI_old/lesson1/lesson1_HW.ipynb".
The first task is:
vgg.test(..) will return the entire set of batches & predictions (labels). We'll extract the ID-number from batches.filenames, and the label from the second column of predictions.
Can you write Python code for it?

batch_size = 32
batches, predictions = vgg.test(path+'test',batch_size=batch_size)
filenames = batches.filenames


I am working on the file "FAI_old/lesson2/L2HW.ipynb".
The first task is:
1. Basic Linear Model
Can you write Python code for it?

LM = keras.model.Sequential([Dense(Num_Classes, input_shape=(784,))])

LM.compile(optimizer=SGD(lr=0.01), loss='mse')
# LM.compile(optimizer=RMSprop(lr=0.01), loss='mse')


I am working on the file "FAI_old/lesson2/L2HW.ipynb".
The first task is:
2. 1-Layer Neural Network
3. Finetuned VGG16
Can you write Python code for it?

import os, sys

sys.path.insert(os.path.join(1, '../utils/'))
import Vgg16


I am working on the file "FAI_old/lesson2/L2HW1_LM.ipynb".
The first task is:
Okay, so we got those separated.. let's do the same thing as before: single Linear Model Perceptron Layer, but with a validation set to check against.
Can you write Python code for it?

# The stuff above's One-Hotted.. so don't do it again..

# # I forgot the onehot encode the labels after loading from disk
# val_labels = onehot(val_labels)
# trn_2_labels = onehot(trn_2_labels)
# print(val_labels.shape)
# print(trn_2_labels.shape)

LM = Sequential([Dense(nb_classes, input_dim=input_dim)])
LM.compile(optimizer='sgd', loss='mse', metrics=['accuracy'])
# LM = Sequential([Dense(nb_classes, activation='sigmoid', input_dim=input_dim)])

LM.fit(newtrn_input, newtrn_labels, nb_epoch=5, batch_size=4,
       validation_data=(val_input, val_labels))

LM2 = Sequential([Dense(nb_classes, activation='sigmoid', input_dim=input_dim)])
LM2.compile(optimizer='sgd', loss='mse', metrics=['accuracy'])
LM2.fit(trn_input, trn_labels, nb_epoch=5, batch_size=4, verbose=1)


I am working on the file "FAI_old/lesson2/L2HW1_LM.ipynb".
The first task is:
Okay, so.. training data is the number of images X (number of pixels + label). The label just adds 1 to the vector's length because it's just a decimal {0..9}, giving 42k X 785..
The training input vector has the label removed so its 42k images X 784 pixels..
The trianing labels vector is one-hot encoded to a ten-bit, & of course the 42k images..
So... nothing is saying I can't just cut input & labels and separate those into new training and validation sets. Not sure why I was getting crashes the other way, but we'll see if this works (it should). Ooo, maybe I.. okay maybe I made a mistake with leaving the labels on or something, before.
Can you write Python code for it?

# Old
# print(val_data.shape, val_input.shape, val_labels.shape)
# print(trn_2_input.shape, trn_2_input.shape, trn_2_labels.shape)

val_input  = trn_input[:2000]
val_labels = trn_labels[:2000]

newtrn_input = trn_input[2000:]
newtrn_labels = trn_labels[2000:]

print(val_input.shape, val_labels.shape)
print(newtrn_input.shape, newtrn_labels.shape)


I am working on the file "FAI_old/lesson2/L2HW1_LM.ipynb".
The first task is:
Unfortunately I don't yet know how to separate a .csv file into a validation set. Wait that sounds easy. Take a random permutation, or for the lazy: the first X amount of inputs and labels from the training set and call that validation. Oh. Okay. Maybe I should do that.
Can you write Python code for it?

# # Wondering if a crash I had earlier was because One-Hotting did something to the labels..
# trn_data = pd.read_csv(path + 'train.csv')
# trn_labels = trn_data.ix[:,0].values.astype('int32')
# trn_input = (trn_data.ix[:,1:].values).astype('float32')

# test_data has 42,000 elements. I'll take 2,000 for validation.
# val_data = trn_data[:2000]
# trn_2_data = trn_data[2000:]

# val_input  = val_data.ix[:,0].values.astype('int32')
# val_labels = (val_data.ix[:,1:].values).astype('float32')

# trn_2_input = trn_2_data.ix[:,0].values.astype('int32')
# trn_2_labels = (trn_2_data.ix[:,1:].values).astype('float32')

# trying to do this in a way that doesn't kill the kernel
print(trn_data.shape)
print(trn_input.shape)
print(trn_labels.shape)


I am working on the file "FAI_old/lesson2/keras_sequential_tutorial.ipynb".
The first task is:
Compilation
Before training a model, the learning process must be configured via the compile moethod. It has 3 parameters:
* Optimizer. Either string identifier of an existing optimizer (rmspropr, adagrad, etc), or an instance of the Optimizer class. See: Optimizers
* Loss Function. String identifier of an existing loss fn (categorical_crossentropy, mse, etc), or an objective function. See: Losses
* List of Metrics. For any classification problem you'll want to set this to metrics=['accuracy']. Metric: string identifier of existing metric, or custom metric function.
Can you write Python code for it?

# For a multi-class classification problem
model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
# For binary classification
model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])
# For mean squared error regression
model.compile(optimizer='rmsprop',
              loss='mse')

# For custom metrics
import keras.backend as K

def mean_pred(y_true y_pred):
    return K.mean(y_pred)

model.compile(optimizer='rmspropr',
              loss='binary_crossentropy',
              metrics=['accuracy', mean_pred])


I am working on the file "FAI_old/lesson2/keras_sequential_tutorial.ipynb".
The first task is:
Keras Examples:
Github Folder
Multilayer Perceptron (MLP) for multi-class softmax classificaiton:
Can you write Python code for it?

from keras.models import Sequential
from keras.layers improt Dense, Dropout, Activation
from keras.optimizers import SGD

# Generate dummy data
import numpy as np
x_train = np.random.random((1000, 20))
y_train = keras.utils.to_categorical(np.random.randint(10, size=(1000, 1)), num_classes=10)
x_test = np.random.random((100, 20))
y_test = keras.utils.to_categorical(np.random.randint(10, size=(100, 1)), num_classes=10)

model = Sequential()
# Dense(64) is a fully-connected layer with 64 hidden nodes
# in the first layer, specify the expected input data shape
# here: 20-dimensional vectors.
model.add(Dense(64, activation='reulu', input_dim=20))
model.add(Dropout(0.5))
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(10, activation='softmax'))

sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy',
              optimizer=sgd,
              metrics=['accuracy'])

mode.fit(x_train, y_train,
         epochs=20,
         batch_size=128)
score = model.evaluate(x_test, y_test, batch_size=128)


I am working on the file "FAI_old/lesson2/keras_sequential_tutorial.ipynb".
The first task is:
MLP for binary classification:
Can you write Python code for it?

import numpy as np
from keras.models import Sequential
from keras.layers import Dense, Dropout

# Generate dummy data
x_train = np.random.random((1000, 20))
y_train = np.random.randint(2, size=(1000, 1))
x_test = np.random.random((100, 20))
y_test = np.random.randint(2, size=(100, 1))

model = Sequential()
model.add(Dense(64, input_dim=20, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])

model.fit(x_train, y_train,
          epochs=20,
          batch_size=128)
score = model.evaluate(x_test, y_test, batch_size=128)


I am working on the file "FAI_old/lesson2/keras_sequential_tutorial.ipynb".
The first task is:
Specifying the input shape
The first layer in a Sequential model needs to receive info about its input shape -- & only the first because the following layers can do automatic shape inference.

Pass an input_shape arg to the first layer. Tuple of ints or None, where None indicates any (+) int may be expected.
2D layers, like Dense (aka. Fully-Connected/Linear) via argument input_dim; and some 3D temporal layers via input_dim and input_length.
To specify a fixed batch size for inputs (useful for stateful RecNets), pass a batch_size argument. If you pass both batch_size=32 & input_shape=(6,8), it'll expect every batch of inputs to have batch shape (32, 6, 8)

Both of the below are strictly equivalent:
Can you write Python code for it?

model = Sequential()
mode.add(Dense(32, input_shape=(784,)))

model = Sequential()
mode.add(Dense(32, input_dim=784))


I am working on the file "FAI_old/lesson2/keras_sequential_tutorial.ipynb".
The first task is:
VGG-like ConvNet:
Can you write Python code for it?


I am working on the file "FAI_old/lesson2/keras_sequential_tutorial.ipynb".
The first task is:
You can also simply add layers via the .add() method:
Can you write Python code for it?

model = Sequential()
model.add(Dense(32, input_dim=784))
model.add(Activation('relu'))


I am working on the file "FAI_old/lesson2/lesson2_LM_SGD_Optz_codealong.ipynb".
The first task is:
Coming up with an update function that'll make our guesses of a & b closer to what a & b actually are, each time it is called.
Can you write Python code for it?

def update():
    global a_guess, b_guess
    y_pred = lin(a_guess, b_guess, x)
    dydb = 2*(y_pred - y)
    dyda = x*dydb
    a_guess -= Lr * dyda.mean() # new guess is minus deriv * (a little bit)
    b_guess -= Lr * dydb.mean()


I am working on the file "FAI_old/lesson2/lesson2_LM_SGD_Optz_codealong.ipynb".
The first task is:
Confirm that our line eventually, actually, fits our data, via animation.
Can you write Python code for it?

fig = plt.figure(figsize=(5,4), dpi=100)
plt.scatter(x, y)
line, = plt.plot(x, lin(a_guess, b_guess, x))
plt.close()

def animate(i):
    line.set_ydata(lin(a_guess, b_guess, x))
    for i in xrange(10): update()
    return line,

ani = animation.FuncAnimation(fig, animate, np.arange(0, 40), interval=100)
ani


I am working on the file "FAI_old/lesson2/lesson2_LM_SGD_Optz_codealong.ipynb".
The first task is:
Let's start out with a line: y = ax + b, where a = 3, b = 8; but where we don't know what a & b are, and starting with guesses for both, use a linear model to find them
Can you write Python code for it?

a_guess = -1
b_guess = 1
avg_loss(y, a_guess, b_guess, x)

Lr = 0.01    # below thanks to Wolfram Alpha
# d[(y - y_pred)**2,b] = d[(y - (a*x+b))**2, b] = 2*(b + a*x - y)
# d[(y - y_pred)**2,a] = d[(y - (a*x+b))**2, a] = 2*x*(b+a*x-y) = x * dy/db


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
& Now we're ready to fit the model!  RMSprop is somewhat better than SGD. It's a minor tweak on SGD that tends to be much faster.
Can you write Python code for it?

batch_size=4

lm.fit(trn_features, trn_labels, batch_size=batch_size, nb_epoch=3,
       validation_data = (val_features, val_labels))

# let's have a look at our model
lm.summary()


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
(I actually ran 9, bc on a tiny set of 350 images it took a bit more to improve: no change on the 1st, dropped to .90 on the 2nd, and finally up to .94 on the final)
Here we haven't done any finetuning. All we did was take the ImageNet model of predictions, and built a model that maps from those predictions to either 'Cat' or 'Dog'
This is actually what most amatuer Machine Learning researchers do. They take a pretrained model, they grab the outputs, stick it into a linear model -- and it actually often works pretty well!
To get this 94% accuracy, we haven't done used any magical libraries at all. We just grabbed our batches up, we turned the images into a Numpy array, we took the Numpy array and ran model.predict(..) on them, we grabbed our labels and One-Hot Encoded them, and finally we took the 1Ht Enc labels and the 1,000 probabilities and fed them to a Linear Model with a thousand inputs and 2 outputs - and trained it and ended up with a validationa ccuracy of 0.9400
1.3.3 About Activation Functions
The last thing we're going to do is take this and turn it into a finetuning model. For that we need to understand activation functions. We've been looking at our Linear Model as a series of matrix multiplies. But a series of matrix multiplies is itself a matrix multiply --> a series of linear models is itself a linear model. Deep Learning must be doing something more than just this. At each stage (layer) it is putting the activations, the results of the previous layer, through a non-Linearity of some sort. tanh, sigmoid, max(0,x) (ReLU), etc.
Using the activation functions at each layer, we now have a genuine, modern (ca.2017), Deep Learning Neural Network. This kind of NN is capable of approximating any given function of arbitrary complexity.
A series of matrix-multiplies & activation (sa. ReLU) is actually what's going on in a DLNN.
Remember how we defined our model:
lm = Sequential([Dense(2, activation='softmax', input_shape(1000,))])
And the definition of a fully connected layer in the original...
Can you write Python code for it?


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
1.1  Intro
We need to find a way to convert the imagenet predictions to a probability of being a cat or a dog, since that is what the Kaggle copmetition requires us to submit. We could use the imagenet hierarchy to download a list of all the imagenet categories in each of the dog and cat groups, and could then solve our problem in various ways, such as:

Finding the largest probability that's either a cat or a dog, and using that label
Averaging the prbability of all the cat categories and comparing it to the average of all the dog categories.

But these approaches have some downsides:

They require manual coding for something that we should be able to learn from the data
They ignore information available in the predictions; for instance, if the models predict that there is a bone in th eimage, it's more likely to be a dog than a cat.

A very simple solution to both of these problems is to learn a linear model that is trained using the 1,000 predictions from the imagenet model for each image as input, and the dog/cat label as target.
Can you write Python code for it?

%matplotlib inline
from __future__ import division, print_function
import os, json
from glob import glob
import numpy as np
import scipy
from sklearn.preprocessing import OneHotEncoder
from sklearn.metrics import confusion_matrix
np.set_printoptions(precision=4,  linewidth=100)
from matplotlib import pyplot as plt
import utils; reload(utils)
from utils import plots, get_batches, plot_confusion_matrix, get_data

from numpy.random import random, permutation
from scipy import misc, ndimage
from scipy.ndimage.interpolation import zoom

import keras
from keras import backend as K
from keras.utils.data_utils import get_file
from keras.models import Sequential
from keras.layers import Input
from keras.layers.core import Flatten, Dense, Dropout, Lambda
from keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D
from keras.preprocessing import image


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
1.2 Linear models in keras
Let's forget the motivating example for a second a see how we can create a simple Linear model in Keras:
Each of the Dense() layers is just a linear model, followed by a simple activation function.
In a linear model each row is calculated as sum(row * weights), where weights need to be learnt from the data & will be the same for every row. Let's create some data that we know is linearly related:
Can you write Python code for it?

# we'll create a random matrix w/ 2 columns; & do a MatMul to get our
# y value using a vector [2, 3] & adding a constant of 1.
x = random((30, 2))
y = np.dot(x, [2., 3.]) + 1.

x[:5]

y[:5]


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
Above is everything Keras is doing behind the scenes.
So, if we pass multiple layers to Keras via Sequential(..), we can start to build & optimize Deep Neural Networks.
Before that, we can still use the single-layer LM to create a pretty decent entry to the dogs-vs-cats Kaggle competition.
1.3 Train Linear Model on Predictions
Forgetting finetuning -- how do we take the output of an ImageNet network and as simply as possible, create a a good entry to the cats-vs-dogs competition? -- Our current ImageNet network returns a thousand probabilities but we need just cat vs dog. We don't want to manually write code to roll of the hierarchy into cats/dogs.
So what we can do is learn a Linear Model that takes the output of the ImageNet model, all it's 1000 predictions, and uses that as input, and uses the dog/cat label as the target -- and that LM would solve our problem.
1.3.1 Training the model
We start with some basic config steps. We copy a small amount of our data into a 'sample' directory, with the exact same structure as our 'train' directory -- this is always a good idea in all Machine Learning, since we should do all of our initial testing using a dataset small enough that we never have to wait for it.
Can you write Python code for it?

# setup the directories
os.mkdir('data')
os.mkdir('data/dogscats')

path = "data/dogscats/"
model_path = path + 'models/'
# if the path to our models DNE, make it
if not os.path.exists(model_path): os.mkdir(model_path)
# NOTE: os.mkdir(..) only works for a single folder
#       Also will throw error if dir already exists


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
After we add our new final layer, we'll setup our batches to use preprocessed images (and we'll also shuffle the traiing batches to add more randomness when using multiple epochs):
Can you write Python code for it?

gen = image.ImageDataGenerator()
batches = gen.flow(trn_data, trn_labels, batch_size=batch_size, shuffle=True)
val_batches = gen.flow(val_data, val_labels, batch_size=batch_size, shuffle=False)


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
Getting the 1,000 categories for each image will take a long time & there's no reason to do it again & again. So after we do it the first time, let's save the resulting arrays.
Can you write Python code for it?

import bcolz
def save_array(fname, arr): c=bcolz.carray(arr, rootdir=fname, mode='w'); c.flush()
def load_array(fname): return bcolz.open(fname)[:]


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
It'll run a bit slowly since it has to calculate all previous layers in order to know what input to pass to the new final layer. We can save time by precalculating the output of the penultimate layer, like we did for the final layer earlier. Note for later work.
Can you write Python code for it?

# compile the new model
opt = RMSprop(lr=0.1)
model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])

# then fit it
fit_model(model, batches, val_batches, nb_epoch=2)


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
It's also time consuming to convert all the images in the 224x224 format VGG 16 expects. So get_data will also store a Numpy array of the results of that conversion.
Can you write Python code for it?

# ?? shows you the source code
??get_data

val_data = get_data(path + 'valid')
trn_data = get_data(path + 'train')

# so what the above does is createa a Numpy array with our full set of
# training images -- 352 imgs, ea. of which is 3 colors, and 224x224
trn_data.shape

save_array(model_path + 'train_data.bc', trn_data)
save_array(model_path + 'valid_data.bc', val_data)


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
Most Deep Learning is done w/ One-Hot Encoding: prediction = 1, all other classes = 0; & Keras expects labels in a very specific format. Example of One Hot Encoding:
Class: 1Ht Enc:
   0    100
   1    010
   2    001
   1    010
   0    100
1Ht Encoding is used because you can perform a MatMul since the num. weights == encoding length. In the above example W would be a vector of: w1, w2, w3
This lets you do Deep Learning very easily with categorical variables
Keras returns classes as a single column, so we convert to 1Ht.
Can you write Python code for it?

def onehot(x): return np.array(OneHotEncoder().fit_transform(x.reshape(-1, 1)).todense())

# So, next thing we want to do is grab our labels and One-Hot Encode them
val_classes = val_batches.classes
trn_classes = batches.classes
val_labels = onehot(val_classes)
trn_labels = onehot(trn_classes)

trn_classes.shape # Keras single col of all imgs

trn_labels.shape # One-Hot Encoded: 2 bit-width col <--> 2 classes

trn_classes[:4] # taking a look at 1st 4 classes

trn_labels[:4] # seeing the 1st 4 labels are 1Ht encoded


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
Not surprisingly, nearly all of these numbers are near zero.
Now we can define our linear model, just like we did earlier; now that we have our 1000 features for each image
Can you write Python code for it?

# 1000 inputs, since those're the saved features, and 2 outputs: dog & cat
lm = Sequential([Dense(2, activation='softmax', input_shape=(1000,))])
lm.compile(optimizer=RMSprop(lr=0.1), loss='categorical_crossentropy', metrics=['accuracy'])


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
Note how little actual code was needed to finetune the model. Because this is such an important and common operation, Keras is set up to make it as easy as possible. Not external helper functions were needed.
It's a good idea to save weights of all your models, so you can re-use them later. Be sure to note the git log number of your model when keeping a research journal of your results.
Can you write Python code for it?

model.save_weights(model_path + 'finetune1.h5')

# We can now use this as a good starting point for future Dogs v Cats models
model.load_weights(model_path + 'finetune1.h5')

model.evaluate(val_data, val_labels)


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
Now we add our final Cat vs Dog layer
Can you write Python code for it?


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
Now we have a model designed to classify Cats vs Dogs instead of the 1,000 ImageNet categories & THEN Cats vs Dogs. After this, everything is done the same as before. Compile the model & choose optimizer, fit the model (btw, whenever we work with batches in Keras, we'll be using model.function_generator(..) instead of model.function(..)
So let's do that and see what we get after 2 epochs of training:
We'll also define a function for fitting models to save time typing.
Can you write Python code for it?

# NOTE: now use batches.n instead of batches.N
def fit_model(model, batches, val_batches, nb_epoch=1):
    model.fit_generator(batches, samples_per_epoch=batches.n, nb_epoch=nb_epoch,
                        validation_data=val_batches, nb_val_samples=val_batches.n)


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
Our overall approach here will be:
1. Get the true labels for every image
2. Get the 1,000 ImageNet category predictions for every image
3. Feed these predictions as input to a simple linear model.
Let's start by grabbing training and validation batches.
(so that's a thousand floats for every image)
use an output of 2 as input to LM
output of 1 as target to our LM, create LM & build predictions
As usual, we start by creating our batches & validation vatches
Can you write Python code for it?

# Use batch size of 1 since we're just doing preprocessing on the CPU
val_batches = get_batches(path + 'valid', shuffle=False, batch_size=1)
batches = get_batches(path + 'train', shuffle=False, batch_size=1)


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
So it ran almost instantly because running 3 epochs on a single layer with 2000 is really quick for my little i5 MacBook :3
We got an accuracy of .92. Let's run another 3 epochs and see if this changes:
Can you write Python code for it?

lm.fit(trn_features, trn_labels, batch_size=batch_size, nb_epoch=3,
       validation_data = (val_features, val_labels))


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
The last layer is a Dense (FC/Linear) layer. It doesn't make sense to add another dense layer atop of a dense layer that's already tuned to classify the 1,000 ImageNet categories. We'll remove it, and use the previous Dense layer with it's 4096 activations to find Cats & Dogs.
We do this by calling model.pop() to pop off the last layer, and set all remaining layers to be fixed, so they aren't altered.
Can you write Python code for it?

model.pop()
for layer in model.layers: layer.trainable=False


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
To see what happened when we called vgg.finetune() earlier:
Basically what it does is a model.pop() and a model.add(Dense(..))
Can you write Python code for it?


I am working on the file "FAI_old/lesson2/lesson2_codealong.ipynb".
The first task is:
We'll process as many images at a time as we can. This is a case of T&E to find the max batch size that doesn't cause a memory error.
Can you write Python code for it?


I am working on the file "FAI_old/lesson3/L3HW_MNIST.ipynb".
The first task is:
5. Batch Normalization + Data Augmentation
See this thread for info on BatchNorm axis.
Can you write Python code for it?

def ConvModelBN():
    model = Sequential([
        Lambda(norm_input, input_shape=(1, 28, 28), output_shape=(1, 28, 28)),
        Convolution2D(32, 3, 3, activation='relu'),
        BatchNormalization(axis=1),
        Convolution2D(32, 3, 3, activation='relu'),
        MaxPooling2D(),
        BatchNormalization(axis=1),
        Convolution2D(64, 3, 3, activation='relu'),
        BatchNormalization(axis=1),
        Convolution2D(64, 3, 3, activation='relu'),
        MaxPooling2D(),
        Flatten(),
        BatchNormalization(),
        Dense(512, activation='relu'),
        BatchNormalization(),
        Dense(10, activation='softmax')
    ])
    model.compile(Adam(), loss='categorical_crossentropy', metrics=['accuracy'])
    return model

CNN_BNAug_model = ConvModelBN()
CNN_BNAug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                              validation_data=tst_batches, nb_val_samples=tst_batches.n)
print("Learning Rate, η = 0.1")
CNN_BNAug_model.optimizer=0.1
CNN_BNAug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=2, verbose=1,
                              validation_data=tst_batches, nb_val_samples=tst_batches.n)
print("Learning Rate, η = 0.01")
CNN_BNAug_model.optimizer=0.01
CNN_BNAug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=6, verbose=1,
                              validation_data=tst_batches, nb_val_samples=tst_batches.n)

# some more training at 0.1 and 0.01:
print("Learning Rate, η = 0.1")
CNN_BNAug_model.optimizer=0.1
CNN_BNAug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                              validation_data=tst_batches, nb_val_samples=tst_batches.n)
print("Learning Rate, η = 0.01")
CNN_BNAug_model.optimizer=0.01
CNN_BNAug_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=6, verbose=1,
                              validation_data=tst_batches, nb_val_samples=tst_batches.n)


I am working on the file "FAI_old/lesson3/L3HW_MNIST.ipynb".
The first task is:
7. Ensembling
Define a function to automatically train a model:
Can you write Python code for it?

# I'll set it to display progress at the start of each LR-change
def train_model():
    model = ConvModelBNDo()
    model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                                    validation_data=tst_batches, nb_val_samples=tst_batches.n)

    model.optimizer.lr=0.1
    model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                                    validation_data=tst_batches, nb_val_samples=tst_batches.n)
    model.fit_generator(trn_batches, trn_batches.n, nb_epoch=3, verbose=0,
                                    validation_data=tst_batches, nb_val_samples=tst_batches.n)

    model.optimizer.lr=0.01
    model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                                    validation_data=tst_batches, nb_val_samples=tst_batches.n)
    model.fit_generator(trn_batches, trn_batches.n, nb_epoch=11, verbose=0,
                                    validation_data=tst_batches, nb_val_samples=tst_batches.n)

    model.optimizer.lr=0.001
    model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                                    validation_data=tst_batches, nb_val_samples=tst_batches.n)
    model.fit_generator(trn_batches, trn_batches.n, nb_epoch=11, verbose=0,
                                    validation_data=tst_batches, nb_val_samples=tst_batches.n)
    return model

# Running a little test on the GPU now
testmodel = ConvModelBNDo()
testmodel.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                        validation_data=tst_batches, nb_val_samples=tst_batches.n)


I am working on the file "FAI_old/lesson3/L3HW_MNIST.ipynb".
The first task is:
Actually, looks like Vgg's ImageNet weights won't be needed.
Can you write Python code for it?

# from vgg16 import Vgg16
# vgg = Vgg16()


I am working on the file "FAI_old/lesson3/L3HW_MNIST.ipynb".
The first task is:
Boom. 0.99699.. ~ 99.7% accuracy. Same as achieved in lecture; took roughly 50 minutes to train. Unfortunately I didn't have the h5py module installed when I ran this, so the weight's can't be saved easily -- simple fix of rerunning after install.
Trying the above again, this time having h5py installed.
Can you write Python code for it?

# this'll take some time
models = [train_model() for m in xrange(6)]

from os import getcwd
import os
path = getcwd() + '/data/mnist/'
model_path = path + 'models/'

if not os.path.exists(path):
    os.mkdir('data')
    os.mkdir('data/mnist')
if not os.path.exists(model_path): os.mkdir(model_path)

for i,m in enumerate(models):
    m.save_weights(model_path + 'MNIST_CNN' + str(i) + '.pkl')

ensemble_preds = np.stack([m.predict(x_test, batch_size=256) for m in models])
avg_preds = ensemble_preds.mean(axis=0)

keras.metrics.categorical_accuracy(y_test, avg_preds).eval()


I am working on the file "FAI_old/lesson3/L3HW_MNIST.ipynb".
The first task is:
Create Data Batch Generator
ImageDataGenerator with no arguments will return a generator. Later, when data is augmented, it'll be told how to do so. I don't know what batch-size should be set to: in Lecture it was 64.
Can you write Python code for it?

gen = image.ImageDataGenerator()
trn_batches = gen.flow(x_train, y_train, batch_size=64)
tst_batches = gen.flow(x_test, y_test, batch_size=64)


I am working on the file "FAI_old/lesson3/L3HW_MNIST.ipynb".
The first task is:
Create an array of predictions from the models on the test-set. I'm using a batch size of 256 because that's what was done in lecture, and prediction is such an easier task that I think the large size just helps things go faster.
Can you write Python code for it?

ensemble_preds = np.stack([m.predict(x_test, batch_size=256) for m in models])


I am working on the file "FAI_old/lesson3/L3HW_MNIST.ipynb".
The first task is:
Finally, take the average of the predictions:
Can you write Python code for it?

avg_preds = ensemble_preds.mean(axis=0)

keras.metrics.categorical_accuracy(y_test, avg_preds).eval()


I am working on the file "FAI_old/lesson3/L3HW_MNIST.ipynb".
The first task is:
I want to import Vgg16 as well because I'll want it's low-level features
Can you write Python code for it?

# import os, sys
# sys.path.insert(1, os.path.join('../utils/'))


I am working on the file "FAI_old/lesson3/L3HW_MNIST.ipynb".
The first task is:
II. Load Data
Can you write Python code for it?

(x_train, y_train), (x_test, y_test) = mnist.load_data()


I am working on the file "FAI_old/lesson3/L3HW_MNIST.ipynb".
The first task is:
III. Preprocessing
Keras Convolutional layers expect color channels, so expand an empty dimension in the input data, to account for no colors.
Can you write Python code for it?

x_train = np.expand_dims(x_train, 1) # can also enter <axis=1> for <1>
x_test = np.expand_dims(x_test, 1)
x_train.shape


I am working on the file "FAI_old/lesson3/L3HW_MNIST.ipynb".
The first task is:
Since this notebook's models are all mimicking Vgg16, the input data should be preprocessed in the same way: in this case normalized by subtracting the mean and dividing by the standard deviation. It turns out this is a good idea generally.
Can you write Python code for it?

x_mean = x_train.mean().astype(np.float32)
x_stdv = x_train.std().astype(np.float32)
def norm_input(x): return (x - x_mean) / x_stdv


I am working on the file "FAI_old/lesson3/L3HW_MNIST.ipynb".
The first task is:
With an accuracy of 0.9823 and validation accuracy of 0.9664, the model's starting to overfit significantly and hit its limits, so it's time to go on to the next technique.
3. Basic 'VGG' style Convolutional Neural Network
I'm specifying an output shape equal to the input shape, to suppress the warnings keras was giving me; and it stated it was defaulting to that anyway. Or maybe I should've written output_shape=input_shape
Aha: yes it's as I thought. See this thread -- output_shape warnings were added to Keras, and neither vgg16.py (nor I until now) were specifying output_shape. It's fine.
The first time I ran this, I forgot to have 2 pairs of Conv layers. At the third λr=0.01 epoch I had acc/val of 0.9964, 0.9878
Also noticing: in lecture JH was using a GPU which I think was an NVidia Titan X. I'm using an Intel Core i5 CPU on a MacBook Pro. His epochs took on average 6 seconds, mine are taking 180~190. Convolutions are also the most computationally-intensive part of the NN being built here.
Interestingly, the model with 2 Conv-layer pairs is taking avg 160s. Best Acc/Val: 0.9968/0.9944
Final: 0.9975/0.9918 - massive overfitting
Can you write Python code for it?

def ConvModel():
    model = Sequential([
        Lambda(norm_input, input_shape=(1, 28, 28), output_shape=(1, 28, 28)),
        Convolution2D(32, 3, 3, activation='relu'),
        Convolution2D(32, 3, 3, activation='relu'),
        MaxPooling2D(),
        Convolution2D(64, 3, 3, activation='relu'),
        Convolution2D(64, 3, 3, activation='relu'),
        MaxPooling2D(),
        Flatten(),
        Dense(512, activation='relu'),
        Dense(10, activation='softmax')
    ])
    model.compile(Adam(), loss='categorical_crossentropy', metrics=['accuracy'])
    return model

CNN_model = ConvModel()
CNN_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1,
                       validation_data=tst_batches, nb_val_samples=tst_batches.n)

CNN_model.optimizer=0.1
CNN_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=1, verbose=1,
                       validation_data=tst_batches, nb_val_samples=tst_batches.n)

CNN_model.optimizer=0.01
CNN_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=4, verbose=1,
                       validation_data=tst_batches, nb_val_samples=tst_batches.n)

# Running again until validation accuracy stops increasing
CNN_model.fit_generator(trn_batches, trn_batches.n, nb_epoch=4, verbose=1,
                       validation_data=tst_batches, nb_val_samples=tst_batches.n)


I am working on the file "FAI_old/lesson3/L3HW_SF.ipynb".
The first task is:
1. Run this the First Time Only
Download the Data and get it into the right directories. Fortunately SF already organized the data, so it's just a matter of assigning path variables. NOTE: kaggle-cli needs to be set up beforhand. Also path vars must be assigned each time this notebook is run.
Can you write Python code for it?

HOME_DIR = os.getcwd()
DATA_DIR = HOME_DIR + '/data'
TRAIN_DIR = DATA_DIR + '/train'
VAL_DIR = DATA_DIR + '/valid'
TEST_DIR = DATA_DIR + '/test'

# create the validation directories
# os.mkdir(VAL_PATH)
# for i in xrange(10):
#     os.mkdir(VAL_PATH + '/c' + str(i))

# # another way to do this:
# %mkdir $VAL_PATH
# for i in xrange(10):
#     %mkdir $VAL_PATH/c"$i"


I am working on the file "FAI_old/lesson3/L3HW_SF.ipynb".
The first task is:
2. Run this if you don't have an Accurate Validation Set
Grab a random permutation from the training data for validation. Do this until validation accuracy matches test accuracy. Also see: http://stackoverflow.com/questions/2632205/how-to-count-the-number-of-files-in-a-directory-using-python
Can you write Python code for it?

# %cd $TRAIN_PATH

# VAL_PORTION = 0.2
# for i in xrange(10):
#     %cd c"$i"
#     g = glob('*.jpg')
#     number = len(g)
#     shuff = np.random.permutation(g)
#     for n in xrange(int(number * VAL_PORTION)):
#         os.rename(shuff[n], VAL_DIR + '/c' + str(i) + '/' + shuff[n])
#     % cd ..

def reset_valid(verbose=1):
    """Moves all images in validation set back to
    their respective classes in the training set."""
    counter = 0
    %cd $valid_path
    for i in xrange(10):
        %cd c"$i"
        g = glob('*.jpg')
        for n in xrange(len(g)):
            os.rename(g[n], TRAIN_DIR + '/c' + str(i) + '/' + g[n])
            counter += 1
        % cd ..
    if verbose: print("Moved {} files.".format(counter))
#         %mv $VALID_DIR/c"$i"/$*.jpg $TRAIN_DIR/c"$i"/$*.jpg

# modified from: http://forums.fast.ai/t/statefarm-kaggle-comp/183/20
def set_valid(number=1, verbose=1):
    """Moves <number> of subjects from training to validation
    directories. Verbosity: 0: Silent; 1: print no. files moved;
    2: print each move operation"""
    counter = 0
    if number < 0: number = 0
    for n in xrange(number):
        # read CSV file into Pandas DataFrame
        dil = pd.read_csv(data_path + 'driver_imgs_list.csv')
        # group frame by subject in image
        grouped_subjects = dil.groupby('subject')
        # pick <number> subjects at random
        subject = grouped_subjects.groups.keys()[np.random.randint(0, high=len(grouped_subjects.groups))] # <-- groups?
        # get the group assoc w/ subject
        group = grouped_subjects.get_group(subject)
        # loop over gropu & move imgs to validation dir
        for (subject, clssnm, img) in group.values:
            source = '{}train/{}/{}'.format(data_path, clssnm, img)
            target = source.replace('train', 'valid')
            if verbose > 1: print('mv {} {}'.format(source, target))
            os.rename(source, target)
            counter += 1
    if verbose: print "Files moved: {}".format(counter)

# dil =...


I am working on the file "FAI_old/lesson3/L3HW_SF.ipynb".
The first task is:
So, I don't know if, when building an FC model taking input from a Vgg16bn Conv model, I'll have to specifiy input shape to the first MaxPooling2D layer to be all but the first part of the output_shape (tensor?) from the ConvLayer. We'll see. Does it matter because it's None? or will it be inferred automatically? Or does it need to take in a 3-dim input -- but then why would it be taking in a 4-dim one in the full model's case? Hmm.
Can you write Python code for it?

predictions = VGG.test(test_path, batch_size=32)

save_array(results_path + 'raw_predictions01.bc', predictions[1])

len(predictions[1])

predictions[1].shape

preds = predictions[1]
# clipping & renorm to score better on logloss metric
preds = utils.do_clip(preds, mx=0.95)

filenames = tst_batches.filenames
# ids = np.array([str(f[8:f.find('.')]) for f in filenames])
ids = np.array([str(f[8:]) for f in filenames])

print(ids.shape)
print(preds.shape)

import pandas as pd

# submissions = np.stack([ids, preds], axis=1)

# couldn't get the older method of using np.stack to work, so trying pandas
classes = sorted(trn_batches.class_indices, key=trn_batches.class_indices.get)
submission = pd.DataFrame(preds, columns=classes)
# submission.insert(0, 'img', [f[12:] for f in filenames])
submission.insert(0, 'img', [f[8:] for f in filenames])
submission.head()
submission.to_csv(results_path + 'submission.csv', index=False, compression=None)

# ??pd.DataFrame.to_csv

from IPython.display import FileLink
FileLink(results_path + 'submission.csv')

??submission.insert


I am working on the file "FAI_old/lesson3/L3HW_SF.ipynb".
The first task is:
enumerate is zero-indexed. The first Conv layer is the 3rd, so that'll give an index of 2. Therefore, when defining a model consisting of all Conv layers, remember to include 1-past the index. Also, Python index notation [x:y] translates to [start-before-here:until-just-before-here] equivalently, [start-with-this:until-this]
So that's why, if your model is VGG.model, entering VGG.model.layers[last_conv_idx] will give you the last convolutional layer, but entering: VGG.model.layers[:last_conv_idx] will give you all layers up to the last Conv layer, but not including it.
Can you write Python code for it?

from keras.layers.convolutional import Convolution2D
last_conv_idx = [index for index, layer in enumerate(VGG.model.layers) if type(layer) is Convolution2D][-1]
conv_layers = VGG.model.layers[:last_conv_idx + 1]
# NOTE: enumerate is zero-indexed.

first_conv_idx = [idx for idx, layer in enumerate(VGG.model.layers) if type(layer) is Convolution2D][0]
first_conv_idx

# print(last_conv_idx)
VGG.model.layers

conv_layers

# from keras.layers.pooling import MaxPooling2D
# ??MaxPooling2D
# ??Convolution2D
# conv_layers[-1].output_shape
# VGG.model.layers[last_conv_idx + 1]

# getting some insight into Conv output_shape and MaxPool2D input_shapes
# Theano ordering: (samples, channels, rows, cols) <- Conv-out / MxP-in
#
print(VGG.model.layers[last_conv_idx].output_shape)
print(VGG.model.layers[last_conv_idx + 1].input_shape)


I am working on the file "FAI_old/lesson3/L3HW_SF2.ipynb".
The first task is:
Create Fully-Connected Net
Can you write Python code for it?

FC_model = Sequential(create_FCbn_layers(p=0.3))
FC_model.compile(Adam(), loss='categorical_crossentropy', metrics=['accuracy'])

(val_classes, trn_classes, val_labels, trn_labels,
    validation_filenames, training_filenames, testing_filenames) = utils.get_classes(data_path)

save_array(results_path + 'conv_features.bc', conv_features)
save_array(results_path + 'conv_val_feat.bc', conv_val_feat)

??utils.get_batches


I am working on the file "FAI_old/lesson3/L3HW_SF2.ipynb".
The first task is:
Creating validation directories
Can you write Python code for it?

# os.mkdir(VAL_DIR)
# for i in xrange(10):
#     os.mkdir(VAL_DIR + '/c' + str(i))

# # # another way to do this:
# # %mkdir $VAL_DIR
# # for i in xrange(10):
# #     %mkdir $VAL_DIR/c"$i"


I am working on the file "FAI_old/lesson3/L3HW_SF2.ipynb".
The first task is:
I just learned that the utils.vgg_ft_bn(num) function called in the JNB I was referencing is just initializes a model, and finetunes it in the standard way: pop off last layer, set all remaining layers to un-trainable, add a single softmax output FC layer, and compile. So as an experiment... how bad of an idea would, say, doing that & training the output layer then training all layers including convolutionals, be?
That kind of messes with the cleaned-up flow of this notebook, but that's fine.
Answer: it's pretty bad. Even with a batch size of 1 the GPU still runs out of memory, if barely. Perhaps using non-augmented data would spare a bit of memory.
Separate Conv layers & create new ConvNet (w/ vgg weights)
Since I'm generating randomly-augmented data each batch, I can't precompute the Conv features. I'd have to experiment and see the speed vs. accuracy tradeoff.
Can you write Python code for it?

last_conv_idx = [index for index, layer in enumerate(VGGbn.model.layers) \
                                            if type(layer) is Convolution2D][-1]
Conv_layers = VGGbn.model.layers[:last_conv_idx + 1]
Conv_model = Sequential(Conv_layers)


I am working on the file "FAI_old/lesson3/L3HW_SF2.ipynb".
The first task is:
Load VGG16BN model & its weights
Can you write Python code for it?

VGGbn = Vgg16BN()

VGGbn.model.compile(Adam(), loss='categorical_crossentropy', metrics=['accuracy'])


I am working on the file "FAI_old/lesson3/L3HW_SF2.ipynb".
The first task is:
Once the above is working:
Ensemble
Work-flow:
To save GPU memory (no easy way to free it yet in JNB) init a single Convolutional and Fully-Connected Net model. The convolutional layer weights will not be trained, only used to produce features, so only the FC-model will have it's weights saved/reinitialized (if I can re-init without increasing memory load, I'll do that).
For each iteration of the ensemble, a randomly-augmented set of data will be passed through the ConvNet, and those features will be used to train a fresh FC Net. After the training phase, The Conv & FC nets will be used to create a set of predictions which will be stored in an array and returned. That array is then averaged into a single list of predictions, clipped & renormalized, then saved to be uploaded as the final submission.
Can you write Python code for it?

def train_FCNet():
    FC_model.fit(conv_features, trn_batches.labels, batch_size=batch_size,
                 nb_epoch=1, validation_data=(conv_val_feat, val_batches.labels))
    FC_model.optimizer.lr=1e-1
    FC_model.fit(conv_features, trn_batches.labels, batch_size=batch_size,
                 nb_epoch=1, validation_data=(conv_val_feat, val_batches.labels))
    FC_model.optimizer.lr=1e-2
    FC_model.fit(conv_features, trn_batches.labels, batch_size=batch_size,
                 nb_epoch=2, validation_data=(conv_val_feat, val_batches.labels))
    FC_model.optimizer.lr=1e-4
    FC_model.fit(conv_features, trn_batches.labels, batch_size=batch_size,
                 nb_epoch=4, validation_data=(conv_val_feat, val_batches.labels))
    FC_model.optimizer.lr=1e-6
    FC_model.fit(conv_features, trn_batches.labels, batch_size=batch_size,
                 nb_epoch=8, validation_data=(conv_val_feat, val_batches.labels))

def Ensemble(num_models=1):
    VGGbn = Vgg16BN()
    last_conv_idx = [index, for index, layer in enumerate(VGGbn.model.layers) \
                                            if type(layer) is Convolution2D][-1]
    Conv_layers = VGGbn.model.layers[:last_conv_idx + 1]
    Conv_model = Sequential(Conv_layers)

    gen = image.ImageDataGenerator(rotation_range=10, width_shift_range=0.05,
                                       height_shift_range=0.05, zoom_range=0.1,
                                           shear_range=0.1, channel_shift_range=10)
    trn_batches = gen.flow_from_directory(train_path, target_size=target_size,
                    batch_size=batch_size, shuffle=False, class_mode='categorical')
    val_batches = gen.flow_from_directory(valid_path, target_size=target_size,
                    batch_size=batch_size, shuffle=False, class_mode='categorical')

    gen_t = image.ImageDataGeneratro()
    tst_batches = gen.flow_from_directory(test_path, batch_size=batch_size,
                                          shuffle=False, class_mode=None)

    pred_array = []
   ...


I am working on the file "FAI_old/lesson3/L3HW_SF2.ipynb".
The first task is:
Run Conv Model on trn/val batches to create features as inputs to FCNet
Can you write Python code for it?

conv_features = Conv_model.predict_generator(trn_batches, trn_batches.nb_sample)
conv_val_feat = Conv_model.predict_generator(val_batches, val_batches.nb_sample)

conv_features = load_array(results_path + 'conv_features.bc')
conv_val_feat = load_array(results_path + 'conv_val_feat.bc')


I am working on the file "FAI_old/lesson3/L3HW_SF2.ipynb".
The first task is:
Run test batches through ConvNet, run ConvNet test features through FCNet
Can you write Python code for it?

conv_tst_feat = Conv_model.predict_generator(tst_batches, tst_batches.nb_sample)
preds = FC_model.predict(conv_tst_feat, batch_size=batch_size*2)


I am working on the file "FAI_old/lesson3/L3HW_SF2.ipynb".
The first task is:
Save results
Can you write Python code for it?

filenames = tst_batches.filenames
classes = sorted(trn_batches.class_indices, key=trn_batches.class_indices.get)
submission = pd.DataFrame(preds, columns=classes)
submission.insert(0, 'img', [f[8:] for f in filenames])
submission.head()
submission.to_csv(results_path + 'submission.csv', index=False, compression=None)

from IPython.display import FileLink
FileLink(results_path + 'submission.csv')


I am working on the file "FAI_old/lesson3/L3HW_SF2.ipynb".
The first task is:
Setting/resetting validation set
Can you write Python code for it?


I am working on the file "FAI_old/lesson3/L3HW_SF2.ipynb".
The first task is:
Train FCNet on ConvNet features
Can you write Python code for it?

# each epoch on GTX870M ~18 minutes
# wow, no, ~18 min for CNN. FCNet only is ~ 1 minute!
# FC_model.fit(conv_features, trn_labels, batch_size=batch_size,
#              nb_epoch=1, validation_data=(conv_val_feat, val_labels))
FC_model.optimizer.lr=1e-1
FC_model.fit(conv_features, trn_labels, batch_size=batch_size,
             nb_epoch=1, validation_data=(conv_val_feat, val_labels))
FC_model.optimizer.lr=1e-2
FC_model.fit(conv_features, trn_labels, batch_size=batch_size,
             nb_epoch=1, validation_data=(conv_val_feat, val_labels))

??utils.to_categorical

from utils import to_categorical as onehot
FC_model.optimizer.lr=1e-4
FC_model.fit(conv_features, onehot(trn_batches.classes), batch_size=batch_size,
             nb_epoch=4, validation_data=(conv_val_feat, onehot(val_batches.classes)))
FC_model.optimizer.lr=1e-4
FC_model.fit(conv_features, onehot(trn_batches.classes), batch_size=batch_size,
             nb_epoch=8, validation_data=(conv_val_feat, onehot(val_batches.classes)))
FC_model.optimizer.lr=1e-6
FC_model.fit(conv_features, onehot(trn_batches.classes), batch_size=batch_size,
             nb_epoch=12, validation_data=(conv_val_feat, onehot(val_batches.classes)))


I am working on the file "FAI_old/lesson3/L3HW_SF2.ipynb".
The first task is:
Train/valid batch generators
Training batches have to be set to not be shuffled. Since the full-model is in two stages, classes & labels will be supplied to the FCNet via the batches-generator; if these are shuffled they won't match up with the output features from the ConvNet. I think.
Can you write Python code for it?

gen = image.ImageDataGenerator(rotation_range=10, width_shift_range=0.05,
                                height_shift_range=0.05, zoom_range=0.1,
                                   shear_range=0.1, channel_shift_range=10)
# does it matter that I don't set dim_ordering='tf'?
trn_batches = gen.flow_from_directory(train_path, target_size=target_size,
                batch_size=batch_size, shuffle=False, class_mode='categorical')
val_batches = gen.flow_from_directory(valid_path, target_size=target_size,
                batch_size=batch_size, shuffle=False, class_mode='categorical')

??utils.get_classes


I am working on the file "FAI_old/lesson3/L3HW_SF2.ipynb".
The first task is:
Utility functions
Can you write Python code for it?

def save_array(fname, arr): c=bcolz.carray(arr, rootdir=fname, mode='w'); c.flush()
def load_array(fname): return bcolz.open(fname)[:]

def reset_valid(verbose=1):
    """Moves all images in validation set back to
    their respective classes in the training set."""
    counter = 0
    %cd $valid_path
    for i in xrange(10):
        %cd c"$i"
        g = glob('*.jpg')
        for n in xrange(len(g)):
            os.rename(g[n], TRAIN_DIR + '/c' + str(i) + '/' + g[n])
            counter += 1
        %cd ..
    if verbose: print("Moved {} files".format(counter))

# modified from: http://forums.fast.ai/t/statefarm-kaggle-comp/183/20
def set_valid(number=1, verbose=1):
    """Moves <number> subjects from training to validation
    directories. Verbosity 0: Silent; 1: print no. files moved;
    2: print each move operation. Default=1"""
    if number < 0: number = 0
    # repeat for <number> subjects
    for n in xrange(number):
        # read CSV file into Pandas DataFrame
        dil = pd.read_csv(data_path + 'driver_imgs_list.csv')
        # grouped frame by subject in image
        grouped_subjects = dil.groupby('subject')
        # pick subject at random
        subject = grouped_subjects.groups.keys()[np.random.randint(0, \
                                         high=len(grouped_subjects.groups)-1)]
        # get group assoc w/ subject
        group = grouped_subjects.get_group(subject)
        # loop over group & move imgs to validation dir
        counter = 0
        for (subject, clssnm, img) in group.values:
            source = '{}train/{}/{}'.format(data_path, clssnm, img)
            target = source.replace('train', 'valid')
            if verbose > 1: print('mv {} {}'.format(source, target))
            os.rename(source, target)
            counter += 1
        if verbose: print("Files moved: {}".format(counter))
# function to build FCNet w/ BatchNormalization & Dropout
def create_FCbn_layers(p=0):
    return [
            MaxPooling2D(input_shape=Conv_model.layers[-1].output_shape[1:]),
#     ...


I am working on the file "FAI_old/lesson4_codealong.ipynb".
The first task is:
Because it's hard to interpret 50 embeddings, we use PCA (Principal Component Analysis) to simplify them down to just 3 vectors.
Can you write Python code for it?

from sklearn.decomposition import PCA
pca = PCA(n_components=3)
movie_pca = pca.fit(movie_emb.T).components_

fac0 = movie_pca[0]
movie_comp = [(f, movie_names[movies[i]]) for f,i in zip(fac0, topMovies)]


I am working on the file "FAI_old/lesson4_codealong.ipynb".
The first task is:
Create subset for Excel
We create a crosstab of the most popular movies and most movie-addicted users which we'll copy into Excel for creating a simple example. This isn't necessary for any of the modeling below however.
Can you write Python code for it?

g=ratings.groupby('userId')['rating'].count()
topUsers=g.sort_values(ascending=False)[:15]

g=ratings.groupby('movieId')['rating'].count()
topMovies=g.sort_values(ascending=False)[:15]

top_r = ratings.join(topUsers, rsuffix='_r', how='inner', on='userId')

top_r = top_r.join(topMovies, rsuffix='_r', how='inner', on='movieId')

pd.crosstab(top_r.userId, top_r.movieId, top_r.rating, aggfunc=np.sum)


I am working on the file "FAI_old/lesson4_codealong.ipynb".
The first task is:
Dot Product
The most basic model is a dot product of a movie embedding and a user embedding. Let's see how well that works:
Can you write Python code for it?

user_in = Input(shape=(1,), dtype='int64', name='user_in')
u = Embedding(n_users, n_factors, input_length=1, W_regularizer=l2(1e-4))(user_in)
movie_in = Input(shape=(1,), dtype='int64', name='movie_in')
m = Embedding(n_movies, n_factors, input_length=1, W_regularizer=l2(1e-4))(movie_in)

x = merge([u, m], mode='dot')
x = Flatten()(x)
model = Model([user_in, movie_in], x)
model.compile(Adam(0.001), loss='mse')

model.fit([trn.userId, trn.movieId], trn.rating, batch_size=64, nb_epoch=1,
          validation_data=([val.userId, val.movieId], val.rating))

model.optimizer.lr=0.01

model.fit([trn.userId, trn.movieId], trn.rating, batch_size=64, nb_epoch=3,
          validation_data=([val.userId, val.movieId], val.rating))

model.optimizer.lr=0.001
model.fit([trn.userId, trn.movieId], trn.rating, batch_size=64, nb_epoch=6,
          validation_data=([val.userId, val.movieId], val.rating))


I am working on the file "FAI_old/lesson4_codealong.ipynb".
The first task is:
First, we'll look at the movie bias term. We create a 'model' - which in Keras is simply a way of associating one or more inputs with one or more outputs, using the functional API. Here, our input is the movie id (a single id), and the output is the movie bias (a single float).
Can you write Python code for it?

get_movie_bias = Model(movie_in, mb)
movie_bias = get_movie_bias.predict(topMovies)
movie_ratings = [(b[0], movie_names[movies[i]]) for i, b in zip(topMovies, movie_bias)]


I am working on the file "FAI_old/lesson4_codealong.ipynb".
The first task is:
Here's the 1st component. It seems to be 'critically acclaimed' or 'classic'.
Can you write Python code for it?

sorted(movie_comp, key=itemgetter(0), reverse=True)[:10]

sorted(movie_comp, key=itemgetter(0))[:10]

fac1 = movie_pca[1]
movie_comp = [(f, movie_names[movies[i]]) for f,i in zip(fac1, topMovies)]


I am working on the file "FAI_old/lesson4_codealong.ipynb".
The first task is:
Just for display purposes, let's read in the movie names too:
Can you write Python code for it?

movie_names = pd.read_csv(path+'movies.csv').set_index('movieId')['title'].to_dict()

users = ratings.userId.unique()
movies = ratings.movieId.unique()

userid2idx = {o:i for i,o in enumerate(users)}
movieid2idx = {o:i for i,o in enumerate(movies)}


I am working on the file "FAI_old/lesson4_codealong.ipynb".
The first task is:
Now we can look at the top and bottom rated movies. These ratings are correected for different levels of reviewer sentiment, as well as different types of movies that different reviewers watch.
Can you write Python code for it?

sorted(movie_ratings, key=itemgetter(0))[:15]

sorted(movie_ratings, key=itemgetter(0), reverse=True)[:15]


I am working on the file "FAI_old/lesson4_codealong.ipynb".
The first task is:
Randomly split into training and validation.
Can you write Python code for it?

msk = np.random.rand(len(ratings)) < 0.8
trn = ratings[msk]
val = ratings[~msk]


I am working on the file "FAI_old/lesson4_codealong.ipynb".
The first task is:
The 2nd is 'hollywood blockbuster'.
Can you write Python code for it?

sorted(movie_comp, key=itemgetter(0), reverse=True)[:10]

sorted(movie_comp, key=itemgetter(0))[:10]

fac2 = movie_pca[2]
movie_comp = [(f, movie_names[movies[i]]) for f,i in zip(fac2, topMovies)]


I am working on the file "FAI_old/lesson4_codealong.ipynb".
The first task is:
The 3rd is 'violent vs happy'.
Can you write Python code for it?

sorted(movie_comp, key=itemgetter(0), reverse=True)[:10]

sorted(movie_comp, key=itemgetter(0))[:10]


I am working on the file "FAI_old/lesson4_codealong.ipynb".
The first task is:
The best benchmarks are a bit over 0.9, so this model doesn't seem to be working that well...
Bias
The problem is likely to be that we don't have bias terms - that is, a single bias for each user and each movie representing how positive or negative each user is, and how good each movie is. We can add that easily by simply creating an embedding with one output for each movie and each user, and adding it to our output.
Can you write Python code for it?

def embedding_input(name, n_in, n_out, reg):
    inp = Input(shape=(1,), dtype='int64', name=name)
    return inp, Embedding(n_in, n_out, input_length=1, W_regularizer=l2(reg))(inp)

user_in, u = embedding_input('user_in', n_users, n_factors, 1e-4)
movie_in, m = embedding_input('movie_in', n_movies, n_factors, 1e-4)

def create_bias(inp, n_in):
    x = Embedding(n_in, 1, input_length=1)(inp)
    return Flatten()(x)

ub = create_bias(user_in, n_users)
mb = create_bias(movie_in, n_movies)

x = merge([u, m], mode='dot')
x = Flatten()(x)
x = merge([x, ub], mode='sum')
x = merge([x, mb], mode='sum')
model = Model([user_in, movie_in], x)
model.compile(Adam(0.001), loss='mse')

model.fit([trn.userId, trn.movieId], trn.rating, batch_size=64, nb_epoch=1,
          validation_data=([val.userId, val.movieId], val.rating))

model.optimizer.lr = 0.01
model.fit([trn.userId, trn.movieId], trn.rating, batch_size=64, nb_epoch=6, verbose=0,
          validation_data=([val.userId, val.movieId], val.rating))

model.optimizer.lr = 0.001
model.fit([trn.userId, trn.movieId], trn.rating, batch_size=64, nb_epoch=10, verbose=0,
          validation_data=([val.userId, val.movieId], val.rating))

model.fit([trn.userId, trn.movieId], trn.rating, batch_size=64, nb_epoch=5,
          validation_data=([val.userId, val.movieId], val.rating))


I am working on the file "FAI_old/lesson4_codealong.ipynb".
The first task is:
We can draw a picture to see how various movies appear on the map of these components. This picture shows the 1st and 3rd components.
Can you write Python code for it?

import sys
stdout, stderr = sys.stdout, sys.stderr # save notebook stdout and stderr
reload(sys)
sys.setdefaultencoding('utf-8')
sys.stdout, sys.stderr = stdout, stderr # restore notebook stdout and stderr

start=50; end=100
X = fac0[start:end]
Y = fac2[start:end]
plt.figure(figsize=(15,15))
plt.scatter(X, Y)
for i, x, y in zip(topMovies[start:end], X, Y):
    plt.text(x,y,movie_names[movies[i]], color=np.random.rand(3)*0.7, fontsize=14)
plt.show()


I am working on the file "FAI_old/lesson4_codealong.ipynb".
The first task is:
We update the movie and user ids so that they are contiguous integers, which we want when using embeddings.
Can you write Python code for it?

ratings.movieId = ratings.movieId.apply(lambda x: movieid2idx[x])
ratings.userId = ratings.userId.apply(lambda x: userid2idx[x])

user_min, user_max, movie_min, movie_max = (ratings.userId.min(),
    ratings.userId.max(), ratings.movieId.min(), ratings.movieId.max())
user_min, user_max, movie_min, movie_max

n_users = ratings.userId.nunique()
n_movies = ratings.movieId.nunique()
n_users, n_movies


I am working on the file "FAI_old/lesson4_codealong.ipynb".
The first task is:
e can use the model to generate predictions by passing a pair of ints - a user id and a movie id. For instance, this predicts that user #3 would really enjoy movie #6.
Can you write Python code for it?


I am working on the file "FAI_old/lesson5_CodeAlong.ipynb".
The first task is:
Create simple models
Single hidden layer NN
This simplest model that tends to give reasonable results is a single hidden layer net. So let's try that. Note that we can't expect to get any useful results by feeding word ids directly into a neural net - so instead we use an embedding to replace them with a vector of 32 (initially random) floats for each word in the vocab.
Can you write Python code for it?

model = Sequential([
    Embedding(vocab_size, 32, input_length=seq_len),
    Flatten(),
    Dense(100, activation='relu'),
    Dropout(0.7),
    Dense(1, activation='sigmoid')])

model.compile(loss='binary_crossentropy', optimizer=Adam(), metrics=['accuracy'])
model.summary()

# model.fit(trn, labels_train, validation_data=(test, labels_test), nb_epoch=2, batch_size=64)

# redoing on Linux
model.fit(trn, labels_train, validation_data=(test, labels_test), nb_epoch=2, batch_size=64)


I am working on the file "FAI_old/lesson5_CodeAlong.ipynb".
The first task is:
Here's the whole review, mapped from ids to words.
Can you write Python code for it?

' '.join([idx2word[o] for o in x_train[0]])


I am working on the file "FAI_old/lesson5_CodeAlong.ipynb".
The first task is:
Look at distribution of lengths of sentences
Can you write Python code for it?

lens = np.array(map(len, trn))
(lens.max(), lens.min(), lens.mean())


I am working on the file "FAI_old/lesson5_CodeAlong.ipynb".
The first task is:
Pad (with zero) or truncate each sentence to make consistent length.
Can you write Python code for it?

seq_len = 500

# keras.preprocessing.sequence
trn = sequence.pad_sequences(trn, maxlen=seq_len, value=0)
test = sequence.pad_sequences(test, maxlen=seq_len, value=0)


I am working on the file "FAI_old/lesson5_CodeAlong.ipynb".
The first task is:
Setup data
We're going to look at the IMDB dataset, which contains movie reviews from IMDB, along with their sentiment. Keras comes with some helpers for this dataset.
Can you write Python code for it?

from keras.datasets import imdb
idx = imdb.get_word_index()


I am working on the file "FAI_old/lesson5_CodeAlong.ipynb".
The first task is:
That's well past the Stanford paper's accuracy - another win for CNNs!
Heh, the above take a lot longer than 4s on my Mac
Can you write Python code for it?

conv1.save_weights(model_path + 'conv1.h5')
# conv1.load_weights(model_path + 'conv1.h5')


I am working on the file "FAI_old/lesson5_CodeAlong.ipynb".
The first task is:
The first word of the first review is 23022. Let's see what that is.
Can you write Python code for it?


I am working on the file "FAI_old/lesson5_CodeAlong.ipynb".
The first task is:
The labels are 1 for positive, 0 for negative
Can you write Python code for it?


I am working on the file "FAI_old/lesson5_CodeAlong.ipynb".
The first task is:
This is the word list:
Can you write Python code for it?

idx_arr = sorted(idx, key=idx.get)
idx_arr[:10]


I am working on the file "FAI_old/lesson5_CodeAlong.ipynb".
The first task is:
We download the reviews using code copied from keras.datasets:
Can you write Python code for it?

# getting the dataset directly bc keras's versn makes some changes
path = get_file('imdb_full.pkl',
                origin='https://s3.amazonaws.com/text-datasets/imdb_full.pkl',
                md5_hash='d091312047c43cf9e4e38fef92437263')
f = open(path, 'rb')
(x_train, labels_train), (x_test, labels_test) = pickle.load(f)

# apparently cpickle can be x1000 faster than pickle? hmm
len(x_train)


I am working on the file "FAI_old/lesson5_CodeAlong.ipynb".
The first task is:
We pass out embedding matrix to the Embedding constructor, and set it to non-trainable.
Can you write Python code for it?

model = Sequential([
    Embedding(vocab_size, 50, input_length=seq_len, dropout=0.2,
              weights=[emb], trainable=False),
    Dropout(0.25),
    Convolution1D(64, 5, border_mode='same', activation='relu'),
    Dropout(0.25),
    MaxPooling1D(),
    Flatten(),
    Dense(100, activation='relu'),
    Dropout(0.7),
    Dense(1, activation='sigmoid')])
# this is copy-pasted of the previous code, with the addition of the
# weights being the pre-trained embeddings.
# We figure the weights are pretty good, so we'll initially set
# trainable to False. Will finetune due to some words missing or etc..

model.compile(loss='binary_crossentropy', optimizer=Adam(), metrics=['accuracy'])

model.fit(trn, labels_train, validation_data=(test, labels_test), nb_epoch=2, batch_size=64)

# running on GPU
model.fit(trn, labels_train, validation_data=(test, labels_test), nb_epoch=2, batch_size=64)


I am working on the file "FAI_old/lesson5_CodeAlong.ipynb".
The first task is:
We use the functional API to create multiple ocnv layers of different sizes, and then concatenate them.
Can you write Python code for it?

graph_in = Input((vocab_size, 50))
convs = [ ]
for fsz in xrange(3, 6):
    x = Convolution1D(64, fsz, border_mode='same', activation='relu')(graph_in)
    x = MaxPooling1D()(x)
    x = Flatten()(x)
    convs.append(x)
out = Merge(mode='concat')(convs)
graph = Model(graph_in, out)

emb = create_emb()


I am working on the file "FAI_old/statefarm-sample-codealong.ipynb".
The first task is:
(Not so in this case, only kind of, but it was indeed predicted 1 or 6 back on the Mac)
Our hypothesis was correct. It's nearly always predicting class 1 or 6, with very high confidence. So let's try a lower learning rate:
Can you write Python code for it?

# here's a way to take a look at the learning rate
import keras.backend as K
LR = K.eval(model.optimizer.lr)
print(LR)

model = Sequential([
            BatchNormalization(axis=1, input_shape=(3,224,224)),
            Flatten(),
            Dense(10, activation='softmax')
        ])
model.compile(Adam(lr=1e-5), loss='categorical_crossentropy', metrics=['accuracy'])
model.fit_generator(batches, batches.nb_sample, nb_epoch=2, validation_data=val_batches,
                    nb_val_samples=val_batches.nb_sample)


I am working on the file "FAI_old/statefarm-sample-codealong.ipynb".
The first task is:
2/3: Making sure we're in the right dir, & moving stuff
Can you write Python code for it?

%pwd

# %cd ~/Deshar/Kaukasos/FAI
%cd ~/Kaukasos/FAI

%cd data/statefarm/
reset_valid()
%cd ..
set_valid(number=3)


I am working on the file "FAI_old/statefarm-sample-codealong.ipynb".
The first task is:
As you can see below, this training is going nowhere...
Can you write Python code for it?

model.compile(Adam(), loss='categorical_crossentropy', metrics=['accuracy'])
model.fit_generator(batches, batches.nb_sample, nb_epoch=2, validation_data=val_batches,
                   nb_val_samples=val_batches.nb_sample)


I am working on the file "FAI_old/statefarm-sample-codealong.ipynb".
The first task is:
Basic Models
Linear Model
First, we try the simplest model and use default parameters. Note the trick of making the first layer a batchnorm layer - that way we don't have to worry about normalizing the input ourselves.
Can you write Python code for it?

model = Sequential([
            BatchNormalization(axis=1, input_shape=(3, 224, 224)),
            Flatten(),
            Dense(10, activation='softmax')
        ])


I am working on the file "FAI_old/statefarm-sample-codealong.ipynb".
The first task is:
Create Batches
Can you write Python code for it?

batches = get_batches(path + 'train', batch_size=batch_size)
val_batches = get_batches(path + 'valid', batch_size=batch_size*2, shuffle=False)

%pwd
os.mkdir(path + 'test')

(val_classes, trn_classes, val_labels, trn_labels, val_filenames, filenames,
    test_filename) = get_classes(path)


I am working on the file "FAI_old/statefarm-sample-codealong.ipynb".
The first task is:
Create Sample
The following assumes you've already created your validation set - remember that the training and validation set should contain different drivers, as mentioned on the Kaggle competition page.
Can you write Python code for it?

%cd data/statefarm
%cd train

%mkdir ../sample
%mkdir ../sample/train
%mkdir ../sample/valid

for d in glob('c?'):
    os.mkdir('../sample/train/' + d)
    os.mkdir('../sample/valid/' + d)

from shutil import copyfile

g = glob('c?/*.jpg')
shuf = np.random.permutation(g)
for i in range(1500): copyfile(shuf[i], '../sample/train/' + shuf[i])

# # removing copied sample training images
# help(os)
# for f in glob('c?/*.jpg'):
#     os.remove(f)

% cd ../../..
%mkdir data/statefarm/results
%mkdir data/statefarm/sample/test


I am working on the file "FAI_old/statefarm-sample-codealong.ipynb".
The first task is:
Height shift: move the image up and down -
Can you write Python code for it?

gen_t = image.ImageDataGenerator(height_shift_range=0.05)
batches = get_batches(path + 'train', gen_t, batch_size=batch_size)

model = conv1(batches)


I am working on the file "FAI_old/statefarm-sample-codealong.ipynb".
The first task is:
Let's first check the number of parameters to see that there's enough parameters to find some useful relationships:
Can you write Python code for it?


I am working on the file "FAI_old/statefarm-sample-codealong.ipynb".
The first task is:
Looks like we can get a bit over 50% (almost, here: 42.8%) accuracy this way. This'll be a good benchmark for our future models - if we can't beat 50%, then we're not even beating a linear model trained on a sample, so we'll know that's not a good approach.
Single hidden layer
The next simplest model is to add a single hidden layer.
Can you write Python code for it?

model = Sequential([
            BatchNormalization(axis=1, input_shape=(3, 224, 224)),
            Flatten(),
            Dense(100, activation='relu'), #¿would λ2 be good here?
            BatchNormalization(),
            Dense(10, activation='softmax')
        ])
model.compile(Adam(lr=1e-5), loss='categorical_crossentropy', metrics=['accuracy'])
model.fit_generator(batches, batches.nb_sample, nb_epoch=2, validation_data=val_batches,
                    nb_val_samples=val_batches.nb_sample)

model.optimizer.lr = 0.01
model.fit_generator(batches, batches.nb_sample, nb_epoch=5, validation_data=val_batches,
                    nb_val_samples=val_batches.nb_sample)


I am working on the file "FAI_old/statefarm-sample-codealong.ipynb".
The first task is:
On GPU running out of memory (2692/3017 MiB) at this point. Restarting with smaller batch size (32?)
Can you write Python code for it?


I am working on the file "FAI_old/statefarm-sample-codealong.ipynb".
The first task is:
The training set here is very rapidly reaching a very high accuracy. So if we could regularize this, perhaps we could get a reasonable results.
So, what kind of regularization should we try first? As we discussed in lesson 3, we should start with data augmentation.
Data Augmentation
To find the best data augmentation parameters, we can try each type of data augmentation, one at a time. For each type, we can try four very different levels of augmentation, and see which is the best. In the steps below we've only kept the single best results we found. We're using the CNN we defined above, since we have already observed it can model the data quickly and accurately.
Width shift: move the image left and right -
Can you write Python code for it?

gen_t = image.ImageDataGenerator(width_shift_range=0.1)
batches = get_batches(path + 'train', gen_t, batch_size=batch_size)

model = conv1(batches)


I am working on the file "FAI_old/statefarm-sample-codealong.ipynb".
The first task is:
Validation Set (Sample)
How I'll do it: create a full val set in the full valid folder, then copy over the same percentage as train to the sample/valid folder.
Acutally: wouldn't it be better if I used the full validation set for more accurate results? Then again, for processing on my MacBook, it may be good enough to go w/ the 1st method.
1/3: function definitions for moving stuff & aming dirs:
Can you write Python code for it?

# run once, make sure you're in datadir first
# path = os.getcwd()
# os.mkdir(path + '/valid')
# for i in xrange(10): os.mkdir(path + '/valid' + '/c' + str(i))

def reset_valid(verbose=1, valid_path='', TRAIN_DIR=''):
    """Moves all images in validation set back to
    their respective classes in the training set."""
    counter = 0
    if not valid_path: valid_path = os.getcwd() + '/valid/'
    if not TRAIN_DIR:  TRAIN_DIR  = os.getcwd() + '/train'
    %cd $valid_path
    for i in xrange(10):
        %cd c"$i"
        g = glob('*.jpg')
        for n in xrange(len(g)):
            os.rename(g[n], TRAIN_DIR + '/c' + str(i) + '/' + g[n])
            counter += 1
        % cd ..
    if verbose: print("Moved {} files.".format(counter))
#         %mv $VALID_DIR/c"$i"/$*.jpg $TRAIN_DIR/c"$i"/$*.jpg

# modified from: http://forums.fast.ai/t/statefarm-kaggle-comp/183/20
def set_valid(number=1, verbose=1, data_path=''):
    """Moves <number> of subjects from training to validation
    directories. Verbosity: 0: Silent; 1: print no. files moved;
    2: print each move operation"""
    if not data_path: data_path = os.getcwd() + '/'
    counter = 0
    if number < 0: number = 0
    for n in xrange(number):
        # read CSV file into Pandas DataFrame
        dil = pd.read_csv(data_path + 'driver_imgs_list.csv')
        # group frame by subject in image
        grouped_subjects = dil.groupby('subject')
        # pick <number> subjects at random
        subject = grouped_subjects.groups.keys()[np.random.randint(0, high=len(grouped_subjects.groups))] # <-- groups?
        # get the group assoc w/ subject
        group = grouped_subjects.get_group(subject)
        # loop over gropu & move imgs to validation dir
        for (subject, clssnm, img) in group.values:
            source = '{}train/{}/{}'.format(data_path, clssnm, img)
            target = source.replace('train', 'valid')
            if verbose > 1: print('mv {} {}'.format(source, target))
            os.rename(source, target)
            counter += 1
    if...


I am working on the file "FAI_old/statefarm-sample-codealong.ipynb".
The first task is:
We're stabilizing at validation accuracy of 0.39 (~.35 in my NB). Not great, but a lot better than random. Before moving on, let's check that our validation set on the sample is large enough that it gives consistent results:
Can you write Python code for it?

rnd_batches = get_batches(path+'valid', batch_size=batch_size*2, shuffle=True)

val_res = [model.evaluate_generator(rnd_batches, rnd_batches.nb_sample) for i in range(10)]

np.round(val_res,2)


I am working on the file "FAI_old/statefarm-sample-codealong.ipynb".
The first task is:
Yup, pretty consistent - if we see imporvements of 3% or more, it's probably not random, based on the above samples.
L2 Regularization
The previous model is over-fitting a lot, but we can't use dropout since we only have one layer. We can try to decrease overfitting in our model by adding l2 regularization (ie: add the sum of squares of the weights to our loss function):
Can you write Python code for it?

model = Sequential([
            BatchNormalization(axis=1, input_shape=(3,224,224)),
            Flatten(),
            Dense(10, activation='softmax', W_regularizer=l2(0.01))
        ])
model.compile(Adam(lr=1e-5), loss='categorical_crossentropy', metrics=['accuracy'])
model.fit_generator(batches, batches.nb_sample, nb_epoch=2, validation_data=val_batches,
                    nb_val_samples=val_batches.nb_sample)

model.optimizer.lr=0.001
model.fit_generator(batches, batches.nb_sample, nb_epoch=4, validation_data=val_batches,
                    nb_val_samples=val_batches.nb_sample)