I am working on the file "FAI_old/statefarm_full_codealong.ipynb".
The first task is:
...and fine-tune our model using that data.
Can you write Python code for it?

bn_model.load_weights(path _ + 'models/da_conv8_1.h5')

bn_model.fit(comb_feat, comb_pseudo, batch_size=batch_size, nb_epoch=1,
             validation_data=(conv_val_feat, val_labels))

bn_model.fit(comb_feat, comb_pseudo, batch_size=batch_size, nb_epoch=4,
             validation_data=(conv_val_feat, val_labels))

bn_model.optimizer.lr=1e-5

bn_model.fit(comb_feat, comb_pseudo, batch_size=batch_size, nb_epoch=4,
             validation_data=(conv_val_feat, val_labels))


I am working on the file "FAI_old/statefarm_full_codealong.ipynb".
The first task is:
Based on some experiments the previous model works well, with bigger dense layers.
Can you write Python code for it?

def get_bn_da_layers(p):
    return [
        MaxPooling2D(input_shape = conv_layers[-1].output_shape[1:]),
        Flatten(),
        Dropout(p),
        Dense(256, activation='relu'),
        BatchNormalization(),
        Dropout(p),
        Dense(256, activation='relu'),
        BatchNormalization(),
        Dropout(p),
        Dense(10, activation='softmax')
        ]

p=0.8

bn_model = Sequential(get_bn_da_layers(p))
bn_model.compile(Adam(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])


I am working on the file "FAI_old/statefarm_full_codealong.ipynb".
The first task is:
BatchNorm Dense layers on pretrained Conv layers
Since we've pre-computed the output of the last convolutional layer, we need to create a network that takes that as input, and predicts our 10 classes. Let's try using a simplified version of VGG's dense layers.
Can you write Python code for it?

def get_bn_layers(p):
    return [
        MaxPooling2D(input_shape=conv_layers[-1].output_shape[1:]),
        Flatten(),
        Dropout(p/2),
        Dense(128, activation='relu'),
        BatchNormalization(),
        Dropout(p/2),
        Dense(128, activation='relu'),
        BatchNormalization(),
        Dropout(p),
        Dense(10, activation='softmax')
        ]

p = 0.8

bn_model = Sequential(get_bn_layers(p))
bn_model.compile(Adam(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])

bn_model.fit(conv_feat, trn_labels, batch_size=batch_size, nb_epoch=1,
             validation_data=(conv_val_feat, val_labels))

bn_model.optimizer.lr=0.01

bn_model.fit(conv_feat, trn_labels, batch_size=batch_size, nb_epoch=2,
             validation_data=(conv_val_feat, val_labels))

bn_model.save_weights(path + 'models/conv8.h5')

# bn_model.load_weights(path + 'models/conv8.h5')


I am working on the file "FAI_old/statefarm_full_codealong.ipynb".
The first task is:
I'm shocked by how good these results are! We're regularly seeing 75-80% accuracy on the validation set, which puts us into the top third or better of the competition. With such a simple model and no dropout or semi-supervised learning, this really speaks to the power of this approach to data augmentation. Noted. I'm seeing the same numbers
Four Conv/Pooling pairs + Dropout
Unfortunately, the results are still very unstable - the validation accuracy jumps from epoch to epoch. Perhaps a deeper model with some dropout would help.
Can you write Python code for it?

gen_t = image.ImageDataGenerator(rotation_range=15, height_shift_range=0.05,
                shear_range=0.1, channel_shift_range=20, width_shift_range=0.1)
batches = get_batches(path + 'train', gen_t, batch_size=batch_size)

model = Sequential([
            BatchNormalization(axis=1, input_shape=(3, 224, 224)),
            Convolution2D(32, 3, 3, activation='relu'),
            BatchNormalization(axis=1),
            MaxPooling2D(),
            Convolution2D(64, 3, 3, activation='relu'),
            BatchNormalization(axis=1),
            MaxPooling2D(),
            Convolution2D(128, 3, 3, activation='relu'),
            BatchNormalization(axis=1),
            MaxPooling2D(),
            Flatten(),
            Dense(200, activation='relu'),
            BatchNormalization(),
            Dropout(0.5),
            Dense(200, activation='relu'),
            BatchNormalization(),
            Dropout(0.5),
            Dense(10, activation='softmax')
        ])

model.compile(Adam(lr=1e-5), loss='categorical_crossentropy', metrics=['accuracy'])

model.fit_generator(batches, batches.nb_sample, nb_epoch=2, validation_data=val_batches,
                    nb_val_samples=val_batches.nb_sample)

model.optimizer.lr=1e-3

model.fit_generator(batches, batches.nb_sample, nb_epoch=10, validation_data=val_batches,
                    nb_val_samples=val_batches.nb_sample)

model.optimizer.lr=1e-5

model.fit_generator(batches, batches.nb_sample, nb_epoch=10, validation_data=val_batches,
                    nb_val_samples=val_batches.nb_sample)

# os.mkdir(path + 'models')
model.save_weights(path + 'models/conv8_prelim.h5')


I am working on the file "FAI_old/statefarm_full_codealong.ipynb".
The first task is:
Interestingly, with no regularization or augmentation, we're getting some reasonable results from our simple convolutional model. So with augmentation, we hopefully will see some very good results.
Data Augmentation
Can you write Python code for it?

gen_t = image.ImageDataGenerator(rotation_range=15, height_shift_range=0.05,
                shear_range=0.1, channel_shift_range=20, width_shift_range=0.1)
batches = get_batches(path + 'train', gen_t, batch_size=batch_size)

model = conv1(batches)

model.optimizer.lr = 1e-4
model.fit_generator(batches, batches.nb_sample, nb_epoch=15, validation_data=val_batches,
                    nb_val_samples=val_batches.nb_sample)


I am working on the file "FAI_old/statefarm_full_codealong.ipynb".
The first task is:
Let's include the real trianing data as well in its non-augmented form.
Can you write Python code for it?

da_conv_feat = np.concatenate([da_conv_feat, conv_feat])


I am working on the file "FAI_old/statefarm_full_codealong.ipynb".
The first task is:
Looks good - let's save those weights.
Can you write Python code for it?

bn_model.save_weights(path + 'models/da_conv8_1.h5')

bn_model.fit(conv_feat, trn_labels, batch_size=batch_size, nb_epoch=1,
             validation_data=(conv_val_feat, val_labels))
bn_model.optimizer.lr=0.01
bn_model.fit(conv_feat, trn_labels, batch_size=batch_size, nb_epoch=4,
             validation_data=(conv_val_feat, val_labels))
bn_model.optimizer.lr=1e-4
bn_model.fit(conv_feat, trn_labels, batch_size=batch_size, nb_epoch=4,
             validation_data=(conv_val_feat, val_labels))

bn_model.save_weights(path + 'models/conv8_bn_1.h5')


I am working on the file "FAI_old/statefarm_full_codealong.ipynb".
The first task is:
Now we can train the model as usual, with pre-computed augmented data.
Can you write Python code for it?

bn_model.fit(da_conv_feat, da_trn_labels, batch_size=batch_size, nb_epoch=1,
             validation_data=(conv_val_feat, val_labels))

bn_model.optimizer.lr=0.01

bn_model.fit(da_conv_feat, da_trn_labels, batch_size=batch_size, nb_epoch=4,
             validation_data=(conv_val_feat, val_labels))

bn_model.optimizer.lr=1e-4

bn_model.fit(da_conv_feat, da_trn_labels, batch_size=batch_size, nb_epoch=4,
             validation_data=(conv_val_feat, val_labels))


I am working on the file "FAI_old/statefarm_full_codealong.ipynb".
The first task is:
Pseudo-Labeling
We're going to try using a combination of psudeo labeling and knowledge distillation to allow us to use unlabeled data (ie: do semi-supervised learning). For our initial experiment we'll use the validation set as the unlabled data, so that we can see that it is working without using the test set. At a layer date we'll try using the test set.
To do this, we can simply calculate the predictions of our model...
Can you write Python code for it?

val_pseudo = bn_model.predict(conv_val_feat, batch_size=batch_size)


I am working on the file "FAI_old/statefarm_full_codealong.ipynb".
The first task is:
Re-run sample experiments on full dataset
We should find that everything that worked on the sample (see statefarm-sample.ipynb), works on the full dataset too. Only better! Because now we have more data. So let's see how they go - the models in this section are exact copies of the sample notebook models.
Single Conv Layer
Can you write Python code for it?

def conv1(batches):
    model = Sequential([
                BatchNormalization(axis=1, input_shape=(3,224,224)),
                Convolution2D(32, 3, 3, activation='relu'),
                BatchNormalization(axis=1),
                MaxPooling2D((3,3)),
                Convolution2D(64, 3, 3, activation='relu'),
                BatchNormalization(axis=1),
                MaxPooling2D((3,3)),
                Flatten(),
                Dense(200, activation='relu'),
                BatchNormalization(),
                Dense(10, activation='softmax')
            ])
    model.compile(Adam(lr=1e-4), loss='categorical_crossentropy', metrics=['accuracy'])
    model.fit_generator(batches, batches.nb_sample, nb_epoch=2, validation_data=val_batches,
                        nb_val_samples=val_batches.nb_sample)

    model.optimizer.lr = 1e-3
    model.fit_generator(batches, batches.nb_sample, nb_epoch=4, validation_data=val_batches,
                        nb_val_samples=val_batches.nb_sample)
    return model

model = conv1(batches)


I am working on the file "FAI_old/statefarm_full_codealong.ipynb".
The first task is:
Setup Batches
Can you write Python code for it?

batches = get_batches(path + 'train', batch_size=batch_size)
val_batches = get_batches(path + 'valid', batch_size=batch_size*2, shuffle=False)
# test_batches = get_batches(path + 'test', batch_size=batch_size, shuffle=False)

(val_classes, trn_classes, val_labels, trn_labels,
    val_filenames, trn_filenames, test_filenames) = get_classes(path)


I am working on the file "FAI_old/statefarm_full_codealong.ipynb".
The first task is:
Since we've now got a dataset 6x bigger than before, we'll need tocopy our labels 6 times too.
Can you write Python code for it?


I am working on the file "FAI_old/statefarm_full_codealong.ipynb".
The first task is:
Submit
We'll find a good clipping amount using the validation set, prior to submitting.
Can you write Python code for it?

def do_clip(arr, mx): return np.clip(arr, (1 - mx)/9, mx)

val_preds = bn_model.predict(conv_val_feat, batch_size=batch_size)

keras.metrics.categorical_crossentropy(val_labels, do_clip(val_preds, 0.93)).eval()

conb_test_feat = conv_model.predict_generator(test_batches, test_batches.n)

conv_test_feat = load_array(path + 'results/conv_test_feat.dat')

preds = bn_model.predict(conv_test_feat, batch_size=batch_size*2)

subm = do_clip(preds, 0.93)

subm_name = path + 'results/subm.gz'

classes = sorted(batches.class_indices, key=batches.class_indices.get)

submission = pd.DataFrame(subm, columns=classes)
submission.insert(0, 'img', [a[4:] for a in test_filenames]) # <-- why a[4:]?
# submission.insert(0, 'img', [f[8:] for f in test_filenames])
submission.head()

submission.to_csv(subm_name, index=False, compression='gzip')

FileLink(subm_name)


I am working on the file "FAI_old/statefarm_full_codealong.ipynb".
The first task is:
We'll use those to create a dataset of convolutional features 5x bigger than the training set.
Can you write Python code for it?

da_conv_feat = conv_model.predict_generator(da_batches, da_batches.nb_smaple*5)

save_array(path + 'results/da_conv_feat.dat', da_conv_feat)

da_conv_feat = load_array('results/da_conv_feat.dat')


I am working on the file "FAI_old/wordvectors_CodeAlong.ipynb".
The first task is:
Just for fun, let's take a look at a 2d projection of the first 350 words, using T-SNE
Can you write Python code for it?

reload(sys)
sys.setdefaultencoding('utf8')

tsne = TSNE(n_components=2, random_state=0)
Y = tsne.fit_transform(vecs[:500])

start=0; end=350
dat = Y[start:end]
plt.figure(figsize=(15,15))
plt.scatter(dat[:, 0], dat[:, 1])
for label, x, y in zip(words[start:end], dat[:, 0], dat[:, 1]):
    plt.text(x,y,label, color=np.random.rand(3)*0.7,
                 fontsize=14)
plt.show()


I am working on the file "FAI_old/wordvectors_CodeAlong.ipynb".
The first task is:
Looking at the vectors
After you've downloaded the pre-processed glove data, you should use tar -zxf to untar them, and put them in the path that (res_path) points to.
Then the following functino will return the word vectors as a matrix, the word list, and the mapping from word to index.
Can you write Python code for it?

def load_glove(loc):
    return (load_array(loc + '.dat'),
        pickle.load(open(loc + '_words.pkl', 'rb')),
        pickle.load(open(loc + '_idx.pkl', 'rb')))

vecs, words, wirdidx = load_glove(res_path + '6B.50d')
vecs.shape


I am working on the file "FAI_old/wordvectors_CodeAlong.ipynb".
The first task is:
Preprocessing
This section shows how we processed the original glove text files. However, there's no need for you to do this, since we provide the pre-processed glove data --- Link broken; instead I'm downloading from nlp.stanford.edu/projects/glove/glove.6B.zip
Can you write Python code for it?

def get_glove(name):
    with open(path+ 'glove.' + name + '.txt', 'r') as f: lines = [line.split() for line in f]
    words = [d[0] for d in lines]
    vecs = np.stack(np.array(d[1:], dtype=np.float32) for d in lines)
    wordidx = {o:i for i,o in enumerate(words)}
    save_array(res_path+name+'.dat', vecs)
    pickle.dump(words, open(res_path+name+'_words.pkl','wb'))
    pickle.dump(wordidx, open(res_path+name+'_idx.pkl','wb'))

get_glove('6B.50d')
# get_glove('6B.100d')
# get_glove('6B.200d')
# get_glove('6B.300d')


I am working on the file "FAI_old/wordvectors_CodeAlong.ipynb".
The first task is:
This is how you can look up a word vector.
Can you write Python code for it?

def w2v(w): return vecs[wirdidx[w]] # mispelled 'wordidx' up above... lol

w2v('of')


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
2. Data
Can you write Python code for it?

df_raw = pd.read_csv(f'{PATH}Train.csv', low_memory=False,
                     parse_dates=["saledate"])


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
2.2.2 Initial Processing
Can you write Python code for it?

m = RandomForestRegressor(n_jobs=-1)
m.fit(df_raw.drop('SalePrice', axis=1), df_raw.SalePrice)


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
2.2.3 Pre-processing
The nas coming out of proc_df() is a dictionary, where the keys are the names of columns with missing values, and the values are the medians.
Optionally you can pass nas as an additional arg to proc_df(), and it'll make sure it adds those specific columns and uses those specific medians. IE: it gives you the ability to say "process this test set exactly the same way we processed the training set." FAML1-L3: 00:07:00
Can you write Python code for it?

# df_raw = pd.read_feather('tmp/bulldozers-raw.feather')

df, y, nas = proc_df(df_raw, 'SalePrice')

??numericalize

df.columns


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
A validation set helps handle the issue of overfitting. Make it st it shares the test set's properties, ie: give it 12k rows just like the test set, and split it as the first n - 12k rows for training and the last 12k rows as validation set.
Can you write Python code for it?

def split_vals(a, n): return a[:n].copy(), a[n:].copy()

n_valid = 12000 # same as Kaggle's test set size
n_trn = len(df) - n_valid
raw_train, raw_valid = split_vals(df_raw, n_trn)
X_train, X_valid = split_vals(df, n_trn)
y_train, y_valid = split_vals(y, n_trn)

X_train.shape, y_train.shape, X_valid.shape


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
Above, preds[:,0] returns an array of the first prediction for each of our 10 trees. np.mean(preds[:,0]) returns the mean of those predictions, and y_valid[0] shows the actual answer. Most of our trees had inaccurate predictions, but the mean of them was actually pretty close.
Note: I probably made a mistake somewhere - or the data was too small - for multiple trees getting the exact answer
The models are based on different random subsets, and so their errors aren't correlated with eachother. The key insight here is to construct multiple models which are better than nothing, and the errors are - as much as possible - not correlated with eachother.
One of our first tunable hyperparameters is our number of trees.
What scikit-learn does by default is for N rows it picks out N rows with replacement: bootstrapping. ~ 63.2% of the rows will be represented, and a bunch of them multiple times.
The whole point of Machine Learning is to identify which variables matter the most and how do they relate to each other and your dependant variable together.
Random Forests were discovered/invented with the aim of creating trees as predictive and as uncorrelated as possible -- 1990s. Recent research has focused more on minimizing correlation: creating forests with trees that are individually less predictive, but with very little correlation.
There's another scikit-learn class called:
sklearn.ensemble.ExtraTreesClassifier

or
sklearn.ensemble.ExtraTreesRegressor

With the exact same API (just replace RandomForestRegressor). It's called an "Extremely Randomized Trees" model. It does exactly what's discussed above, but instead of trying every split of every variable, it randomly tries a few splits of a few variables.
So it's much faster to train, and has more randomness. With the time saved, you can build more trees - and therefore get better generalization.
In practice: if you have crappy individual trees, you just need more models to get a good overall model.

Now the obvious question: isn't this computationally expensive?...
Can you write Python code for it?

plt.plot([metrics.r2_score(y_valid, np.mean(preds[:i+1], axis=0)) for i in range(10)]);


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
After fitting the model and printing the score, the R2 score has dropped from 0.77 to 0.39. This is not a good model. It's better than the Mean-model (being > 0) but still not good.
But we can draw this model to take a look at it:
Can you write Python code for it?

draw_tree(m.estimators_[0], df_trn, precision=3)

df_raw.fiProductClassDesc.cat.categories

# df_raw.fiProductClassDesc.cat.codes


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
At first glance it doesn't look like anything's changed, but if you take a deeper look, you'll see the data type has changed to 'category'. 'category' is a Pandas class, with attributes accesible via .cat.xxxx
The index below shows that 'High' --> 0, 'Low' --> 1, 'Medium' --> 2
Can you write Python code for it?

df_raw.UsageBand.cat.categories

# we can do .cat.codes to get the actual numbers
df_raw.UsageBand.cat.codes


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
At this point, it looks like we're inside signal noise. More trees is never going to make the model worse - but a lower score is easily explained as whatever diminished accuracy gain being overwhelmed by noise in the random-sampling of the data.
If that's the case, I'd expect being able to see an R2 score greater than 0.79786, with the same hyperparameters:
Can you write Python code for it?

m = RandomForestRegressor(n_estimators=160, n_jobs=-1)
%time m.fit(X_train, y_train)
print_score(m)


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
Here we see our model, which had 0.982 R2 on the training set, got only 0.887 on the validation set, which makes us think it's overfitting quite badly. However it's not too badly because the RMSE on the logs of the prices (0.25) would've put us in the top 25% of the competition anyway (100/407).
*reran this on another machine
3.2 Speeding things up
Fast feedback is important for iteration and good interactive analysis. To this end we can pass in the subset par to proc_df() which'll randomly sample the data. We want no more than a 10sec wait when experimenting.
When you do this you still have to be careful your validation set doesn't change, and your training set doesn't overlap with it. So after sampling 30k items, we'll then taken the first 20k (since they're sorted by date) for our training data, and ignore the other 10k -- keeping our validation set the same as before.
Can you write Python code for it?

df_trn, y_trn, nas = proc_df(df_raw, 'SalePrice', subset=30000, na_dict=nas)
X_train, _ = split_vals(df_trn, 20000)
y_train, _ = split_vals(y_trn, 20000)

m = RandomForestRegressor(n_jobs=-1) # n_jobs=-1: set to num. cores on CPU
%time m.fit(X_train, y_train)
print_score(m)


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
If we don't limit depth, the training R^2 is of of course, a perfect 1.0. Because we can exactly predict every training element because it's in a leaf-node all it's own.
But the validation R^2 is not 1.0. It's a lot better than our super-shallow tree, but not as good as we'd like.
We want to find another way of making these trees better. And we'll do that by making a forest.
What's a forest?
To create a forest we're going to use a statistical technique called bagging.
3.4 Bagging
3.4.1 Intro to Bagging
You can bag any kind of model. The Random Forest is a way of bagging Decision Trees.
Bagging: what if we created 5 different models, each of which was only somewhat predictive, but the models weren't at all correlated with each other -- their predictions weren't correlated. That would mean the 5 models would've had to've found different insights into relationships in the data.
If you took the average of those models, you're effectively taking in the insights from each of them.
Averaging models: Ensembling.
Let's come up with a more specific idea of how to do this. What if we created a hole lot of these trees: big, deep, massively-overfit D-Trees. But each tree gets a random 1/10th of the data. And do that a hundred times with different random samples.
All the trees will have errors, but random errors. What's the average of a bunch of random errors? Zero. So if we take the average the error will average to zero and what's left is the true relationship.
That's a Random Forest.

After making those trees, we'll take our test data, run it through the tree, get to the leaf node, take the average in that leaf node for all the trees, and average them all together.
To do that we call RandomForestRegressor(.). An 'estimator' is what scikit-learn calls a tree. By default n_estimators = 10
Can you write Python code for it?

m = RandomForestRegressor(n_jobs=-1)
m.fit(X_train, y_train)
print_score(m)


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
Instead of 83 seconds of total compute time (15.2s thanks to multi-cores), we now run in only 2.94 total seconds of compute.
3.3 Single tree
Let's use that subset to build a model that's so simple we can actually take a look at it. We'll build a forest made of trees - and before we look at the forest, we'll look at the trees.
In scikit-learn the trees are called 'estimators'. Below we'll make a forest with a single tree n_estimators=1, and a small tree at that max_depth=3, and we'll turn off the random-component of the RandomForest bootstrap=False. Now it'll create a small deteriministic tree.
Can you write Python code for it?

m = RandomForestRegressor(n_estimators=1, max_depth=3, bootstrap=False, n_jobs=-1)
m.fit(X_train, y_train)
print_score(m)


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
Setting min_samples_leaf = 3 stops the RF when each leaf-node has 3 or fewer samples in it. In practice this means 1 or 2 fewer levels of decisions being made - which means around half the number of decision criteria, so it'll train much quicker. It also means when we look at an individual tree, instead of taking one point, wer're taking the average of 3 points - so we'd expect the trees to generalize better, though each tree is likely to be less powerful than before.
Values of in the range of 1, 3, 5, 10, 25 tend to work well for min_samples_leaf.
If you have a massive dataset and aren't using small samples, you may need a min_samples_leaf in the hundreds or thousands.
In this case, going from the default leaf-size of 1 to 3 has increased our valset R2 from 0.899 to 0.903.
We can also increase the amount of variation amongst the trees by not only using a sample of rows for each tree, but also using a sample of columns for each split. We do this by specifying max_features, which is the proportion of features to randomly select from at each split.
Idea: the less correlated your trees are with eachother: the better. Image you had 1 column that was so much better than all the other cols at being predictive, that every tree you built - regardless of which subset of rows - always started with that column. So the trees are all going to be pretty similar. But you can imagine there being some interaction of variables where that interaction is more important than that individual column.
So if each tree always splits on the same thing the first time, you're not going to get much variation on those trees.
So what we do in addition to just taking a subset of those rows, is at every single split point take a different subset of columns.
This is slightly different to the row sampling. In row-sampling, each new tree is based on a random set of rows. In column sampling every individual binary split chooses from a different subset of columns.
IoW: rather than looking at every possible level of every possible column, we look...
Can you write Python code for it?

m = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, max_features=0.5, n_jobs=-1, oob_score=True)
m.fit(X_train, y_train)
print_score(m)


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
The R^2 score shown below shows the variance (or mean?) of the data. It shows how much the data varies.
Can you write Python code for it?

m = RandomForestRegressor(n_jobs=-1)
m.fit(df, y)
m.score(df, y)


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
The basic idea is this: rather than limit the total amount of data that our model can access, let's instead limit it to a different random subset per tree. That way, given enough trees, the model can still see all the data, but for each individual tree, it'll be just as fast as if we'd cut down our dataset as before.
When we run set_rf_samples(20000), when we run a RF, it's not going to bootstrap an entire set of 390k rows, it's going to grab a subset of 20k rows.
So when running, it'll be just as fast as when we did a random sample of 20k, but now every tree can have access to the entire dataset. So if we have enough estimators/trees, the model will eventually see everything.
Can you write Python code for it?

set_rf_samples(20000)

m = RandomForestRegressor(n_jobs=-1, oob_score=True)
%time m.fit(X_train, y_train)
print_score(m)


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
The new date columns are numbers, but:
The categorical variables are currently stored as strings, which is inefficient, and doesn't provide the numeric coding required for a random forest. Therefore we call train_cats to convert strings to Pandas categories.
Can you write Python code for it?


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
This gets us a 0.899 R2 on the validation set, or a 0.908 on the OOB.
L2 1:21:54 Our OOB is better than our ValSet bc our ValSet is actually a different time period (future), whereas the OOB is a random sample. It's harder to predict the future.
Another way to reduce over-fitting is to grow our trees less deeply. We do this by specifying (with min_samples_leaf) that we require some minimum number of rows in every leaf node. This has two benefits:

There are less decision rules for each leaf node; simpler models should generalize better.

The predictions are made by average more rows in the leaf node, resulting in less volatility.
Can you write Python code for it?

m = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, n_jobs=-1, oob_score=True)
%time m.fit(X_train, y_train)
print_score(m)


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
To actually use this dataset and turn it into numbers, what we need to do is to take every categorical column and replace it with .cat.codes
This is done further below in 2.2.3 Pre-processing via proc_df()
Can you write Python code for it?

df_raw.UsageBand.cat.set_categories(['High', 'Medium', 'Low'], ordered=True, inplace=True)

df_raw.UsageBand = df_raw.UsageBand.cat.codes

display_all(df_raw.isnull().sum().sort_index()/len(df_raw))

os.makedirs('tmp', exist_ok=True)
df_raw.to_feather('tmp/bulldozers-raw.feather')


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
We can see those new date columns below where 'saledate' used to be:
Can you write Python code for it?


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
We don't see that much of an improvement over the R2 with the 20k data-subset, because we haven't used many estimators yet.
Since each additional tree allows th emodel to see more data, this approach can make additional trees more useful.
Can you write Python code for it?

m = RandomForestRegressor(n_estimators=40, n_jobs=-1, oob_score=True)
%time m.fit(X_train, y_train)
print_score(m)


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
We see a shape of 10 different sets of predictions and for each one our validation set of size 12,000 -- so 12,000 predictions for each of the 10 trees:
Can you write Python code for it?


I am working on the file "FAML1/Lesson1-RandomForests.ipynb".
The first task is:
With more estimators the model can see a larger portion of the data, and the R2 (2nd last value) has gone up from 0.8591 to 0.8755.
The Favorita groceries competition has over a hundred-million rows of data. There's no way you'll create an RF using 128M rows in every tree. It'll take forever. Instead you can use set_rf_samples(.) set to 100k or 1M.
The trick here is with a Random Forest using this technique, no dataset is too big. Even if it has 100B rows. You can just create a bunch of trees, each with a different subset.
NOTE: right now OOB Scores and set_rf_samples(.) are not compatible with each other, so you need to set oob_score=False if you use set_rf_samples(.) because the OOB score will be meaningless.
To turn of set_rf_samples(.) just call: reset_rf_samples()

A great big tip - that very few people in Industry or Academia use:
Most people run all their models on all their data all the time using their best possible pars - which is just pointless.
If you're trying to find which features are important and how they relate to oneanother, having that 4th dec place of accuracy isn't going to change anything.
Do most of your models on a large-enough sample size that your accuracy is reasonable (within a reasonable distance of the best accuracy you can get)  and it's taking a few seconds to train - so you can do your analysis.
3.5.2 Tree Building Parameters
We revert to using a full bootstrap sample in order to show the impact of the other over-fitting avoidance methods.
Can you write Python code for it?


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
'High', 'Low', 'Medium' in UsageBand will be seen by the RF as cats 0, 1, 2. It'll form a split first on either 0 vs 1, 2, or 2 vs 0, 1. That translates to 'High' vs 'Low' & 'Medium' or 'Medium' vs 'High' & 'Low'. That's a bit odd, and though the DT can get to a correct split regardless, by using a sensible ordering we can ensure it gets there in fewer splits - thus improving our model.
So we reorder 'High', 'Low', 'Medium' st. they're ordered wrt the category numbers, ie: so that any split starts by comparing 'High' and 'Low':
'High','Medium','Low' $\longrightarrow$ 0, 1, 2
ordered=True preserved supplied order, inplace=True changes the DataFrame in place instead of returning a new one.
Can you write Python code for it?

df_raw.UsageBand.cat.set_categories(['High','Medium','Low'], ordered=True, inplace=True)

print(df_raw.UsageBand[100:110])
print(df_raw.UsageBand.cat.codes[100:110])


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
'SalePrice' is now absent from the DataFrame's columns, and all columns with a non-zero value for null-fractions have corresponding '_na' columns.
Can you write Python code for it?


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
2. Data
low_memory=False tells Pandas to read more of the file to decide what the types are.
parse_dates=[...] is used for any columns that contain dates.
Can you write Python code for it?

df_raw = pd.read_csv(f'{PATH}Train.csv', low_memory=False, parse_dates=['saledate'])


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
2.2.2 Initial Processing
A Random Forest is smth of a universal machine learning technique. Could be a category or a continous variable - and can predict w/ cols of almost any kind (pixel data, zip codes, etc). RFs genrly don't overfit (prevention: easy). RFs don't genrly req validation sets, & can tell you how well they generalize - even w/ only 1 dataset. RFs have few if any statistical assumptions of your data, & req v.few pieces of feature engineering.
model.fit(Independant Variables,Dependent Variables)
Indep: used to predict; Dep: predicted. pandas.DataFrame.drop(..) returns a new DataFrame w/ a list of rows/cols removed. So we use everything but the SalePrice to predict the SalePrice.
Can you write Python code for it?

model = RandomForestRegressor(n_jobs=-1) # n_jobs: number of cores to use. -1 ==> all
model.fit(df_raw.drop('SalePrice', axis=1), df_raw.SalePrice)


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
2.2.3 Preprocessing
We still have a number of Null values. Here we display the fraction of Null values in each category:
Can you write Python code for it?


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
3. Random Forests
3.1 Base Model
Now we'll run our model again, but with the separate training and validation sets:
Can you write Python code for it?

def rmse(x,y): return math.sqrt(((x-y)**2).mean())

def print_score(m):
    res = [rmse(m.predict(X_train), y_train), rmse(m.predict(X_valid), y_valid),
           m.score(X_train, y_train), m.score(X_valid, y_valid)]
    if hasattr(m, 'oob_score_'): res.append(m.oob_score_)
    print(res)

model = RandomForestRegressor(n_jobs=-1)
%time model.fit(X_train, y_train)
print_score(model)


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
3.4 Bagging
3.4.1 Intro to Bagging
We can improve these D.Trees by making forests of them. We create forests with a statistical technique called 'bagging'. Any kind of model can be bagged. A Random Forest is a way of bagging trees.
What if we created N different models, each of which was only somewhat predictive, but the models weren't at all corelated with eachother. That means the N models would had to have found different insights into the relationships in the data. If you took the average of those N models, you're effectively bringing in the insights from each of them. This is Ensembling.
What if we made a bunch of these big, deep, strongly-overfit trees, but each one only gets a random 1/10th of the data. So each tree will be perfect on that subset but bad at the rest. So each of the trees will be better than nothing, and all overfit in different ways on different things because they use different random samples.
So they all have errors, but the errors are random. The average of a bunch of random errors is Zero.
So if we take the average of all these trees (ea. of which trained on a dfnt rand subset) the errors will average out to zero and what's left is the true relationship. That is a Random Forest. Lecture 2

Grab random subset of data.
Build a Decision Tree on it.
Put that D.Tree aside and repeat N times
For each D.Tree make predictions by running test data through tree to get to leaf node
Take average in that leaf node $\forall$ the trees
Average them all together.

To do that we call RandomForestRegressor. An estimator (specfd by n_estimators) is a D.Tree.

The key insight is to construct multitple models that are better than nothing and where the errors are as much as possible uncorrelated with eachother. If the errors are correlated this breaks down.
For subsets, Scikit-Learn picks out n rows with replacement. This is called bootstrapping. This on average represents 62.3% of the rows, with a bunch represented multiple times. (I think this means 62.3% rows used on any given tree).
Lecture 2: So...
Can you write Python code for it?

model = RandomForestRegressor(n_jobs=-1) # default is 10 estimators
model.fit(X_train, y_train)
print_score(model)


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
Also note the plot's flattening out. (tested in original notebook and course nb): adding more trees won't improve the model much beyond this point.
The n_estimators hyperparameter is chosen based on:
1. amount of time you have for fitting
2. point of diminshing returns
More trees slows model fit/train time, but fewer trees can still offer valuable insight. J.Howard will often work through a day with a 20-tree RF, and at the end expand that to a 1000-tree model.
3.4.2 Out-of-Bag (OoB) Score
Sometimes your dataset will be too small to create a validation set and a good model at the same time. There's a trick unique to Random Forests for this:
Recognize that some for each tree, some dataset rows did not get used. So pass those rows through those trees as their validation sets.
So you end up with a different validation set for each tree. Now to calculate our prediction we average all the trees where that row was not used for training. As long as you have enough trees every row will appear in the OoB sample for one of them at least.
So you create an OoB prediction by averaging all the trees you didn't use to train each individual row, then calculate your RMSE, R2, etc on that.
You can do this automatically by specifying the oob_score=True parameter in Scikit-Learn's RandomForestRegressor, creating a .oob_score_ attribute in the resulting model.
Can you write Python code for it?

model = RandomForestRegressor(n_estimators=40, n_jobs=-1, oob_score=True)
model.fit(X_train, y_train)
print_score(model)


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
Each of the estimators will train all the way down until the leaf nodes have 1 sample in them. NOTE that our OoB score is better than our validation R2 score (.89278) because our validation set is not a random sample: it's a different time period, and it's much harder to predict an entirely different time period than it is to predict random dates.

Another way to reduce over-fitting is to grow our trees less deeply. We do this by specifying (with min_samples_leaf) that we require some minimum number of rows in every leafe node. This has 2 benefits:
* There are fewer decision rules for each leaf node; simpler models should generalize better
* The predictions are made by averaging more rows in the leaf node, resulting in less volatility
example: min_samples_leaf=3: stop training the tree further when your leaf node has 3 or less samples in it.
In practice this means there'll be 1 or 2 fewer levels of decisions being made, which means about half or a quarter the number of actual decision criteria we have to do -- so it'll train quicker. It means also when we look at an individual tree, rather than just taking 1 point, we're taking the average of at least 3 points -- so we expect each tree to generalize a bit better; but ea. tree is also likely to be less powerful on its own.
Can you write Python code for it?

model = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, n_jobs=-1, oob_score=True)
%time model.fit(X_train, y_train)
print_score(model)


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
Entering a DataFrame to display it will truncate it if it's too long.
This function sets the truncation threshold to 1000 rows & cols.
Can you write Python code for it?

def display_all(df):
    with pd.option_context("display.max_rows", 1000):
        with pd.option_context("display.max_columns", 1000):
            display(df)


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
First we'll save our dataframe to disk in feather format since we have a good starting point.
Can you write Python code for it?

os.makedirs(PATH + 'tmp', exist_ok=True)
df_raw.to_feather(PATH + 'tmp/raw')


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
If we check the DataFrame, we see that everything is now a number:
Can you write Python code for it?


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
Increasing to 40 esimates increases our R2 score from 0.858 to 0.877.

set_rf_samples will be very useful for working with massive structured datasets.
Unfortunately as of 31/10/2017 (and now 25/3/2018) it will not change how OoB is calculated (set_rf_samples is a hack that replaces scikit-learn's internal function call with a lambda function with the desired behavior). OoB should be turned off for now when set_rf_samples is used.
NOTE to reset the RF sampling: call fastai.structured.reset_rf_samples()

When doing EDA (Exploratory Data Analysis) ie: when working and probing at a problem / doing interactive machine learning, J.Howard will use set_rf_samples (subsets) and reasonable small forests, because:

all the insights that I'm going to get are exactly the same as the big ones, but I can run them in 3 or 4 seconds instead of hours.
this is one of the biggest tips I can give you, and very very few people in industry or academia actually do this. Most people run all of their models on all of the data all of the time using their best possible parameters, which is just pointless.
if you're trying to find out which features are important and how they're related to each other and so forth: having that 4th decimal place of accuracy isn't going to change any of your insights at all.
do most of your models on a large enough sample size that your accuracy is reasonable (w/n a reasonable distance of the best accuracy you can get) and is taking a small number of seconds to train - so you can interactively do your analysis.

3.5.2 Tree-building Parameters
We revert to using a full bootstrap sample in order to show the impact of other over-fitting avoidance methods.
Can you write Python code for it?


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
NOTE: Random Forests are trivially parallelizable, meaning computation time more/less linearly scales (negatively) with number of CPUs.
The score is the R$^2$ value. Range is < 1. 1 is perfect. If your R$^2$ score is < 0 your model is worse than predicting the mean. FastAI ML1 L2 bit on R2. Gist of R$^2$: the ratio between how good your model is (RMSE) vs. how good is the naïve mean model (RMSE).
We'll create a simple validation set to test this. The dataset is sorted by date, so the most recent n rows will make up the validation set.
Can you write Python code for it?

def split_vals(a, n): return a[:n].copy(), a[n:].copy()

n_valid = 12000 # same as Kaggle's test set size # 12000 rows for val set
n_trn = len(df) - n_valid                        # all else in trn set
raw_train, raw_valid = split_vals(df_raw, n_trn)
X_train, X_valid = split_vals(df, n_trn)
y_train, y_valid = split_vals(y, n_trn)

X_train.shape, y_train.shape, X_valid.shape


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
Now the datatype for 'saledate' is numerical (int64). If we check the columns of the DataFrame we'll see the new ones added by add_datepart:
Can you write Python code for it?


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
Now we have something we can pass into a Random-Forest Regressor.
Can you write Python code for it?

model = RandomForestRegressor(n_jobs=-1)
model.fit(df, y)
model.score(df, y)


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
Now we want to replace the string categories with their numeric codes, handle missing continous values, and pull out the dependant variable (SalePrice) into a separate variable. The fastai.structured.proc_df is what we'll use to do this.

Overview of proc_df:
df: DataFrame | y_fld: name of dependent variable
• Makes copy of DataFrame. • Grabs y values. • Drops DepVar from DataFrame. • Then fixes missing via fastai.structured.fix_missing.

Overview of fix_missing:
• Check that the column has missing values (pd.isnull(col).sum() != 0). • Create new column with same name as original + '_na' that's a boolean column w/ 1 any time a value is missing, 0 otherwise. • Then replace all Null values with the columns median.
ie: All NaNs replaced by col's median. New col added keeping track of NaNs.

That is done for numeric variables (cols) -- Pandas automatically handles categorical variables by setting them to -1 if missing.
• Then call fastai.structured.numericalize.

Overview of numericalize:
• If column is Not numeric and is a categorical type: replace column w/ it's category codes (integers) + 1.
Can you write Python code for it?

df, y, nans = proc_df(df_raw, 'SalePrice')


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
Our model now has a validation R2 of 0.906. Our RMSE of log(price) has dropped from 0.233 to 0.229 as well. How good is that? Well on the Kaggle public leaderboard a loss of 0.2289 puts us in the top 20 of the competition. That's with a "totally brainless random forest with some totally brainless minor hyperparameter tuning."

This why the Random Forest is such an important - not just first step but often only step in Machine Learning. Because it's hard to screw it up (even when we didn't tune the hyperparameters we still got a good result), and a small amt of hypar tuning got a much better result.
So any kind of model - specifically Linear-type models which have a whole bunch of statistical assumptions and require a bunch of prereqs to get right before they start to work at all - can really throw you off track because they can give you totally wrong answers about how accurate the predictions can be.
The Random Forest generally-speaking tends to work on most datasets most of the time with most sets of hypars.

-- J.Howard Fast.ai ML1 Lecture 2
Random Forests work because the trees are p.much infinitely flexible. Even with a categorical variable - if there're particular categories which have different levels of price: it can gradually zoom in on those groups by using multiple splits.
You can help it by telling it the order of your CatVar, but even if you don't: it's okay, it'll just take a few more decisions to get there.
In a Linear model, or almost any other kind of model, especially non-tree models, encoding  CatVars the way RFs do won't work - because there's no linear relationship between arbitrary identifiers.

What does max_features do? The idea is that the less correlated your trees are w.eachother, the better. Imagine you had 1 column that was so much better than all the others at being predictive, that every single tree you built - regardless of which subset of rows - always started with that column. So the trees will all be pretty similar.
But you can imagine there might be some interaction of...
Can you write Python code for it?


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
RMSLE is used in the Kaggle competition. So by taking the log of all sale prices, we can just use RMSE later to calculate our loss. RMSLE: $Σ\big(($log(prediction) - log(actual)$)^2\big)$ : this means ratios not absolutes.
Here we also replace a column w/ a new column:
Can you write Python code for it?


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
The OoB Score will usually slightly underestimate the generalizability of the model -- the more trees you have, the less the underestimation of the model - but it works well enough anyway.
3.5 Reducing Over-Fitting
3.5.1 Subsampling
One of the easiest ways to avoid over-fitting is also one of the best ways to speed up analysis: subsampling. Let's return to using our full dataset, so we can demonstrate the impact of this technique.

NOTE: before we took a subset of 30k rows of the data and built every model on that. Meaning every tree in the RF is a different subset of that subset of 30k. Why not pick a totally different subset of 30k each time. ie: leave the entire dataset of 300k records as is, and if we want to make things faster: pick a different subset of 30k each time. So rather than bootstrapping the entire set of rows: let's just randomly sample a subset of data.

So we'll do this by calling proc_df without our subset parameter:
Can you write Python code for it?

df_trn, y_trn, nans = proc_df(df_raw, 'SalePrice')
X_train, X_valid = split_vals(df_trn, n_trn)
y_train, y_valid = split_vals(y_trn, n_trn)


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
There was some overfitting going on -- but this 0.252 loss gets into the top 25% of the Kaggle public leaderboard.

Fast.ai ML1 L2 p.much picks up from here.
Since the data/competition is predicted time-series data -- you want your validation set to reflect that by being a range of consecutive dates (specifically some tail slice of the dataset).
3.2 Speeding things up
One way to speed up iteration time for model development is to use the subset parameter in proc_df: which returns only a subset of the data to work on. This returns a randomly sampled subset of the data.
We need to make sure our train-subset doesn't overlap with our validation set. We also want to use our original val set, and not overwrite it, so of the 30k subset, we set the first 20k (this may overlap a bit..) to be training, and throw the rest away.

Create df_trn, y_trn from a random 30k subset of df_raw.
Set X_train, y_train to be the first 20k rows of df_trn, y_trn.
Can you write Python code for it?

df_trn, y_trn, nans = proc_df(df_raw, 'SalePrice', subset=30000)
X_train, _ = split_vals(df_trn, 20000)
y_train, _ = split_vals(y_trn,  20000)

model = RandomForestRegressor(n_jobs=-1) ## initialize Model
%time model.fit(X_train, y_train)        ## train Model
print_score(model)                       ## run predictions - still using orig valset


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
This dataset contains a mix of continuous and categorical variables. Most ML models (incl. RFs) req numbers -- so we need to convert all our cols to numbers.
sklearn.ensemble.RandomForestRegressor: predict continuous variables
sklearn.ensemble.RandomForestClassifier: predict categorical variables

One issue is saledate was parsed as a date $ \longrightarrow $ as a number. But if we look at it, it isn't a number, it's a datetime64 -- which is not a number. So we need to do our first bit of feature engineering.
Can you write Python code for it?


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
This isn't enough. One more bit of feature engineering is needed: there are strings in the dataset ('Low', 'High') etc. FastAI has function to automatically create categorical variables for all strings - by creating a column (backend) mapping integers to strings.
FastAI also has a apply_cats function to preserve training-set category mappings for validation & test set use.
Can you write Python code for it?


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
Values of 1, 3, 5, 10, 25 tend to work well for min_samples_leaf.
If working with a massive dataset without subsampling, you may need values of hundreds or thousands.

Here we see increasing min_samples_leaf from 1 to 3 has increased our Validation R$^2$ from 0.898 to 0.903. So it's a slight improvement and trains a bit faster.

We can also increase the amount of variation amongst the trees by not only using a sample of rows for each tree, but also using a sample of columns for each split. We do this by specifying max_features, which is the proportion of features to randomly select from at each split.
Can you write Python code for it?

model = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, max_features=0.5, n_jobs=-1, oob_score=True)
%time model.fit(X_train, y_train)
print_score(model)


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
We have 20k samples at the start of the this tree - because that's what we made the training set as when we split our data.
Looking at first node: in our whole dataset (X_train) there're 20k rows, the average sale price is ~10.1, and if we built a model where we used that average all the time: then our MSE would be 0.452. Iow that's the denominator of an R2. This 1st node is the most basic model: a tree with zero splits - just predict the average.
The best single split the RF is able to make is based on whether the Coupler_System is ≤ 0.5 (True / False). If it does that, the MSE of Coupler_System > 0.5 (False) goes to 0.114: a large improvement. In the other group: Coupler_System ≤ 0.5 (True) improves slightly to 0.397. The False group is a small fraction: ~1,700/20,000.

How to find the best possible split with a Single Decision Tree?:

For each categorical variable:
For each value of that variable:
Find the split with the Minimum weighted-average MSE

Return the categorry:value split w/ Minimum weighted-average MSE

Equivalent to this is to take the MSE of a hypothetical model where all values of a binary split are set to their decisions average.

Now we can improve this Decision Tree by setting depth=None to continue until each leaf node has only one decision possible for it. If we do that (surprise) we get a model that perfectly overfits our data. Our validation loss/error is not 1, but is better than our shallow tree.
Can you write Python code for it?

model = RandomForestRegressor(n_estimators=1, bootstrap=False, n_jobs=-1)
model.fit(X_train, y_train)
print_score(model)


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
We'll grab predictions for each individual tree and look at one example. After you've built a RF, each tree is stored in the attribute: .estimators_
Can you write Python code for it?

preds = np.stack([t.predict(X_valid) for t in model.estimators_])
preds[:,0], np.mean(preds[:,0]), y_valid[0] # print first tree's predictions

preds.shape # 12,000 predictions for each of 10 trees


I am working on the file "FAML1/RandomForests-Redux.ipynb".
The first task is:
df_raw.tail() will show the last few rows of the DataFrame. By default it shows the
cols at top and rows on side. There're a lot of cols, so using .transpose()
displays the table on its side.
Can you write Python code for it?

# display_all(df_raw.tail().transpose())

# display_all(df_raw.describe(include='all').transpose())

# df_raw.head()


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
An r^2 in the high-80's isn't bad at all (and the RMSLE puts us around rank 100 of 470 on the Kaggle leaderboard), but we can see from the validation set score that we're over-fitting badly. To understand this issue, let's simplify things down to a single small tree.
Speeding things up
Can you write Python code for it?

df_trn, y_trn, nas = proc_df(df_raw, 'SalePrice', subset=30000, na_dict=nas)
X_train, _ = split_vals(df_trn, 20000)
y_train, _ = split_vals(y_trn, 20000)

m = RandomForestRegressor(n_jobs=-1)
%time m.fit(X_train, y_train)
print_score(m)


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
Another way to reduce over-fitting is to grow our trees less deeply. We do this by specifying (with min_samples_leaf) that we require some minimum number of rows in every leaf node. This has two benefits:

There are less decision rules for each leaf node; simpler models should generalize better
The predictions are made by averaging more rows in the leaf node, resulting in less volatility
Can you write Python code for it?

m = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, n_jobs=-1, oob_score=True)
m.fit(X_train, y_train)
print_score(m)


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
But let's save this file for now, since it's already in format can we be stored and accessed efficiently.
Can you write Python code for it?


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
In any sort of data science work, it's important to look at your data, to make sure you understand the format, how it's stored, what type of values it holds, etc. Even if you've read descriptions about your data, the actual data may not be what you expect.
Can you write Python code for it?

def display_all(df):
    with pd.option_context("display.max_rows", 1000):
        with pd.option_context("display.max_columns", 1000):
            display(df)

display_all(df_raw.tail().transpose())

display_all(df_raw.describe(include='all').transpose())


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
Introduction to Blue Book for Bulldozers
About...
...our teaching
At fast.ai we have a distinctive teaching philosophy of "the whole game".  This is different from how most traditional math & technical courses are taught, where you have to learn all the individual elements before you can combine them (Harvard professor David Perkins call this elementitis), but it is similar to how topics like driving and baseball are taught.  That is, you can start driving without knowing how an internal combustion engine works, and children begin playing baseball before they learn all the formal rules.
...our approach to machine learning
Most machine learning courses will throw at you dozens of different algorithms, with a brief technical description of the math behind them, and maybe a toy example. You're left confused by the enormous range of techniques shown and have little practical understanding of how to apply them.
The good news is that modern machine learning can be distilled down to a couple of key techniques that are of very wide applicability. Recent studies have shown that the vast majority of datasets can be best modeled with just two methods:

Ensembles of decision trees (i.e. Random Forests and Gradient Boosting Machines), mainly for structured data (such as you might find in a database table at most companies)
Multi-layered neural networks learnt with SGD (i.e. shallow and/or deep learning), mainly for unstructured data (such as audio, vision, and natural language)

In this course we'll be doing a deep dive into random forests, and simple models learnt with SGD. You'll be learning about gradient boosting and deep learning in part 2.
...this dataset
We will be looking at the Blue Book for Bulldozers Kaggle Competition: "The goal of the contest is to predict the sale price of a particular piece of heavy equiment at auction based on it's usage, equipment type, and configuaration.  The data is sourced from auction result postings and includes information on usage and equipment configurations."
This is a very...
Can you write Python code for it?

df_raw = pd.read_csv(f'{PATH}Train.csv', low_memory=False,
                     parse_dates=["saledate"])


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
It's important to note what metric is being used for a project. Generally, selecting the metric(s) is an important part of the project setup. However, in this case Kaggle tells us what metric to use: RMSLE (root mean squared log error) between the actual and predicted auction prices. Therefore we take the log of the prices, so that RMSE will give us what we need.
Can you write Python code for it?


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
Let's see what happens if we create a bigger tree.
Can you write Python code for it?

m = RandomForestRegressor(n_estimators=1, bootstrap=False, n_jobs=-1)
m.fit(X_train, y_train)
print_score(m)


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
Pre-processing
In the future we can simply read it from this fast format.
Can you write Python code for it?


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
Random Forests
Base model
Let's try our model again, this time with separate training and validation sets.
Can you write Python code for it?

def rmse(x,y): return math.sqrt(((x-y)**2).mean())

def print_score(m):
    res = [rmse(m.predict(X_train), y_train), rmse(m.predict(X_valid), y_valid),
                m.score(X_train, y_train), m.score(X_valid, y_valid)]
    if hasattr(m, 'oob_score_'): res.append(m.oob_score_)
    print(res)

m = RandomForestRegressor(n_jobs=-1)
%time m.fit(X_train, y_train)
print_score(m)


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
Single tree
Can you write Python code for it?

m = RandomForestRegressor(n_estimators=1, max_depth=3, bootstrap=False, n_jobs=-1)
m.fit(X_train, y_train)
print_score(m)

draw_tree(m.estimators_[0], df_trn, precision=3)


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
The categorical variables are currently stored as strings, which is inefficient, and doesn't provide the numeric coding required for a random forest. Therefore we call train_cats to convert strings to pandas categories.
Can you write Python code for it?


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
The shape of this curve suggests that adding more trees isn't going to help us much. Let's check. (Compare this to our original model on a sample)
Can you write Python code for it?

m = RandomForestRegressor(n_estimators=20, n_jobs=-1)
m.fit(X_train, y_train)
print_score(m)

m = RandomForestRegressor(n_estimators=40, n_jobs=-1)
m.fit(X_train, y_train)
print_score(m)

m = RandomForestRegressor(n_estimators=80, n_jobs=-1)
m.fit(X_train, y_train)
print_score(m)


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
The training set result looks great! But the validation set is worse than our original model. This is why we need to use bagging of multiple trees to get more generalizable results.
Bagging
Intro to bagging
To learn about bagging in random forests, let's start with our basic model again.
Can you write Python code for it?

m = RandomForestRegressor(n_jobs=-1)
m.fit(X_train, y_train)
print_score(m)


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
This shows that our validation set time difference is making an impact, as is model over-fitting.
Reducing over-fitting
Subsampling
It turns out that one of the easiest ways to avoid over-fitting is also one of the best ways to speed up analysis: subsampling. Let's return to using our full dataset, so that we can demonstrate the impact of this technique.
Can you write Python code for it?

df_trn, y_trn = proc_df(df_raw, 'SalePrice')
X_train, X_valid = split_vals(df_trn, n_trn)
y_train, y_valid = split_vals(y_trn, n_trn)


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
Tree building parameters
We revert to using a full bootstrap sample in order to show the impact of other over-fitting avoidance methods.
Can you write Python code for it?


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
We can also increase the amount of variation amongst the trees by not only use a sample of rows for each tree, but to also using a sample of columns for each split. We do this by specifying max_features, which is the proportion of features to randomly select from at each split.

None
0.5

'sqrt'

1, 3, 5, 10, 25, 100
Can you write Python code for it?

m = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, max_features=0.5, n_jobs=-1, oob_score=True)
m.fit(X_train, y_train)
print_score(m)


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
We can specify the order to use for categorical variables if we wish:
Can you write Python code for it?

df_raw.UsageBand.cat.categories

df_raw.UsageBand.cat.set_categories(['High', 'Medium', 'Low'], ordered=True, inplace=True)

df_raw.UsageBand = df_raw.UsageBand.cat.codes


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
We'll grab the predictions for each individual tree, and look at one example.
Can you write Python code for it?

preds = np.stack([t.predict(X_valid) for t in m.estimators_])
preds[:,0], np.mean(preds[:,0]), y_valid[0]

preds.shape

plt.plot([metrics.r2_score(y_valid, np.mean(preds[:i+1], axis=0)) for i in range(10)]);


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
We'll replace categories with their numeric codes, handle missing continuous values, and split the dependent variable into a separate variable.
Can you write Python code for it?

df, y, nas = proc_df(df_raw, 'SalePrice')


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
We're still not quite done - for instance we have lots of missing values, wish we can't pass directly to a random forest.
Can you write Python code for it?


I am working on the file "FAML1/lesson1-rf-Copy1.ipynb".
The first task is:
todo define r^2
Wow, an r^2 of 0.98 - that's great, right? Well, perhaps not...
Possibly the most important idea in machine learning is that of having separate training & validation data sets. As motivation, suppose you don't divide up your data, but instead use all of it.  And suppose you have lots of parameters:

Underfitting and Overfitting

The error for the pictured data points is lowest for the model on the far right (the blue curve passes through the red points almost perfectly), yet it's not the best choice.  Why is that?  If you were to gather some new data points, they most likely would not be on that curve in the graph on the right, but would be closer to the curve in the middle graph.
This illustrates how using all our data can lead to overfitting. A validation set helps diagnose this problem.
Can you write Python code for it?

def split_vals(a,n): return a[:n].copy(), a[n:].copy()

n_valid = 12000  # same as Kaggle's test set size
n_trn = len(df)-n_valid
raw_train, raw_valid = split_vals(df_raw, n_trn)
X_train, X_valid = split_vals(df, n_trn)
y_train, y_valid = split_vals(y, n_trn)

X_train.shape, y_train.shape, X_valid.shape


I am working on the file "FB15K-237 Evaluation.ipynb".
The first task is:
Execute evaluation protocol
From [2]:

Given a set of triples in a set disjoint from a training
knowledge graph, we test models on predicting the subject or
object of each triple, given the relation type and the other
argument. We rank all entities in the training knowledge base in
order of their likelihood of filling the argument position. We
report the mean reciprocal rank of the correct entity, as well as
HITS@10 – the percent of test triples for which the correct
argument was ranked in the top ten. We use filtered measures
following the protocol proposed in Bordes et al. (2013) – that
is, when we rank entities for a given position, we remove all
other entities that are known to be part of an existing triple in
the training, development, or test set. This avoids penalizing
the model for ranking other correct fillers higher than the
tested argument. We thus report filtered mean reciprocal
rank (labeled MRR in the Figures), and filtered HITS@10. In the
figures we present MRR values scaled by 100, so that the maximum
possible MRR is 100.

 Note: filtering not yet implemented, code neither complete nor debugged for non-filtering case anyways
Can you write Python code for it?

def sp_query_reciprocal_rank(model, subj, rid, obj, entities):
    objs = [ result[0] for result in sp_query_results(model, subj, rid, entities) ]
    return reciprocal_rank(obj, objs)

def sp_query_hits_at_10(model, subj, rid, obj, entities):
    objs = [ result[0] for result in sp_query_results(model, subj, rid, entities) ]
    if obj in objs[:10]:
        return 1.0
    else:
        return 0.0

def sp_query_results(model, subj, rid, entities):
    return sorted([ [ subj, model.rank(pid, rid) ] for pid in sp_query_pairs(subj, entities) ],
                  reverse=True,
                  key=itemgetter(1))

def sp_query_pairs(subj, entities):
    return [ pair_id(subj, obj) for obj in entities if pair_id(subj, obj) > -1 ]

def po_query_reciprocal_rank(model, subj, rid, obj, entities):
    subjs = [ result[0] for result in po_query_results(model, obj, rid, entities) ]
    return reciprocal_rank(subj, subjs)

def po_query_hits_at_10(model, subj, rid, obj, entities):
    subjs = [ result[0] for result in po_query_results(model, obj, rid, entities) ]
    if subj in subjs[:10]:
        return 1.0
    else:
        return 0.0

def po_query_results(model, obj, rid, entities):
    return sorted([ [ obj, model.rank(pid, rid) ] for pid in sp_query_pairs(subj, entities) ],
                  reverse=True,
                  key=itemgetter(1))

def po_query_pairs(obj, entities):
    return [ pair_id(subj, obj) for subj in entities if pair_id(subj, obj) > -1]

def pair_id(subj, obj):
    pair = entity_pairs[(entity_pairs['subj'] == subj) & (entity_pairs['obj'] == obj)]
    if len(pair) > 0:
        return pair['pid'].values[0]
    else:
        return -1

def reciprocal_rank(correct_response, responses):
    return 1. / np.float(np.where(responses == correct_response)[0][0])

pairs = entity_pairs.to_dict(orient='records')
subj_idx = {}
obj_idx = {}
for pair in pairs:
    subj = pair['subj']
    obj = pair['obj']
    pid = pair['pid']
    if subj not in subj_idx.keys():
        subj_idx[subj] = {}
    if obj not in...


I am working on the file "FB15K-237 Evaluation.ipynb".
The first task is:
Print basic dataset statistics
Can you write Python code for it?

n_pairs = triples['pid'].max() + 1
m_relations = triples['rid'].max() + 1
l_entities = len(entities)
print n_pairs, 'pairs,', m_relations, 'relations,', l_entities, 'entities'


I am working on the file "FBA_tutorials/0_running_code_in_notebook.ipynb".
The first task is:
Large outputs
To better handle large outputs, the output area can be collapsed. Run the following cell and then single- or double- click on the active area to the left of the output:
Can you write Python code for it?


I am working on the file "FBA_tutorials/0_running_code_in_notebook.ipynb".
The first task is:
Managing the Kernel
Code is run in a separate process called the Kernel.  The Kernel can be interrupted or restarted.
Assignment (1 min):
Try running the following cell, it contains a sleep command that will do absolutely nothing for 20 seconds. During this time the kernal will be busy. Notice that in the top-right corner the circle will be black to indicate this. Hit the <button class='btn btn-default btn-xs'></button> button in the toolbar above to interrupt the computation.
Can you write Python code for it?


I am working on the file "FBA_tutorials/0_running_code_in_notebook.ipynb".
The first task is:
Plots: The power of powers
"The greatest shortcoming of the human race is our inability to understand the exponential function." --Albert Allen Bartlett (https://en.wikipedia.org/wiki/Albert_Allen_Bartlett)
Exponential growth is fast. Consider a population of bacteria or cancer cells. Each generation each bacteria in the population divides in two. The code below shows the (perhaps surprising) rate of growth in the number of bacteria.
Can you write Python code for it?

import matplotlib.pyplot as plt

population_size = {0:1} # in generation 0 there is one bacteria
for generation in range(1,25): # simulation of generations 1-24
    population_size[generation] = population_size[generation-1]*2

plt.plot(list(population_size.values()))
plt.show()


I am working on the file "FBA_tutorials/0_running_code_in_notebook.ipynb".
The first task is:
Restarting the kernels manually
The kernel maintains the state of a notebook's computations. You can reset this state by restarting the kernel. This is done by clicking on the <button class='btn btn-default btn-xs'></button> in the toolbar above.
Note: Restarting the kernel whipes the memory clean.
Assignment (2 min):
1. Execute the cell below. This will save the variable dna_seq, containing a short string of DNA sequence, in the memory.
2. Now restart the kernel manually as described above.
3. In the second cell below try showing the contents of dna_seq.
Does python still know what dna_seq is?
Can you write Python code for it?


I am working on the file "FBA_tutorials/1_introduction_to_python.ipynb".
The first task is:
Assignment (1 min): Calculate and print the number of molecules in $0.33~\mu M/L$
Getting help in the notebook
When you want to know what a command or function does you can type a question mark in fron of the command.
Can you write Python code for it?

import math # load some common mathematical operations
?math.sqrt(2)


I am working on the file "FBA_tutorials/1_introduction_to_python.ipynb".
The first task is:
Assignment (1 min): print the name of your favorite protein below.
Congratulations! You are now a Python programmer.
Numbers matter in biology
Luckily we can use Python as a calculator.
Below we show some examples. Do you understand what each one does?

TIP (Division): In Python 3, 3 / 2 returns 1.5 (floating-point division). 3 // 2 for integer division.
Can you write Python code for it?


I am working on the file "FBA_tutorials/1_introduction_to_python.ipynb".
The first task is:
Here, 1:3 represents a slice going from element 1 included (this is the second element of the list) to element 3 excluded. Thus, we get a sublist with the second and third element of the original list. The first-included/last-excluded asymmetry leads to an intuitive treatment of overlaps between consecutive slices. Also, note that a sublist refers to a dynamic view of the original list, not a copy; changing elements in the sublist automatically changes them in the original list.
Assignment (5 min): In the code cell below make a list of the first 10 prime numbers: https://en.wikipedia.org/wiki/Prime_number. Use the sum() function to figure out the sum of the first 10 prime numbers.
Assignment (1 min):
Print the second-to-last prime number from your list to the screen.
Dictionaries
Dictionaries contain key-value pairs. They are extremely useful and common. They allow you to map, or point, keys to values. In the example below the letters a,b,c are now pointing to the numbers 1,2,3.
For a flux balance analysis application of a dictionary, you can think of a dictionary that points each of the reactions in a network to its flux in the FBA solution.
You can access the value a certain key points to with square bracket notation:
Can you write Python code for it?

my_dict = {'a': 1, 'b': 2, 'c': 3}
print('a:', my_dict['a'])

list(my_dict.keys())


I am working on the file "FBA_tutorials/1_introduction_to_python.ipynb".
The first task is:
Lists
A list contains a sequence of items. You can concisely instruct Python to perform repeated actions on the elements of a list. Let's first create a list of numbers:
Can you write Python code for it?


I am working on the file "FBA_tutorials/1_introduction_to_python.ipynb".
The first task is:
Note the syntax we used to create the list: square brackets [], and commas , to separate the items.
The built-in function len() returns the number of elements in a list:
Can you write Python code for it?


I am working on the file "FBA_tutorials/1_introduction_to_python.ipynb".
The first task is:
Python tries to tell you where it stopped understanding, but in the above examples, each program is only 1 line long.
It also tries to show you where on the line the problem happened with caret ("^").
Finally it tells you the type of thing that went wrong, (NameError, SyntaxError, ZeroDivisionError) and a bit more information like "name 'gibberish' is not defined" or "unexpected EOF while parsing".
Unfortunately you might not find "unexpected EOF while parsing" too helpful. EOF stands for End of File, but what file? What is parsing? Python does it's best, but it does take a bit of time to develop a knack for what these messages mean. If you run into an error you don't understand please ask a tutor.
Types of variables in python
When you define a variable in python it has a type. Above we dealt with numbers which are of type ... Below we briefly introduce the other types you might see in the rest of the tutorial.
Simply put there is text, i.e. strings, and two kinds of containers, e.g. lists and dictionaries.
All together now
An example for each of the most important types in python
Can you write Python code for it?

x = 'One definition of systems biology: the study of the interactions between the components of biological systems, \
and how these interactions give rise to the function and behavior of that system (for example, the enzymes and \
metabolites in a metabolic pathway or the heart beats)' # string
print(type(x))
print(x)
print()

x = [0.5,2,24] # list
print(type(x))
print(x)
print('The (rough) cell cycle time of e.coli =',x[0],'hrs, of yeast =',x[1],'hrs, of a human cell',x[2],'hrs')
print()

x = {'e. coli':5,'yeast':12,'human':2.9e3} # dictionary
print(type(x))
print(x)
print('The genome size of e.coli =',x['e. coli'],'Mbp, of yeast =',x['yeast'],'Mbp, of a human cell',x['human'],'Mbp')


I am working on the file "FBA_tutorials/1_introduction_to_python.ipynb".
The first task is:
Python's built-in mathematical operators include +, -, *, **, for exponentiation, / for division, // for integer division (without rest), % for division but giving you the rest.
Assignment (1 min): Experiment with these a bit
See: http://bionumbers.hms.harvard.edu/KeyNumbers.aspx
According to bionumbers (see link) e. coli has a volume of up to $5~\mu m^3$ and yeast has a volume of up to $160~\mu m^3$
Assignment (3 min): Calculate in the cell below how many times bigger the volume of yeast is compared to e. coli.
Assignment (3 min): Using the bionumbers page find the "water molecule diameter" and the diameter of a yeast cell. Then calculate how many water molecules you could theoretically lay side by side in a yeast cell.
Variables
form a fundamental concept of any programming language. A variable has a name and a value. Here is how to create a new variable in Python:
Can you write Python code for it?

avogadro = 6e23
print('How many molecules are contained in a mole? Answer:',avogadro,'molecules.')


I am working on the file "FBA_tutorials/1_introduction_to_python.ipynb".
The first task is:
Strings are surrounded by quotes. Without the quotes Hello by itself would be viewed as a variable name.
You can use either double quotes (") or single quotes (') for text/strings. As we saw before we can also save text in variables.
Let's use strings with variables!
Can you write Python code for it?

your_name = "James Watson"
print("Hello,",your_name)


I am working on the file "FBA_tutorials/1_introduction_to_python.ipynb".
The first task is:
Strings in Python are a bit more complicated because the operations on them aren't just + and * (though those are valid operations).
Dot notation and object oriented programming
Python, like many programming languages, supports Object Oriented Programming or OOP for short. In this paradigm, we approach ideas as Objects much as we do in the real world. Each Object is an instance of a Class or a type of object. Such an object may have certain properties or function that can be applied to them.
So what does all this have to do with dot notation? Dot notation allows us to tell a instance of a class to use one of the functions inside that class. That is why we access the sqrt function from the math module with the dot. And why we access the 'upper' function of a string as my_string.upper()
Assignment (2 min): Below, after your_string, type a dot and press Tab. You will get a list of function you can apply to the string. Pick two commands that seem interesting to you.
Note that you have to define a string first to be able to get help. So first execute the cell so that the variable your_string is known. Then using the Tab key find one of your choice.
Note: Don't worry about the ones that are difficult to understand. This is just to get you comfortable with the dot notation and finding functions and properties of objects.
Can you write Python code for it?


I am working on the file "FBA_tutorials/1_introduction_to_python.ipynb".
The first task is:
The Written Word, i.e. strings
Numbers are great... but most of our day to day computing needs involves text, from emails to tweets to documents. Or in biology: DNA sequences, chemical formulas, hyperlinks between databases etc.
We have already seen a couple strings in Python. Programmers call text strings because they are weird like that. From now on we will only refer to strings, but we just mean pieces of text inside our code.
Can you write Python code for it?


I am working on the file "FBA_tutorials/1_introduction_to_python.ipynb".
The first task is:
There are several things to note here:

The for organism in genome_sizes.keys() syntax means that a temporary variable named organism is created at every iteration. This variable contains the value of every item in the list, one at a time.
Note the colon : at the end of the for statement. Forgetting it will lead to a syntax error!
The  print statement will be executed for all items in the list.
Note the four spaces before print: this is called the indentation. You will find more details about indentation in the next subsection.

Assignment (3 min): Write your own for loop that prints each element of your list of 10 prime numbers divided by 2 separately to the screen.
list comprehensions
Python supports a concise syntax to perform a given operation on all elements of a list using for loops:
Can you write Python code for it?

items = [1,2,3,4,5,6]
squares = [item * item for item in items]
squares


I am working on the file "FBA_tutorials/1_introduction_to_python.ipynb".
The first task is:
We can access sublists with the following syntax:
Can you write Python code for it?


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
Assignment (3 min): Do you know the biological role of all these compartments?
See here for more information: http://www.ncbi.nlm.nih.gov/books/NBK26907/
Some metabolites (like Glucose for example) can be associated with multiple compartments.
Can you write Python code for it?


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
Assignment (3 min): Investigate the metabolites that are produced (i.e. in the OUT FLUXES column). Are these what you would have expected?
Assignment (5 min): What would happen if we force the cell to produce something else than lactate? Try this by setting the upper_bound of the exchange reaction of lactate 'EX_lac_D_e' to zero. Do this using the . notation to access the upper_bound attribute of the lactate exchange reaction. Then rerun model.optimize() and look at the model.summary() again.
What new products have been produced? Did the growth rate remain the same?
If you want repeat this process a couple times. Be sure to block multiple metabolites simultaneously.
Can you write Python code for it?

model = M.copy()

# change some upper bounds here

model.optimize()
model.summary()


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
Assignment (3 min): Reflect for a moment on the differences compared to the ATPM objective. Why is the ATPS4 flux down? Why is the oxygen/glucose ratio down?
Parsimonious enzyme-usage flux balance analysis (pFBA)
(Lewis et al., 2010) http://doi.org/10.1038/msb.2010.47
pFBA is an often-used extension of normal flux balance balance analysis. It performs FBA as usual but with an additional optimization that minimizes the total flux in the network. Intuitively this means that if there are two solutions that achieve the same growth rate but one solution is a lot longer than the other, i.e. it has more steps and therefore the total flux is higher, pFBA will return the shorter solution.
Biologically, this may also be interpreted as giving you an optimal solution that is efficient in its enzyme usage.
Note: There is one pitfall with pFBA, the 'objective_value' returned refers to the total flux in the network not the objective flux. You can get the fluxes, as with model.optimize() through the 'fluxes' attribute
Can you write Python code for it?

model_pfba = M.copy()
sol = pfba(model_pfba)
print('Solution status: {}'.format(sol.status))
print('Objective value: {}'.format(sol.fluxes['biomass_reaction']))
print('Total flux in pFBA solution: {}'.format(sol.objective_value))


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
Assignment (5 min): Above you might have realized that the model can grow optimally while only producing CO2 as a carbon product.
Can you imagine why this happens? What could we change about the medium to force the model into producing other carbon exits? Try something below.
What happens now when you block the carbon exits like lactate?
Can you write Python code for it?

model = M.copy()

# change some upper/lower bounds here

model.optimize()
model.summary()


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
Are essential amino acids essential?
We will now look into which reactions the model predicts to be essential for growth. Are the "essential" amino acids predicted to be essential?
Calculating the essential genes using flux variability analysis
One of the uses of FVA is to predict essential reactions. If we require growth to be non-zero then reactions that are essential will have an associated FVA interval that does not contain zero.
Assignment (3 min): Make sure you understand the statement made above. A helpful example is to think of glucuse exchange reaction. What would happen to this flux when the cell tries to grow optimally? Does this reaction need to carry flux? What does that say about the FVA interval?
The cell below returns a table with essential reactions predicted by FVA. It might take a while to run...
Can you write Python code for it?

model = M.copy()

# first perform pFBA to find a list of potentially essential reactions
# reasoning: if reaction v is essential it ALWAYS has to carry flux. So also in this pFBA solution.
# doing this first allows us to perform FVA on a smaller set of reactions
sol = pfba(model)

# take only those reactions that carry flux
reactions_with_flux = sol.fluxes[sol.fluxes.abs() > 1e-9].index.tolist()

# perform fba on the flux carrying reactions
fva_sol = cobra.flux_analysis.flux_variability_analysis(model,reaction_list=reactions_with_flux,fraction_of_optimum=0.8)

# filter out
fva_sol = fva_sol.loc[(fva_sol['minimum'] > 1e-9) | (fva_sol['maximum'] < -1e-9) ]
df = fva_sol.copy()
for r in df.index.tolist():
    df = df.set_value(r,'reaction',model.reactions.get_by_id(r).reaction)
    df = df.set_value(r,'genes',model.reactions.get_by_id(r).gene_reaction_rule)
df


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
Flux variability analysis (FVA)
Questions
* How uniquely determined is the flux distribution returned by flux balance analysis?
Objectives
* Learn how to determine flux capacities using flux variability analysis.
* Learn how to generate phenotypic phase planes for different fluxes.

Flux balance analysis gives you one optimal steady state solution for the model you are simulating. Above you saw that you could alter the network, e.g. by blocking one lactate exit, and still get an optimal solution. This means that the FBA solution is not uniquely determined.
What is flux variability analysis?
(Mahadevan and Schilling, 2003) http://doi.org/10.1016/j.ymben.2003.09.002
Flux variability analysis is closely related to flux balance analysis.
Remember that FBA calculate one flux distribution such that the whole network is at steady state, mass-balanced, thermodynamically feasible and optimal.
In contrast, FVA calculates for each reaction the range of feasible flux that is consistent with these constraints FBA uses. So for a given reaction $v$, FVA will tell you what the minimal and maximal attainable flux is through that reaction while keeping the solution at steady state and optimal.
Technically, FVA for one reaction, for instance pyruvate kinase, involves once performing FBA with the objective of maximizing the flux through pyruvate kinase and once performing FBA to minimize its flux. It then returns this interval [minimum_feasible flux, maximum_feasible_flux].
Why would we do FVA?

Flux Balance Analysis solutions are not necessariliy unique. Flux Variablity Analysis is a good tool for estimating the space of alternative optimal solutions. If you see that there is an entire range of feasible fluxes in the optimal growth state for pyruvate kinase. It means there are multiple solutions that are equally optimal but with different pyruvate kinase fluxes.
Below we will look at how to find essential reactions with FVA
We will see later in the tutorial that FVA may be used to predict biomarkers of network...
Can you write Python code for it?

model = M.copy()

# a short list of reactions to calculate to reduce computational time. Feel free to adjust the list
interesting_reactions = ['EX_bhb_e','EX_acac_e','EX_urea_e','PHETHPTOX2','PYK','SUCD1m','ATPS4m',]

result = cobra.flux_analysis.flux_variability_analysis(model,reaction_list=interesting_reactions)
result[['minimum','maximum']]

result.plot.bar(stacked=True)


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
How is ATP generated?
When maximizing ATPM using only glucose and o2
Below we first remove the amino acids from the medium in order to just ask for ATP production from glucose and oxygen.
Then we make the ATP maintenance reaction: atp -> adp + pi, the objective reaction.
Assignment (3 min): Before running the cell below. Based on your textbook knowledge: what would you expect the maximum number of ATP to be given 1 unit of glucose and unlimited oxygen?
How much oxygen would be used for this?
Now execute the cell and compare the results.
Can you write Python code for it?

model = M.copy()

for rxn in ['EX_his_L_e','EX_ile_L_e','EX_leu_L_e','EX_lys_L_e','EX_met_L_e','EX_phe_L_e','EX_thr_L_e','EX_trp_L_e','EX_val_L_e']:
    model.reactions.get_by_id(rxn).lower_bound = 0

model.objective = model.reactions.ATPM
model.optimize()

model.summary()
print()
model.metabolites.atp_c.summary()
print()
model.metabolites.atp_m.summary()
print()
model.metabolites.h_i.summary()


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
How many metabolites are there?
Can you write Python code for it?


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
If you are not sure which ID the metabolite you are looking has, you can use a query to look for it. Suppose we want to find lactate
Can you write Python code for it?

for metabolite in model.metabolites.query('lactate','name'):
    print(metabolite.id, metabolite.name,metabolite.compartment)


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
Let's take a closer look at the reactions associated with Glyceraldehyde 3-phosphate (g3p). Reactions like metabolites have both a short ID, a longer name attribute and an attribute called reaction that contains the chemical reactions equation. Lets see all of these for the reactions g3p engages in.
Can you write Python code for it?

for reaction in model.metabolites.g3p_c.reactions:
    print(reaction.id, reaction.name, reaction.reaction)


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
Metabolites are associated with compartments in the cell. Glyceraldehyde 3-phosphate (g3p_c) is associated with the c (Cytosol) compartment.
Can you write Python code for it?


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
Metabolites are not isolated things. They participate in reactions as substrates and products.
Can you write Python code for it?


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
Model components
Above we loaded the human metabolic reconstruction model. You can think of this model as having two key components:
1. The metabolic network of metabolites that are connected through reactions catalyzed by enzymes. This is captured mathematically as the "stoichiometry matrix".
2. Coupling of reactions to an identifier for the gene(s) that encode the enzyme catalyzing the reaction.
As a consequence these models allow you to investigate the contents of the network and perturb them by doing gene knockouts. Since the model contains the information linking a specific gene to the reactions its enzyme catalyzes it becomes possible to see how such knockouts interfere with the metabolic network.
In Python you can interact with the model through broadly 3 different kinds of object: metabolites, reactions and genes. We will look at these below.
The stoichiometry matrix
The stoichiometry matrix completely defines the network. The model for the human metabolic network contains 5324 metabolites and roughly 7785 reactions. The stoichiometry matrix therefore has 5324 rows and 7785 columns. You can see that below in the image. A blue dot in a certain row & column means that the metabolite that row refers to partakes in the reaction that column refers to.
If metabolite number 5 is involved in reaction 99 with a stoichiometry coefficient of -1 (i.e. it is a substrate) in that reaction, then the stoichiometry matrix will have the value -1 in the rows 5 column 99.
One metabolite like g3p maybe engages in 10 reactions or so. Whereas ATP might engage in hundreds of reactions. However, no metabolite gets even close to 7000 reactions. As a consequence the stoichiometry matrix mostly consists of zeros.
Can you write Python code for it?

S = model.copy().to_array_based_model().S.todense()
S

import matplotlib.pyplot as plt
plt.spy(S, precision=0.01, markersize=.1)
plt.show()


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
One can access a specific metabolite using dot notation.
Can you write Python code for it?


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
One can look up the molecular formula of glucose and its weight.
The .elements attribute returns a dictionary representation of the formula.
Can you write Python code for it?

model.metabolites.glc_D_c.formula
model.metabolites.glc_D_c.formula_weight

model.metabolites.g3p_c.elements


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
Reactions
Reactions have similar properties as metabolites
How many metabolites are there?
Can you write Python code for it?

len(model.reactions)

for reaction in model.reactions[1000:1010]:
    print(reaction.id,reaction.name)


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
So the objective is biomass. Don't get confused by the subtraction of biomass_reaction_reverse. The details do not matter now, but internally the two directions are handled separately and if the flux is "reversed" with a flux of 5 you would want the objective score to be -5. That is why it is subtracted.
Assignment (5 min): Is optimal growth a realistic assumption for human cells? For bacterial cells? Discuss...
Doing FBA is simple...
Actually performing FBA with cobrapy once you have loaded a model is really easy and fast.
Can you write Python code for it?


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
Solution: Use the method get_by_id instead!
Can you write Python code for it?


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
The reactions in the human metabolic reconstruction are categorized in various compartments.
Can you write Python code for it?


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
Warning: One cannot use dot notation to access metabolites, reactions, or genes if their identifiers do not resemble proper Python variable names.
Can you write Python code for it?


I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
When maximizing biomass
Below we do the same again but for biomass as the objective reaction.
Can you write Python code for it?

model = M.copy()

model.optimize()

model.summary()
print()
model.metabolites.atp_c.summary()
print()
model.metabolites.atp_m.summary()
print()
model.metabolites.h_i.summary()