I am working on the file "FADL1/cifar10_redux_pt-practice.ipynb".
The first task is:
3. ConvNet
Can you write Python code for it?

class ConvNet(nn.Module):
    def __init__(self, layers, c):    # c: num classes
        super().__init__()
        self.layers = nn.ModuleList([
            nn.Conv2d(layers[i], layers[i+1], kernel_size=3, stride=2)
            for  i in range(len(layers) - 1)
        ])
        self.pool = nn.AdaptiveMaxPool2d(1)  # define pooling layer to use in forward()
        self.out = nn.Linear(layers[-1], c)  # final FC/Lin layer

    def forward(self, x):
        for λ in self.layers:
            x = F.relu(λ(x))  # same as λ_x=λ(x); x=F.relu(λ_x)
        x = self.pool(x)      # I guess only 1 Conv layer so only 1 Pool at end?
        x = x.view(x.size(0), -1)  # trsfm tensor for final Linlayer
        return F.log_softmax(self.out(x), dim=-1)

learner = ConvLearner.from_model_data(ConvNet([3, 20, 40, 80], 10), data)

learner.summary()


I am working on the file "FADL1/cifar10_redux_pt-practice.ipynb".
The first task is:
4. Refactored ConvNet
Can you write Python code for it?

class ConvLayer(nn.Module):
    def __init__(self, ni, nf):
        super().__init__()
        self.conv = nn.Conv2d(ni, nf, kernel_size=3, stride=2, padding=1)

    def forward(self, x):
        return F.relu(self.conv(x))


I am working on the file "FADL1/cifar10_redux_pt-practice.ipynb".
The first task is:
5. Batch Normalization
Batch Normalization is used to allow creation of deeper networks. You'll find if you try to make the previous networks much deeper, they'll start encountering NaN and 0 losses and little in between. BatchNorm increases the resiliency of a neural network.
Can you write Python code for it?

class BnLayer(nn.Module):
    def __init__(self, ni, nf, stride=2, kernel_size=3):
        super().__init__()
        self.conv = nn.Conv2d(ni, nf, kernel_size=kernel_size, stride=stride,
                              bias=False, padding=1)
        self.a = nn.Parameter(torch.zeros(nf, 1, 1)) # adder
        self.m = nn.Parameter(torch.ones(nf, 1, 1))  # multiplier

    def forward(self, x):
        x = F.relu(self.conv(x))
        x_chan = x.transpose(0,1).contiguous().view(x.size(1), -1)
        if self.training: # true during training; false during evaluation
            self.means = x_chan.mean(1)[:, None, None] # calc mean of each conv filter (channel)
            self.stds  = x_chan.std(1)[:, None, None]  # calc stdev ofeach conv filter (channel)
        return (x - self.means) / self.stds*self.m + self.a # subtract means, divide by stdevs


I am working on the file "FADL1/cifar10_redux_pt-practice.ipynb".
The first task is:
6. Deep BatchNorm
We can't just add more stride=2 Conv layers because we're already down to $2\times2$ by our 5th conv layer (the 80 filters $\rightarrow$ in, 160 fltrs $\rightarrow$ out layer).
Instead what we do is create stride=1 layers, and build our network as pairs of $[CONV(stride=2),CONV(stride=1)]$, effectively doubling the depth of our model. ($\forall$ the looped-layers, that is)
Can you write Python code for it?

class ConvBnNet2(nn.Module):
    def __init__(self, layers, c):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 10, kernel_size=5, stride=1, padding=2)
        self.layers = nn.ModuleList([BnLayer(layers[i], layers[i+1])
                                     for i in range(len(layers) - 1)])
        self.layers2 = nn.ModuleList([BnLayer(layers[i+1], layers[i+1], 1)
                                      for i in range(len(layers) - 1)])
        self.out = nn.Linear(layers[-1], c)

    def forward(self, x):
        x = self.conv1(x)
        for λ, λ2 in zip(self.layers, self.layers2):
            x = λ(x)
            x = λ2(x)
        x = F.adaptive_max_pool2d(x, 1)
        x = x.view(x.size(0), -1)
        return F.log_softmax(self.out(x), dim=-1)

# creates a 12-layer ConvNet. 1 larger Conv in, 10 conv middle, 1 Lin out.
learner = ConvLearner.from_model_data(ConvBnNet2([10, 20, 40, 80, 160], 10), data)

%time learner.fit(1e-2,2)


I am working on the file "FADL1/cifar10_redux_pt-practice.ipynb".
The first task is:
Fast.ai DL1(2018) Lecture 7
NB: now we don't need to normalize input at all because its automatically normalizing it per/channel, and for later layers per/filter. But this wont do anything bc SGD will simply undo it all every minibatch.
So we introduce learnable parameters self.a & self.m that SGD can optimize. Now, if SGD wants to scale up or shift the layer, it doesn't need to change the entire layer's weight-matrix, it can just change the numbers of that particular filter $\longrightarrow$ via the channel multiplier or adder.
torch.nn.Parameter tells SGD it can learn these as weights ie: we flag it for PyTorch to learn it by backprop
NB2: BatchNorm also regularizes, meaning you can decrease or remove Dropout and WeightDecay.
The reason is the μ&σ of each minibatch is different $\Longrightarrow$ self.m and self.a will change w/ea minibatch. This means that the filters are changing slightly w/ea minibatch $\leftrightarrow$ which has the effect of introducing noise $\leftrightarrow$ which is a regularization effect.
NB3: for real BatchNorm you take an exponentially-weighted moving average of the μ & σ, instead of the current minibatch's μ & σ.
NB4: V.Important: <<if self.training:>> ensures you only use BatchNorm while training. You do not want to change the meaning of the model while running on testdata.
...oh my god... you're shitting me...
THIS is why you need to specify multiple versions of the same model in Keras, because it's a Static-Computation framework. That was very confusing for me in v1 of the GLoC Detector adapting Keras-RetinaNet's training API via L.Fridman's Boring Detector...
Oy.
Can you write Python code for it?

class ConvBnNet(nn.Module):
    def __init__(self, layers, c):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 10, kernel_size=5, stride=1, padding=2)
        self.layers = nn.ModuleList([BnLayer(layers[i], layers[i+1])
                                     for i in range(len(layers) - 1)])
        self.out = nn.Linear(layers[-1], c)

    def forward(self, x):
        x = self.conv1(x)
        for λ in self.layers:
            x = λ(x)
        x = F.adaptive_max_pool2d(x, 1)
        x = x.view(x.size(0), -1)
        return F.log_softmax(self.out(x), dim=-1)


I am working on the file "FADL1/cifar10_redux_pt-practice.ipynb".
The first task is:
Here a large conv layer is added to the start, with a large kernel size and stride 1. The idea is to have an input layer with a richer input. We use bigger filters on bigger areas; and output 10 5x5 filters. Current (2018) modern approach. (sometimes even 11x11 kernel & 32 out filters). $Stride = 1$ along with $padding = \dfrac{kernelsize - 1}{2}$ ie: padding=2 here, allows us to have an output with the same size as our input, but with more filters.
Can you write Python code for it?

# 5 Conv layerse + 1 FC layer
learner = ConvLearner.from_model_data(ConvBnNet([10,20,40,80,160], 10), data)

learner.summary()

%time learner.fit(3e-2, 2)

%time learner.fit(1e-1, 4, cycle_len=1)


I am working on the file "FADL1/cifar10_redux_pt-practice.ipynb".
The first task is:
I am so happy that worked. I swear Python analytics programming with Jupyter is super rewarding compared to other programming.
Can you write Python code for it?


I am working on the file "FADL1/cifar10_redux_pt-practice.ipynb".
The first task is:
So it looks like the Adaptive Max Pooling layers are sorta defined along with or folded into the Conv Layers. They were defined more explicitly earlier via: self.pool = nn.AdaptiveMaxPool2d(out_size) so the layer shows up as 'AdaptiveMaxPool2d-[num]', but here we have the AdvMPool defined in the forward function via x=F.adaptive_max_pool2d(x,1) so it shows up as 'ConvLayer-[num]' instead. Hmm.
Oh interesting, that is the case. Before the AdvPool layer was defined as a torch.nn.AdaptiveMaxPool2d() as an attribute during initialization, but now it's defined as a torch.nn.functional.adaptive_max_pool2d(input, out_size) as part of the forward pass function. Hmm.
I like the flexibility, but I'm gonna want a way for the layers to at least be named right so I can glance and know what they are.
Aha! From lecture: maxpool in 3. was done as an object, but maxpool doesnt have any state (there're no weights in maxpooling), so it can be done w/ a little less code by calling it as a function in forward.
A lot of (everything?) you can do as a class in PyTorch, you can also do as a function. eg: torch.nn.AdaptiveMaxPool2d $\longleftrightarrow$ torch.nn.functional.adaptive_max_pool2d
Can you write Python code for it?

%time learner.fit(1e-1, 2)

%time learner.fit(1e-1, 2, cycle_len=1)


I am working on the file "FADL1/cifar10_redux_pt-practice.ipynb".
The first task is:
That means every layer of SimpleNet after the first is a $1 \times V$ vector, where $V$ is the 'Tensor volume' of the first layer (product of all dimensions).
What does that mean in English (I'm also explaining this to myself)?
It means that each layer is a straight up Linear, or, Fully-Connected layer. The classic neural-net image of columns of nodes.
Then why isn't the first layer also a $1 \times V$ vector?
That question is actually wrong: all the layers are linear. Notice it is x in forward() not the layers in __init__() that gets shaped. If you imagine x as a tensor traveling forward through the network: you see if first arrives as the image itself, which then goes through the first layer and is transformed via the forward() function from a $1\times{3}\times{32}\times{32}$ (I think that's the CIFAR10 size) tensor into a $1\times{V}$ tensor (vector), and continues forward in that shape through to the end of the network (where it becomes $1\times{C}$ (where $C$ is the number of classes), but I don't think that's been defined yet).
Aha! layers aint a number! layers is an actual array (or smth) of PyTorch layers. So their in/out dims * are * included, $\Longrightarrow$ the output shape of $C=10$ * is * encoded. Cool.
I've been confused in Deep Learning by conflating layers with their activations. Once you separate the actual layers of nodes from their tensors of activations, Neural Networks start to become much easier to imagine. For example, this is necessary to understand how come a Fully-Convolutional Network's architecture remains unchanged when input size is changed, despite the fact that the produced tensors vary directly with input size. The Conv filters remain stoically as they were.
Goooot iit. Okay, so SimpleNet is initialized as an instance (or sub/child instance?) of the torch.nn.Module class, inheriting its intialization via super().__init__(). $\Longrightarrow$ it's base initialization is as the PyTorch base neural-network class, torch.nn.Module.
But! more important than that: the layers...
Can you write Python code for it?

layers = [32*32*3, 40, 10]

print([[layers[i], layers[i+1] ]for i in range(len(layers)-1)])

torch.nn.ModuleList([
    nn.Linear(layers[i], layers[i+1]) for i in range(len(layers) -1)
])

# using the Fast.ai library to do its black magic
# by passing our SimpleNet class into fastai.conv_learner.ConvLearner
learner = ConvLearner.from_model_data(SimpleNet([32*32*3, 40, 10]), data)

print(learner)
print([par.numel() for par in learner.model.parameters()])

learner.lr_find()
learner.sched.plot()


I am working on the file "FADL1/cifar10_redux_pt-practice.ipynb".
The first task is:
That's it. Everything else will be identical.. except we can now do way more layers than before.
We're going to make it 3x as deep as before and it's going to train great, just because of that one line: x + super().forward(x)
Can you write Python code for it?

class ResNet(nn.Module):
    def __init__(self, layers, c):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 10, kernel_size=5, stride=1, padding=2)
        self.layers = nn.ModuleList([BnLayer(layers[i], layers[i+1])
                                     for i in range(len(layers) - 1)])
        self.layers2 = nn.ModuleList([ResnetLayer(layers[i+1], layers[i+1], 1)
                                      for i in range(len(layers) - 1)])
        self.layers3 = nn.ModuleList([ResnetLayer(layers[i+1], layers[i+1], 1)
                                      for i in range(len(layers) - 1)])
        self.out = nn.Linear(layers[-1], c)

    def forward(self, x):
        x = self.conv1(x)
        for λ, λ2, λ3 in zip(self.layers, self.layers2, self.layers3):
            x = λ3(λ3(λ(x)))
        x = F.adaptive_max_pool2d(x, 1)
        x = x.view(x.size(0), -1)
        return F.log_softmax(self.out(x), dim=-1)

learner = ConvLearner.from_model_data(ResNet([10,20,40,80,160],10), data)

# weight decay
wd = 1e-5

%time learner.fit(1e-2, 2, wds=wd)


I am working on the file "FADL1/cifar10_redux_pt-practice.ipynb".
The first task is:
checking I moved the validation set correctly
Can you write Python code for it?


I am working on the file "FADL1/custom_model.ipynb".
The first task is:
Alrighty, that gave me some idea of how a custom model is added to FastAI.
Getting length of dataset. Dataset is my custom-built set for my G-LOC-Detector.
Can you write Python code for it?

df = pd.read_csv(path + 'train.csv')
len(df['id'])


I am working on the file "FADL1/custom_model.ipynb".
The first task is:
Here's code to do cross-validation:
Can you write Python code for it?

def get_model_i(i=0):
    val_idxs = get_cv_idxs(n, i, val_pct=0.1)
    tfms = tfms_from_model(f_model, sz, aug_tfms=transforms_basic, max_zoom=1.05)
    data = ImageClassifierData.from_csv(path, 'train', f'{path}train.csv', bs, tfms,
                                        val_idxs=val_idxs, suffix='.jpg', continuous=True)
    learn = ConvLearnerVGG.pretrained(data, ps=0.0, precompute=False)
    return learn


I am working on the file "FADL1/custom_model.ipynb".
The first task is:
How to train is a W.I.P. --- great..
Can you write Python code for it?

def fit_and_predict(learn):
    learn.fit(1e-3, 3)
    learn.fit(1e-4, 4)
    print("unfreezing")
    learn.unfreeze_prev_layer()
    #learn.fit(1e-5, 3, cycle_len=1, cycle_mult=2)
    learn.fit(1e-5, 3)
    return learn.TTA()


I am working on the file "FADL1/custom_model.ipynb".
The first task is:
Predictions
Can you write Python code for it?

preds = []
for i in range(11):
    print("iteration ", i)
    learn = get_model_i(i)
    preds.append(fit_and_predict(learn))


I am working on the file "FADL1/custom_model.ipynb".
The first task is:
So, the numbers in the top layers are different from Yannet's, but the Conv layer pars are exactly the same. Pretty sure this is because when you finetune a model, the FC layer/s you stack ontop are randomly initialized, but the Conv layers coming from a pretrained network (PyTorch VGG 16), are necessarily exactly the same, given you got the same pretrained weights.
Cross Validation
Can you write Python code for it?

bs=32; sz=224
n = 7637
transforms_basic = [RandomRotateXY(10), RandomDihedralXY()]
transforms_basic = [RandomRotateXY(10)]


I am working on the file "FADL1/custom_model.ipynb".
The first task is:
This'll actually take forever on an i5 CPU, but it does start training so, that's good.
Can you write Python code for it?

def reshape_preds(preds):
    predictions = [preds[i][0] for i in range(11)]
    y = [preds[i][1] for i in range(11)]
    pp = np.vstack(predictions)
    yy = np.vstack(y)
    print(yy.shape)
    pp = np.maximum(pp, 0.0)
    err = np.abs(pp - yy).mean()
    print("err", err)

reshape_preds(preds)


I am working on the file "FADL1/dogbreeds.ipynb".
The first task is:
Rerunning without validation for predictions:
Can you write Python code for it?

from sklearn import metrics
PATH = "data/dogbreeds/"
arch = resnext50
sz = 224
bs = 64
labels_csv = f'{PATH}labels.csv'
# n = len(list(open(labels_csv)))-1
val_idxs = get_cv_idxs(0)
def get_data(sz, bs):
    tfms = tfms_from_model(arch, sz, aug_tfms=transforms_side_on, max_zoom=1.1)
    data = ImageClassifierData.from_csv(PATH, 'train', labels_csv, bs=bs, tfms=tfms,
                                        val_idxs=val_idxs, suffix='.jpg', test_name='test')
    return data if sz < 300 else data.resize(340, 'tmp')

data = get_data(sz, bs)
learn = ConvLearner.pretrained(arch, data, precompute=True)
learn.fit(1e-2, 2)
learn.precompute=False
learn.fit(1e-2, 5, cycle_len=1)
learn.set_data(get_data(299, bs=32))
learn.fit(1e-2, 3, cycle_len=1)
learn.fit(1e-2, 3, cycle_len=1, cycle_mult=2)


I am working on the file "FADL1/keras_lesson1.ipynb".
The first task is:
Data Augmentation parameters copied from fastai --> copied from Keras Docs
Instead of creating a single data object, in Keras we have to define a data generator to specfy how to generate the data. We have to tell it what kind of dat aug, and what kind of normalization.
Generally, copy-pasting Keras code from the internet works.
Keras uses the same directory structure as FastAI.
2 possible outcomes: class_mode='binary'. Multipe: 'categorical'
In Keras you have to specify a data generator without augmentation for the testing set.
Important to NOT shuffle the validation set, or else accuracy tracking can't be done.
Can you write Python code for it?

train_datagen = ImageDataGenerator(rescale=1./255, shear_range=0.2,
                                   zoom_range=0.2, horizontal_flip=True)
test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(train_data_dir,
                                                    target_size=(sz,sz),
                                                    batch_size=batch_size,
                                                    class_mode='binary')
validation_generator = train_datagen.flow_from_directory(valid_data_dir,
                                                         shuffle=False,
                                                         target_size=(sz,sz),
                                                         batch_size=batch_size,
                                                         class_mode='binary')


I am working on the file "FADL1/keras_lesson1.ipynb".
The first task is:
In Keras you have to manually specify the base model and construct on top the layers you want to add.
Can you write Python code for it?

base_model = ResNet50(weights='imagenet', include_top=False)
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024, activation='relu')(x)
predictions = Dense(1, activation='sigmoid')(x)


I am working on the file "FADL1/keras_lesson1.ipynb".
The first task is:
Keras expects to be told how many batches there are per epoch. num_batches = size of generator / batch_size
Keras also defaults to zero workers. For good speed: include num workers.
Can you write Python code for it?

%%time
model.fit_generator(train_generator, train_generator.n // batch_size, epochs=3, workers=4,
        validation_data=validation_generator, validation_steps=validation_generator.n // batch_size)


I am working on the file "FADL1/keras_lesson1.ipynb".
The first task is:
Specify the model. There's no concept of automatically freezing layers in Keras, so you have to loop through the layers you want to freeze and call .trainable=False
Keras also has a concept of compiling a model, which DNE in FastAI / PyTorch
Can you write Python code for it?

model = Model(inputs=base_model.input, outputs=predictions)
for layer in base_model.layers: layer.trainable = False
model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])


I am working on the file "FADL1/lesson1-dogbreeds.ipynb".
The first task is:
2. Initial Exploration
Can you write Python code for it?

!ls {PATH}

label_df = pd.read_csv(label_csv)
label_df.head()

# use Pandas to create pivot table which shows how many of each label:
label_df.pivot_table(index='breed', aggfunc=len).sort_values('id', ascending=False)

tfms = tfms_from_model(arch, sz, aug_tfms=transforms_side_on, max_zoom=1.1)
data = ImageClassifierData.from_csv(PATH, folder='train', csv_fname=f'{PATH}labels.csv',
                                    test_name='test', val_idxs=val_idxs, suffix='.jpg',
                                    tfms=tfms, bs=bs)

fn = PATH + data.trn_ds.fnames[0]; fn

img = PIL.Image.open(fn); img

img.size

size_d = {k: PIL.Image.open(PATH + k).size for k in data.trn_ds.fnames}

row_sz, col_sz = list(zip(*size_d.values()))

row_sz = np.array(row_sz); col_sz = np.array(col_sz)

row_sz[:5]

plt.hist(row_sz);

plt.hist(row_sz[row_sz < 1000])

plt.hist(col_sz);

plt.hist(col_sz[col_sz < 1000])

len(data.trn_ds), len(data.test_ds)

len(data.classes), data.classes[:5]


I am working on the file "FADL1/lesson1-dogbreeds.ipynb".
The first task is:
3. Initial Model
starting w/ small images, large batch sizes to train model v.fast in beginning; increase image size and decrease batch-size as go along.
Can you write Python code for it?

def get_data(sz, bs):
    tfms = tfms_from_model(arch, sz, aug_tfms=transforms_side_on, max_zoom=1.1)
    data = ImageClassifierData.from_csv(PATH, 'train', f'{PATH}labels.csv', test_name='test',
                                        num_workers=4, val_idxs=val_idxs, suffix='.jpg',
                                        tfms=tfms, bs=bs)
    return data if sz > 300 else data.resize(340, 'tmp')


I am working on the file "FADL1/lesson1-dogbreeds.ipynb".
The first task is:
3.1 Precompute
Can you write Python code for it?

data = get_data(sz, bs)

learn = ConvLearner.pretrained(arch, data, precompute=True) # GTX870M;bs=64;sz=224;MEM:2431/3017

learn.fit(1e-2, 5)


I am working on the file "FADL1/lesson1-dogbreeds.ipynb".
The first task is:
3.3 Increase Size

If you train smth on a smaller size, you can call learn.set_data() and pass in a larger sized dataset. That'll take your model, however it's trained so far, and continue to train on larger images.
This is another way to get SotA results. Starting training on small images for a few epochs, then switching to larger images and continuing training is an amazing effective way to avoid overfitting.

J.Howard (paraphrased)
NOTE: Fully-Convolutional Architectures only.
Can you write Python code for it?

learn.set_data(get_data(299, bs=32))
learn.freeze() # just making all but last layer already frozen

learn.fit(1e-2, 3, cycle_len=1) # precompute is off so DataAugmentation is back on

learn.fit(1e-2, 3, cycle_len=1, cycle_mult=2)

log_preds, y = learn.TTA()
probs = np.exp(log_preds)
accuracy(log_preds, y), metrics.log_loss(y, probs)

learn.save('299_pre')

# learn.load('299_pre')

learn.fit(1e-2, 1, cycle_len=2)

learn.save('299_pre')

log_preds, y = learn.TTA()
probs = np.exp(log_preds)
accuracy(log_preds, y), metrics.log_loss(y, probs)

SUBM = f'{PATH}subm/'
os.makedirs(SUBM, exist_ok=True)
df.to_csv(f'{SUBM}subm.gz', compression='gzip', index=False)

FileLink(f'{SUBM}subm.gz')


I am working on the file "FADL1/lesson1-dogbreeds.ipynb".
The first task is:
6. Individual Prediction
Can you write Python code for it?

fn = data.val_ds.fnames[0]

fn

Image.open(PATH+fn).resize((150,150))

trn_tfms, val_tfms = tfms_from_model(arch, sz)

learn = ConvLearner.pretrained(arch, data)
learn.load('299_pre')

# ds = FilesIndexArrayDataset([fn], np.array([0]), val_tfms, PATH)
# dl = DataLoader(ds)
# preds = learn.predict_dl(dl)
# np.argmax(preds)

im = trn_tfms(Image.open(PATH+fn))
preds = to_np(learn.model(V(T(im[None]).cuda())))
np.argmax(preds)

trn_tfms, val_tfms = tfms_from_model(arch, sz)

im = val_tfms(Image.open(PATH+fn)) # or could apply trn_tfms(.)
preds = learn.predict_array(im[None]) # index into image as[None] to create minibatch of 1 img
np.argmax(preds)


I am working on the file "FADL1/lesson1-rxt50-CA.ipynb".
The first task is:

Can you write Python code for it?

# This file contains all the main external libs we'll use
from fastai.imports import *

from fastai.transforms import *
from fastai.conv_learner import *
from fastai.model import *
from fastai.dataset import *
from fastai.sgdr import *
from fastai.plots import *

PATH = "data/dogscats/"
sz = 299
ARCH = resnext50
bs = 8 # if a TitanX is maxing out at 28, I'll give this 870M 8..

tfms = tfms_from_model(ARCH, sz, aug_tfms=transforms_side_on, max_zoom=1.1)
data = ImageClassifierData.from_paths(PATH, tfms=tfms, bs=6, num_workers=4, test_name='test1')
learn = ConvLearner.pretrained(ARCH, data, precompute=False, ps=0.5)
learn.load('RNx_224_all_50')


I am working on the file "FADL1/lesson1-rxt50-CA.ipynb".
The first task is:
Good, so there's no loss of data when saving as a DataFrame.

This model got 0.08151 on the Kaggle Dogs vs Cats Redux competition. Tied for 266/1314.

Testing FDs Issue
https://github.com/fastai/fastai/issues/23#issuecomment-345412558
THIS TESTING SECTION:
fastai library updated to use loops instead of list comprehension to fix max-open file-descriptors issue -- conda-installed PyTorch uninstalled and replaced w/ source-installs.
Can you write Python code for it?

!ulimit -n

%reload_ext autoreload
%autoreload 2
%matplotlib inline

from fastai.imports import *

from fastai.transforms import *
from fastai.conv_learner import *
from fastai.model import *
from fastai.dataset import *
from fastai.sgdr import *
from fastai.plots import *

PATH = "data/dogscats/"
sz = 299
ARCH = resnext50
bs = 8

tfms = tfms_from_model(ARCH, sz, aug_tfms=transforms_side_on, max_zoom=1.1)
data = ImageClassifierData.from_paths(PATH, tfms=tfms, bs=6, num_workers=4, test_name='test1')
learn = ConvLearner.pretrained(ARCH, data, ps=0.5, precompute=False)

learn.load('RNx_224_all_50')

log_preds = learn.TTA(is_test=True)[0]

log_preds.shape


I am working on the file "FADL1/lesson1-rxt50-CA.ipynb".
The first task is:
Hell yeah. Precomputed Activations kick ass. Just a few seconds..
... And now that fun ends:
Can you write Python code for it?

learn.activations[0].shape # ResNeXt50 uses FC layers? I thought it was fully Conv
# yup: check learn.summary(); after a flatten, 2 FC (aka Linear) layers. Flatten
# operation produces a 4096-long vector/tensor-thing.

# Oh that's interesting: the way learn's activations are structured: for
# train, valid, and test:
print("Train Activs: {}\nValid Activs: {}\n Test Activs: {}".format(
        learn.activations[0].shape, learn.activations[1].shape, learn.activations[2].shape))

learn.precompute=False
learn.fit(lrs=1e-2, n_cycle=2, cycle_len=1)


I am working on the file "FADL1/lesson1-rxt50-CA.ipynb".
The first task is:
Just looking at the fastai library source code while the above works:
class ConvLearner(Learner):
    def __init__(self, data, models, precompute=False, **kwargs):
        self.precompute = False
        super().__init__(data, models, **kwargs)
        self.crit = F.binary_cross_entropy if data.is_multi else F.nll_loss
        if data.is_reg: self.crit = F.l1_loss
        elif self.metrics is None:
            self.metrics = [accuracy_multi] if self.data.is_multi else [accuracy]
        if precompute: self.save_fc1()
        self.freeze()
        self.precompute = precompute
further below:
```
    def save_fc1(self):
        self.get_activations()
        act, val_act, test_act = self.activations
        m=self.models.top_model
        if len(self.activations[0])==0:
            predict_to_bcolz(m, self.data.fix_dl, act)
        if len(self.activations[1])==0:
            predict_to_bcolz(m, self.data.val_dl, val_act)
        if len(self.activations[2])==0:
            if self.data.test_dl: predict_to_bcolz(m, self.data.test_dl, test_act)
    self.fc_data = ImageClassifierData.from_arrays(self.data.path,
            (act, self.data.trn_y), (val_act, self.data.val_y), self.data.bs, classes=self.data.classes,
            test = test_act if self.data.test_dl else None, num_workers=8)

``
Wait so does this mean whenprecompute=Trueis specified, .. oh of course.. fuck, that's awesome: fastai automatically computes all the activations for the train, validation, AND test data set (if provided) whenprecompute` is set to True. I fucking love this library..
Can you write Python code for it?

len(data.aug_dl.dataset), bs * 250

len(data.fix_dl.dataset), bs * 2875


I am working on the file "FADL1/lesson1-rxt50-CA.ipynb".
The first task is:
NOTE: Had GPU MEM crashes even at bs=2.. However: restarting Kernel, and re-initializing a learner and loading weights: I can do a batch size of at least 6 (2500 MB).
Can you write Python code for it?

learn.fit(lrs=lr, n_cycle=3, cycle_len=1)

learn.save('RNx_224_all_50')

learn.load('RNx_224_all_50')

log_preds, y = learn.TTA()
accuracy(log_preds, y)


I am working on the file "FADL1/lesson1-rxt50-CA.ipynb".
The first task is:
Previous testing below:
Can you write Python code for it?


I am working on the file "FADL1/lesson1-rxt50-CA.ipynb".
The first task is:
See: https://github.com/fastai/fastai/issues/23 and https://github.com/pytorch/pytorch/issues/973
Something about max open file descriptors (fds) -- maybe bc new Archs in fastai make new files instead of folders? Anyway, to avoid the error 2 lines down: specify !ulimit -n 2048 or so. Hopefully a more permanent solution is available soon. Also the fix only seems to work in the terminal, before starting the Jupyter session.
Can you write Python code for it?

!ulimit -n

log_preds = learn.TTA(is_test=True)


I am working on the file "FADL1/lesson1-rxt50-CA.ipynb".
The first task is:
Submission
Submission format is id,label, with label being the predicted likelihood of being a dog. data.classes shows that 'dogs' is the 2nd category, so save the 2nd column of predictions to the submission file. Also the LogLoss eval metric judges strongly against total wrong answers (1 or 0 when the answer is NOT 1 or 0), so the predictions will be clipped to [0.05:0.95] as that gives better results.
Can you write Python code for it?

test_preds = np.exp(log_preds)

data.test_dl.dataset.fnames[:10]

learn.data.test_dl.dataset.fnames[:10]

preds = np.clip(test_preds[:,1], 0.05, 0.95)
ids   = [i[6:-4] for i in learn.data.test_dl.dataset.fnames]
submission = pd.DataFrame({'id':ids, 'label':preds})

SUBM = 'subm/'
submission.to_csv(PATH+SUBM+'submission_RNx_224_all_50.csv.gz', compression='gzip', index=False)
FileLink(PATH+SUBM+'submission_RNx_224_all_50.csv.gz')

temp = pd.read_feather(PATH+'results/'+'RNx_225_all_50_logpreds.feather')

temp['cats'][0]

log_preds[:,0][0]


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
An even better option for picking a validation set is using the exact same length of time period as the test set uses - this is implemented here:
Can you write Python code for it?

val_idx = np.flatnonzero(
    (df.index<=datetime.datetime(2014,9,17)) & (df.index>=datetime.datetime(2014,8,1)))

val_idx=[0]


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
Create datasets
In addition to the provided data, we will be using external datasets put together by participants in the Kaggle competition. You can download all of them here.
For completeness, the implementation used to put them together is included below.
Can you write Python code for it?

def concat_csvs(dirname):
    path = f'{PATH}{dirname}'
    filenames=glob.glob(f"{path}/*.csv")

    wrote_header = False
    with open(f"{path}.csv","w") as outputfile:
        for filename in filenames:
            name = filename.split(".")[0]
            with open(filename) as f:
                line = f.readline()
                if not wrote_header:
                    wrote_header = True
                    outputfile.write("file,"+line)
                for line in f:
                     outputfile.write(name + "," + line)
                outputfile.write("\n")

# concat_csvs('googletrend')
# concat_csvs('weather')


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
Data Cleaning / Feature Engineering
As a structured data problem, we necessarily have to go through all the cleaning and feature engineering, even though we're using a neural network.
Can you write Python code for it?

train, store, store_states, state_names, googletrend, weather, test = tables

len(train),len(test)


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
Feature Space:
* train: Training set provided by competition
* store: List of stores
* store_states: mapping of store to the German state they are in
* List of German state names
* googletrend: trend of certain google keywords over time, found by users to correlate well w/ given data
* weather: weather
* test: testing set
Can you write Python code for it?

table_names = ['train', 'store', 'store_states', 'state_names',
               'googletrend', 'weather', 'test']


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
It's usually a good idea to back up large tables of extracted / wrangled features before you join them onto another one, that way you can go back to it easily if you need to make changes to it.
Can you write Python code for it?

df.to_feather(f'{PATH}df')

df = pd.read_feather(f'{PATH}df', index_col=0)

df["Date"] = pd.to_datetime(df.Date)

df.columns

joined = join_df(joined, df, ['Store', 'Date'])

joined_test = join_df(joined_test, df, ['Store', 'Date'])


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
Join weather/state names.
Can you write Python code for it?

weather = join_df(weather, state_names, "file", "StateName")


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
Next we want to drop the Store indices grouped together in the window function.
Often in pandas, there is an option to do this in place. This is time and memory efficient when working with large datasets.
Can you write Python code for it?

bwd.drop('Store',1,inplace=True)
bwd.reset_index(inplace=True)

fwd.drop('Store',1,inplace=True)
fwd.reset_index(inplace=True)

df.reset_index(inplace=True)


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
Next we'll extract features "CompetitionOpenSince" and "CompetitionDaysOpen". Note the use of apply() in mapping a function across dataframe values.
Can you write Python code for it?

for df in (joined,joined_test):
    df["CompetitionOpenSince"] = pd.to_datetime(dict(year=df.CompetitionOpenSinceYear,
                                                     month=df.CompetitionOpenSinceMonth, day=15))
    df["CompetitionDaysOpen"] = df.Date.subtract(df.CompetitionOpenSince).dt.days


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
Next we'll fill in missing values to avoid complications with NA's. NA (not available) is how Pandas indicates missing values; many models have problems when missing values are present, so it's always important to think about how to deal with them. In these cases, we are picking an arbitrary signal value that doesn't otherwise appear in the data.
Can you write Python code for it?

for df in (joined,joined_test):
    df['CompetitionOpenSinceYear'] = df.CompetitionOpenSinceYear.fillna(1900).astype(np.int32)
    df['CompetitionOpenSinceMonth'] = df.CompetitionOpenSinceMonth.fillna(1).astype(np.int32)
    df['Promo2SinceYear'] = df.Promo2SinceYear.fillna(1900).astype(np.int32)
    df['Promo2SinceWeek'] = df.Promo2SinceWeek.fillna(1).astype(np.int32)


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
Now that we've engineered all our features, we need to convert to input compatible with a neural network.
This includes converting categorical variables into contiguous integers or one-hot encodings, normalizing continuous features to standard normal, etc...
Can you write Python code for it?

cat_vars = ['Store', 'DayOfWeek', 'Year', 'Month', 'Day', 'StateHoliday', 'CompetitionMonthsOpen',
    'Promo2Weeks', 'StoreType', 'Assortment', 'PromoInterval', 'CompetitionOpenSinceYear', 'Promo2SinceYear',
    'State', 'Week', 'Events', 'Promo_fw', 'Promo_bw', 'StateHoliday_fw', 'StateHoliday_bw',
    'SchoolHoliday_fw', 'SchoolHoliday_bw']

contin_vars = ['CompetitionDistance', 'Max_TemperatureC', 'Mean_TemperatureC', 'Min_TemperatureC',
   'Max_Humidity', 'Mean_Humidity', 'Min_Humidity', 'Max_Wind_SpeedKm_h',
   'Mean_Wind_SpeedKm_h', 'CloudCover', 'trend', 'trend_DE',
   'AfterStateHoliday', 'BeforeStateHoliday', 'Promo', 'SchoolHoliday']

n = len(joined); n

dep = 'Sales'
joined_test[dep] = 0
joined = joined[cat_vars+contin_vars+[dep, 'Date']].copy()
joined_test = joined_test[cat_vars+contin_vars+[dep, 'Date', 'Id']].copy()

for v in cat_vars: joined[v] = joined[v].astype('category').cat.as_ordered()

apply_cats(joined_test, joined)

for v in contin_vars:
    joined[v] = joined[v].astype('float32')
    joined_test[v] = joined_test[v].astype('float32')


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
Now we can outer join all of our data into a single dataframe. Recall that in outer joins everytime a value in the joining field on the left table does not have a corresponding value on the right table, the corresponding row in the new table has Null values for all right table fields. One way to check that all records are consistent and complete is to check for Null values post-join, as we do here.
Aside: Why note just do an inner join?
If you are assuming that all records are complete and match on the field you desire, an inner join will do the same thing as an outer join. However, in the event you are wrong or a mistake is made, an outer join followed by a null-check will catch it. (Comparing before/after # of rows for inner join is equivalent, but requires keeping track of before/after row #'s. Outer join is easier.)
Can you write Python code for it?

store = join_df(store, store_states, "Store")
len(store[store.State.isnull()])

joined = join_df(train, store, "Store")
joined_test = join_df(test, store, "Store")
len(joined[joined.StoreType.isnull()]),len(joined_test[joined_test.StoreType.isnull()])

joined = join_df(joined, googletrend, ["State","Year", "Week"])
joined_test = join_df(joined_test, googletrend, ["State","Year", "Week"])
len(joined[joined.trend.isnull()]),len(joined_test[joined_test.trend.isnull()])

joined = joined.merge(trend_de, 'left', ["Year", "Week"], suffixes=('', '_DE'))
joined_test = joined_test.merge(trend_de, 'left', ["Year", "Week"], suffixes=('', '_DE'))
len(joined[joined.trend_DE.isnull()]),len(joined_test[joined_test.trend_DE.isnull()])

joined = join_df(joined, weather, ["State","Date"])
joined_test = join_df(joined_test, weather, ["State","Date"])
len(joined[joined.Mean_TemperatureC.isnull()]),len(joined_test[joined_test.Mean_TemperatureC.isnull()])

for df in (joined, joined_test):
    for c in df.columns:
        if c.endswith('_y'):
            if c in df.columns: df.drop(c, inplace=True, axis=1)


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
Now we'll merge these values onto the df.
Can you write Python code for it?

df = df.merge(bwd, 'left', ['Date', 'Store'], suffixes=['', '_bw'])
df = df.merge(fwd, 'left', ['Date', 'Store'], suffixes=['', '_fw'])

df.drop(columns,1,inplace=True)

df.head()


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
Same process for Promo dates.
Can you write Python code for it?

for df in (joined,joined_test):
    df["Promo2Since"] = pd.to_datetime(df.apply(lambda x: Week(
        x.Promo2SinceYear, x.Promo2SinceWeek).monday(), axis=1).astype(pd.datetime))
    df["Promo2Days"] = df.Date.subtract(df["Promo2Since"]).dt.days

for df in (joined,joined_test):
    df.loc[df.Promo2Days<0, "Promo2Days"] = 0
    df.loc[df.Promo2SinceYear<1990, "Promo2Days"] = 0
    df["Promo2Weeks"] = df["Promo2Days"]//7
    df.loc[df.Promo2Weeks<0, "Promo2Weeks"] = 0
    df.loc[df.Promo2Weeks>25, "Promo2Weeks"] = 25
    df.Promo2Weeks.unique()

joined.to_feather(f'{PATH}joined')
joined_test.to_feather(f'{PATH}joined_test')


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
The authors also removed all instances where the store had zero sale / was closed. We speculate that this may have cost them a higher standing in the competition. One reason this may be the case is that a little exploratory data analysis reveals that there are often periods where stores are closed, typically for refurbishment. Before and after these periods, there are naturally spikes in sales that one might expect. By ommitting this data from their training, the authors gave up the ability to leverage information about these periods to predict this otherwise volatile behavior.
Can you write Python code for it?


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
Then set null values from elapsed field calculations to 0.
Can you write Python code for it?

columns = ['SchoolHoliday', 'StateHoliday', 'Promo']

for o in ['Before', 'After']:
    for p in columns:
        a = o+p
        df[a] = df[a].fillna(0)


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
This is very representative of a typical industry dataset.
The following returns summarized aggregate information to each table accross each field.
Can you write Python code for it?

for t in tables: display(DataFrameSummary(t).summary())


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
To run on the full dataset, use this instead:
Can you write Python code for it?

samp_size = n
joined_samp = joined.set_index("Date")


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
We add "CompetitionMonthsOpen" field, limiting the maximum to 2 years to limit number of unique categories.
Can you write Python code for it?

for df in (joined,joined_test):
    df["CompetitionMonthsOpen"] = df["CompetitionDaysOpen"]//30
    df.loc[df.CompetitionMonthsOpen>24, "CompetitionMonthsOpen"] = 24
joined.CompetitionMonthsOpen.unique()


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
We can now process our data...
Can you write Python code for it?

joined_samp.head(2)

df, y, nas, mapper = proc_df(joined_samp, 'Sales', do_scale=True)
yl = np.log(y)

joined_test = joined_test.set_index("Date")

df_test, _, nas, mapper = proc_df(joined_test, 'Sales', do_scale=True, skip_flds=['Id'],
                                  mapper=mapper, na_dict=nas)

df.head(2)


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
We turn state Holidays to booleans, to make them more convenient for modeling. We can do calculations on pandas fields using notation very similar (often identical) to numpy.
Can you write Python code for it?

train.StateHoliday = train.StateHoliday!='0'
test.StateHoliday = test.StateHoliday!='0'


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
We'll be applying this to a subset of columns:
Can you write Python code for it?

columns = ["Date", "Store", "Promo", "StateHoliday", "SchoolHoliday"]

df = train[columns]

df = test[columns]


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
We'll be using the popular data manipulation framework pandas. Among other things, pandas allows you to manipulate tables/data frames in python as one would in a database.
We're going to go ahead and load all of our csv's as dataframes into the list tables.
Can you write Python code for it?

tables = [pd.read_csv(f'{PATH}{fname}.csv', low_memory=False) for fname in table_names]

from IPython.display import HTML


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
We'll do this for two more fields.
Can you write Python code for it?

fld = 'StateHoliday'
df = df.sort_values(['Store', 'Date'])
get_elapsed(fld, 'After')
df = df.sort_values(['Store', 'Date'], ascending=[True, False])
get_elapsed(fld, 'Before')

fld = 'Promo'
df = df.sort_values(['Store', 'Date'])
get_elapsed(fld, 'After')
df = df.sort_values(['Store', 'Date'], ascending=[True, False])
get_elapsed(fld, 'Before')


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
We'll replace some erroneous / outlying data.
Can you write Python code for it?

for df in (joined,joined_test):
    df.loc[df.CompetitionDaysOpen<0, "CompetitionDaysOpen"] = 0
    df.loc[df.CompetitionOpenSinceYear<1990, "CompetitionDaysOpen"] = 0


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
We're going to run on a sample.
Can you write Python code for it?

idxs = get_cv_idxs(n, val_pct=150000/n)
joined_samp = joined.iloc[idxs].set_index("Date")
samp_size = len(joined_samp); samp_size


I am working on the file "FADL1/lesson3-rossman-Copy1-old.ipynb".
The first task is:
join_df is a function for joining tables on specific fields. By default, we'll be doing a left outer join of right on the left argument using the given fields for each table.
Pandas does joins using the merge method. The suffixes argument describes the naming convention for duplicate fields. We've elected to leave the duplicate field names on the left untouched, and append a "_y" to those on the right.
Can you write Python code for it?

def join_df(left, right, left_on, right_on=None, suffix='_y'):
    if right_on is None: right_on = left_on
    return left.merge(right, how='left', left_on=left_on, right_on=right_on,
                      suffixes=("", suffix))


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
Create datasets
In addition to the provided data, we will be using external datasets put together by participants in the Kaggle competition. You can download all of them here.
For completeness, the implementation used to put them together is included below.
Can you write Python code for it?

def concat_csvs(dirname):
    path = f'{PATH}{dirname}'
    filenames=glob.glob(f"{path}/*.csv")

    wrote_header = False
    with open(f"{path}.csv","w") as outputfile:
        for filename in filenames:
            name = filename.split(".")[0]
            with open(filename) as f:
                line = f.readline()
                if not wrote_header:
                    wrote_header = True
                    outputfile.write("file,"+line)
                for line in f:
                     outputfile.write(name + "," + line)
                outputfile.write("\n")

# concat_csvs('googletrend')
# concat_csvs('weather')


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
Data Cleaning / Feature Engineering
As a structured data problem, we necessarily have to go through all the cleaning and feature engineering, even though we're using a neural network.
Can you write Python code for it?

train, store, store_states, state_names, googletrend, weather, test = tables

len(train),len(test)


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
Durations
It is common when working with time series data to extract data that explains relationships across rows as opposed to columns, e.g.:
* Running averages
* Time until next event
* Time since last event
This is often difficult to do with most table manipulation frameworks, since they are designed to work with relationships across columns. As such, we've created a class to handle this type of data.
We'll define a function get_elapsed for cumulative counting across a sorted dataframe. Given a particular field fld to monitor, this function will start tracking time since the last occurrence of that field. When the field is seen again, the counter is set to zero.
Upon initialization, this will result in datetime na's until the field is encountered. This is reset every time a new store is seen. We'll see how to use this shortly.
Can you write Python code for it?

def get_elapsed(fld, pre):
    day1 = np.timedelta64(1, 'D')
    last_date = np.datetime64()
    last_store = 0
    res = []

    for s,v,d in zip(df.Store.values,df[fld].values, df.Date.values):
        if s != last_store:
            last_date = np.datetime64()
            last_store = s
        if v: last_date = d
        res.append(((d-last_date).astype('timedelta64[D]') / day1).astype(int))
    df[pre+fld] = res


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
Feature Space:
* train: Training set provided by competition
* store: List of stores
* store_states: mapping of store to the German state they are in
* List of German state names
* googletrend: trend of certain google keywords over time, found by users to correlate well w/ given data
* weather: weather
* test: testing set
Can you write Python code for it?

table_names = ['train', 'store', 'store_states', 'state_names',
               'googletrend', 'weather', 'test']


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
In pandas you can add new columns to a dataframe by simply defining it. We'll do this for googletrends by extracting dates and state names from the given data and adding those columns.
We're also going to replace all instances of state name 'NI' to match the usage in the rest of the data: 'HB,NI'. This is a good opportunity to highlight pandas indexing. We can use .loc[rows, cols] to select a list of rows and a list of columns from the dataframe. In this case, we're selecting rows w/ statename 'NI' by using a boolean list googletrend.State=='NI' and selecting "State".
Can you write Python code for it?

googletrend['Date'] = googletrend.week.str.split(' - ', expand=True)[0]
googletrend['State'] = googletrend.file.str.split('_', expand=True)[2]
googletrend.loc[googletrend.State=='NI', "State"] = 'HB,NI'


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
Let's walk through an example.
Say we're looking at School Holiday. We'll first sort by Store, then Date, and then call add_elapsed('SchoolHoliday', 'After'):
This will apply to each row with School Holiday:
* A applied to every row of the dataframe in order of store and date
* Will add to the dataframe the days since seeing a School Holiday
* If we sort in the other direction, this will count the days until another holiday.
Can you write Python code for it?

fld = 'SchoolHoliday'
df = df.sort_values(['Store', 'Date'])
get_elapsed(fld, 'After')
df = df.sort_values(['Store', 'Date'], ascending=[True, False])
get_elapsed(fld, 'Before')


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
Next we want to drop the Store indices grouped together in the window function.
Often in pandas, there is an option to do this in place. This is time and memory efficient when working with large datasets.
Can you write Python code for it?

bwd.drop('Store',1,inplace=True)
bwd.reset_index(inplace=True)

fwd.drop('Store',1,inplace=True)
fwd.reset_index(inplace=True)

df.reset_index(inplace=True)


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
Next we'll demonstrate window functions in pandas to calculate rolling quantities.
Here we're sorting by date (sort_index()) and counting the number of events of interest (sum()) defined in columns in the following week (rolling()), grouped by Store (groupby()). We do the same in the opposite direction.
Can you write Python code for it?

bwd = df[['Store']+columns].sort_index().groupby("Store").rolling(7, min_periods=1).sum()

fwd = df[['Store']+columns].sort_index(ascending=False
                                      ).groupby("Store").rolling(7, min_periods=1).sum()


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
Now that we've engineered all our features, we need to convert to input compatible with a neural network.
This includes converting categorical variables into contiguous integers or one-hot encodings, normalizing continuous features to standard normal, etc...
Can you write Python code for it?

cat_vars = ['Store', 'DayOfWeek', 'Year', 'Month', 'Day', 'StateHoliday', 'CompetitionMonthsOpen',
    'Promo2Weeks', 'StoreType', 'Assortment', 'PromoInterval', 'CompetitionOpenSinceYear', 'Promo2SinceYear',
    'State', 'Week', 'Events', 'Promo_fw', 'Promo_bw', 'StateHoliday_fw', 'StateHoliday_bw',
    'SchoolHoliday_fw', 'SchoolHoliday_bw']

contin_vars = ['CompetitionDistance', 'Max_TemperatureC', 'Mean_TemperatureC', 'Min_TemperatureC',
   'Max_Humidity', 'Mean_Humidity', 'Min_Humidity', 'Max_Wind_SpeedKm_h',
   'Mean_Wind_SpeedKm_h', 'CloudCover', 'trend', 'trend_DE',
   'AfterStateHoliday', 'BeforeStateHoliday', 'Promo', 'SchoolHoliday']

n = len(joined); n

dep = 'Sales'
joined = joined[cat_vars+contin_vars+[dep, 'Date']].copy()

joined_test[dep] = 0
joined_test = joined_test[cat_vars+contin_vars+[dep, 'Date', 'Id']].copy()

for v in cat_vars: joined[v] = joined[v].astype('category').cat.as_ordered()

apply_cats(joined_test, joined)

for v in contin_vars:
    joined[v] = joined[v].astype('float32')
    joined_test[v] = joined_test[v].astype('float32')


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
Now we can outer join all of our data into a single dataframe. Recall that in outer joins everytime a value in the joining field on the left table does not have a corresponding value on the right table, the corresponding row in the new table has Null values for all right table fields. One way to check that all records are consistent and complete is to check for Null values post-join, as we do here.
Aside: Why note just do an inner join?
If you are assuming that all records are complete and match on the field you desire, an inner join will do the same thing as an outer join. However, in the event you are wrong or a mistake is made, an outer join followed by a null-check will catch it. (Comparing before/after # of rows for inner join is equivalent, but requires keeping track of before/after row #'s. Outer join is easier.)
Can you write Python code for it?

store = join_df(store, store_states, "Store")
len(store[store.State.isnull()])

joined = join_df(train, store, "Store")
joined_test = join_df(test, store, "Store")
len(joined[joined.StoreType.isnull()]),len(joined_test[joined_test.StoreType.isnull()])

joined = join_df(joined, googletrend, ["State","Year", "Week"])
joined_test = join_df(joined_test, googletrend, ["State","Year", "Week"])
len(joined[joined.trend.isnull()]),len(joined_test[joined_test.trend.isnull()])

joined = joined.merge(trend_de, 'left', ["Year", "Week"], suffixes=('', '_DE'))
joined_test = joined_test.merge(trend_de, 'left', ["Year", "Week"], suffixes=('', '_DE'))
len(joined[joined.trend_DE.isnull()]),len(joined_test[joined_test.trend_DE.isnull()])

joined = join_df(joined, weather, ["State","Date"])
joined_test = join_df(joined_test, weather, ["State","Date"])
len(joined[joined.Mean_TemperatureC.isnull()]),len(joined_test[joined_test.Mean_TemperatureC.isnull()])

for df in (joined, joined_test):
    for c in df.columns:
        if c.endswith('_y'):
            if c in df.columns: df.drop(c, inplace=True, axis=1)


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
RF
Can you write Python code for it?

from sklearn.ensemble import RandomForestRegressor

((val,trn), (y_val,y_trn)) = split_by_idx(val_idx, df.values, yl)

m = RandomForestRegressor(n_estimators=40, max_features=0.99, min_samples_leaf=2,
                          n_jobs=-1, oob_score=True)
m.fit(trn, y_trn);

preds = m.predict(val)
m.score(trn, y_trn), m.score(val, y_val), m.oob_score_, exp_rmspe(preds, y_val)


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
Same process for Promo dates.
Can you write Python code for it?

for df in (joined,joined_test):
    df["Promo2Since"] = pd.to_datetime(df.apply(lambda x: Week(
        x.Promo2SinceYear, x.Promo2SinceWeek).monday(), axis=1).astype(pd.datetime))
    df["Promo2Days"] = df.Date.subtract(df["Promo2Since"]).dt.days

for df in (joined,joined_test):
    df.loc[df.Promo2Days<0, "Promo2Days"] = 0
    df.loc[df.Promo2SinceYear<1990, "Promo2Days"] = 0
    df["Promo2Weeks"] = df["Promo2Days"]//7
    df.loc[df.Promo2Weeks<0, "Promo2Weeks"] = 0
    df.loc[df.Promo2Weeks>25, "Promo2Weeks"] = 25
    df.Promo2Weeks.unique()

joined.to_feather(f'{PATH}joined')
joined_test.to_feather(f'{PATH}joined_test')


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
Test
Can you write Python code for it?

m = md.get_learner(emb_szs, len(df.columns)-len(cat_vars),
                   0.04, 1, [1000,500], [0.001,0.01], y_range=y_range)
lr = 1e-3

m.fit(lr, 3, metrics=[exp_rmspe])

m.fit(lr, 3, metrics=[exp_rmspe], cycle_len=1)

m.save('val0')

m.load('val0')

x,y=m.predict_with_targs()

exp_rmspe(x,y)

pred_test=m.predict(True)

pred_test = np.exp(pred_test)

joined_test['Sales']=pred_test

csv_fn=f'{PATH}tmp/sub.csv'

joined_test[['Id','Sales']].to_csv(csv_fn, index=False)

FileLink(csv_fn)


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
The Google trends data has a special category for the whole of the US - we'll pull that out so we can use it explicitly.
Can you write Python code for it?

trend_de = googletrend[googletrend.file == 'Rossmann_DE']


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
Then set null values from elapsed field calculations to 0.
Can you write Python code for it?

columns = ['SchoolHoliday', 'StateHoliday', 'Promo']

for o in ['Before', 'After']:
    for p in columns:
        a = o+p
        df[a] = df[a].fillna(0)


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
This is very representative of a typical industry dataset.
The following returns summarized aggregate information to each table accross each field.
Can you write Python code for it?

for t in tables: display(DataFrameSummary(t).summary())


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
To run on the full dataset, use this instead:
Can you write Python code for it?

samp_size = n
joined_samp = joined.set_index("Date")


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
We add "CompetitionMonthsOpen" field, limiting the maximum to 2 years to limit number of unique categories.
Can you write Python code for it?

for df in (joined,joined_test):
    df["CompetitionMonthsOpen"] = df["CompetitionDaysOpen"]//30
    df.loc[df.CompetitionMonthsOpen>24, "CompetitionMonthsOpen"] = 24
joined.CompetitionMonthsOpen.unique()


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
We can create a ModelData object directly from out data frame.
Can you write Python code for it?

md = ColumnarModelData.from_data_frame(PATH, val_idx, df, yl.astype(np.float32), cat_flds=cat_vars, bs=128,
                                       test_df=df_test)


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
We now have our final set of engineered features.
While these steps were explicitly outlined in the paper, these are all fairly typical feature engineering steps for dealing with time series data and are practical in any similar setting.
Create features
Can you write Python code for it?

joined = pd.read_feather(f'{PATH}joined')
joined_test = pd.read_feather(f'{PATH}joined_test')

joined.head().T.head(40)


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
We turn state Holidays to booleans, to make them more convenient for modeling. We can do calculations on pandas fields using notation very similar (often identical) to numpy.
Can you write Python code for it?

train.StateHoliday = train.StateHoliday!='0'
test.StateHoliday = test.StateHoliday!='0'


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
We'll be applying this to a subset of columns:
Can you write Python code for it?

columns = ["Date", "Store", "Promo", "StateHoliday", "SchoolHoliday"]

df = train[columns]

df = test[columns]


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
We'll do this for two more fields.
Can you write Python code for it?

fld = 'StateHoliday'
df = df.sort_values(['Store', 'Date'])
get_elapsed(fld, 'After')
df = df.sort_values(['Store', 'Date'], ascending=[True, False])
get_elapsed(fld, 'Before')

fld = 'Promo'
df = df.sort_values(['Store', 'Date'])
get_elapsed(fld, 'After')
df = df.sort_values(['Store', 'Date'], ascending=[True, False])
get_elapsed(fld, 'Before')


I am working on the file "FADL1/lesson3-rossman-Copy2.ipynb".
The first task is:
We're going to set the active index to Date.
Can you write Python code for it?


I am working on the file "FADL1/lesson6-rnn_codealong.ipynb".
The first task is:
2. Three char model
2.1 Create inputs

Create a list of every 4th character, starting at the 0th, 1st, 2nd, then 3rd characters.
Can you write Python code for it?

cs = 3
c1_dat = [idx[i]   for i in range(0, len(idx)-cs, cs)] # every 1st char
c2_dat = [idx[i+1] for i in range(0, len(idx)-cs, cs)] # every 2nd
c3_dat = [idx[i+2] for i in range(0, len(idx)-cs, cs)] # every 3rd
c4_dat = [idx[i+3] for i in range(0, len(idx)-cs, cs)] # every 4th


I am working on the file "FADL1/lesson6-rnn_codealong.ipynb".
The first task is:
2.2 Create and train model

Pick a size for our hidden state
Can you write Python code for it?


I am working on the file "FADL1/lesson6-rnn_codealong.ipynb".
The first task is:
2.3 Test model
Can you write Python code for it?

def get_next(inp):
    """
    Takes a 3-char string.
    Turns it into a Tensor of an array of the char index of the string.
    Passes that tensor to the model.
    Does an argmax to get the predicted char-number; then coverts to char.
    """
    idxs = T(np.array([char_indices[c] for c in inp]))
#     pred = model(*idxs)
    pred = model(*VV(idxs))
    i = np.argmax(to_np(pred))
    return chars[i]

get_next('y. '), get_next('ppl'), get_next(' th'), get_next('and')


I am working on the file "FADL1/lesson6-rnn_codealong.ipynb".
The first task is:
3. Our first RNN
Lecture 6
3.1 Create inputs

This is the size of our unrolled RNN:
Can you write Python code for it?


I am working on the file "FADL1/lesson6-rnn_codealong.ipynb".
The first task is:
3.2 Create and train model
Can you write Python code for it?

val_idx = get_cv_idxs(len(idx) - cs - 1)

mdata = ColumnarModelData.from_arrays('.', val_idx, xs, y, bs=512)

class CharLoopModel(nn.Module):
    """This is an RNN."""
    def __init__(self, vocab_size, n_fac):
        super().__init__()
        self.e = nn.Embedding(vocab_size, n_fac)
        self.l_in = nn.Linear(n_fac, n_hidden)
        self.l_hidden = nn.Linear(n_hidden, n_hidden)
        self.l_out = nn.Linear(n_hidden, vocab_size)

    def forward(self, *cs):
        bs = cs[0].size(0)
#         h  = torch.zeros(bs, n_hidden).cuda()
        h = V(torch.zeros(bs, n_hidden).cuda())
        for c in cs:
#             inp = torch.tanh(self.l_in(self.e(c)))   # the torch.tanh vs F.tanh warning didnt pop
#             h   = torch.tanh(self.l_hidden(h + inp)) # up on Mac, but did on Linux-gpu. Odd.
            inp = F.relu(self.l_in(self.e(c)))
            h   = F.tanh(self.l_hidden(h+inp))

        return F.log_softmax(self.l_out(h), dim=-1)

model = CharLoopModel(vocab_size, n_fac).cuda()
optimizer = optim.Adam(model.parameters(), 1e-2)

fit(model, mdata, 1, optimizer, F.nll_loss)

set_lrs(optimizer, 1e-3)
fit(model, mdata, 1, optimizer, F.nll_loss)


I am working on the file "FADL1/lesson6-rnn_codealong.ipynb".
The first task is:
3.3 Test Model
Can you write Python code for it?

if '0.3' in torch.__version__:
    def get_next(inp):
        idxs = T(np.array([char_indices[c] for c in inp]))
        p = model(*VV(idxs))
        i = np.argmax(to_np(p))
        return chars[i]
else:
    def get_next(inp):
    #     idxs = [T(np.array([char_indices[c] for c in inp]))]
        idxs = [T(np.array([char_indices[c]])) for c in inp]
        p = model(*idxs)
        i = np.argmax(to_np(p))
    #     pdb.set_trace()
        return chars[i]

get_next('for thos')

get_next('part of ')

get_next('queens a')


I am working on the file "FADL1/lesson6-rnn_codealong.ipynb".
The first task is:
4.1 Test model
Can you write Python code for it?

def get_next(inp):
    idxs = T(np.array([char_indices[c] for c in inp]))
    p = model(*VV(idxs))
    i = np.argmax(to_np(p))
    return chars[i]

get_next('for thos')

def get_next_n(inp, n):
    res = inp
    for i in range(n):
        c = get_next(inp)
        res += c
        inp = inp[1:] + c
    return res

get_next_n('for thos', 40)


I am working on the file "FADL1/lesson6-rnn_codealong.ipynb".
The first task is:
5. Multi-output model
5.1 Setup
Lecture 1:58:07

Let's take non-overlapping sets of characters this time.
Can you write Python code for it?

c_in_dat = [[idx[i+j] for i in range(cs)] for j in range(0, len(idx) - cs - 1, cs)]


I am working on the file "FADL1/lesson6-rnn_codealong.ipynb".
The first task is:
6. Stateful model
Lecture 7
Can you write Python code for it?

from torchtext import vocab, data

from fastai.nlp import *
from fastai.lm_rnn import *

PATH = 'data/nietzsche/'
TRN_PATH = 'trn/'
VAL_PATH = 'val/'
TRN = f'{PATH}{TRN_PATH}'
VAL = f'{PATH}{VAL_PATH}'

## line counting: https://stackoverflow.com/a/3137099
# $ wc -l nietzsche/nietzsche.txt
## splitting: https://stackoverflow.com/a/2016918
# $ split -l 7947 nietzsche/nietzsche.txt
# $ mv xaa nietzsche/trn.txt
# $ mv xab nietzsche/val.txt

%ls {PATH}

%ls {PATH}trn

TEXT = data.Field(lower=True, tokenize=list) # torchtext
bs = 64; bptt = 8; n_fac = 42; n_hidden = 256

FILES = dict(train=TRN_PATH, validation=VAL_PATH, test=VAL_PATH)
mdata = LanguageModelData.from_text_files(PATH, TEXT, **FILES, bs=bs, bptt=bptt, min_freq=3)

len(mdata.trn_dl), mdata.nt, len(mdata.trn_ds), len(mdata.trn_ds[0].text)


I am working on the file "FADL1/lesson6-rnn_codealong.ipynb".
The first task is:
6.3 RNN loop
Can you write Python code for it?

# # From pytorch source:
# def RNNCell(input, hidden, w_ih, w_hh, b_ih, b_hh):
#     return F.tanh(F.linear(input, w_ih, b_ih) + F.linear(hidden, w_hh, b_hh))

class CharSeqStatefulRNN2(nn.Module):
    def __init__(self, vocab_size, n_fac, bs):
        super().__init__()
        self.vocab_size = vocab_size
        self.e = nn.Embedding(vocab_size, n_fac)
        self.rnn = nn.RNNCell(n_fac, n_hidden)
        self.l_out = nn.Linear(n_hidden, vocab_size)
        self.init_hidden(bs)

    def forward(self, cs):
        bs = cs[0].size(0)
        if self.h.size(1) != bs: self.init_hidden(bs)
        outp = []
        o = self.h
        for c in cs:
            o = self.rnn(self.e(c), o)
            outp.append(o)
        outp = self.l_out(torch.stack(outp))
        self.h = repackage_var(o)
        return F.log_softmax(outp, dim=-1).view(-1, self.vocab_size)

    def init_hidden(self, bs): self.h = V(torch.zeros(1, bs, n_hidden))

m = CharSeqStatefulRNN2(mdata.nt, n_fac, 512).cuda()
opt = optim.Adam(m.parameters(), 1e-3)

fit(m, mdata, 4, opt, F.nll_loss)


I am working on the file "FADL1/lesson6-rnn_codealong.ipynb".
The first task is:
Our outputs:
Can you write Python code for it?


I am working on the file "FADL1/lesson6-rnn_codealong.ipynb".
The first task is:
So each column below is one series of 8 characters from the text.
Can you write Python code for it?


I am working on the file "FADL1/lesson6-rnn_codealong.ipynb".
The first task is:
The first 4 inputs and outputs:
Can you write Python code for it?


I am working on the file "FADL1/lesson6-rnn_codealong.ipynb".
The first task is:
The input and hidden states represent qualitatively different types of information, so adding them together can potentially lose information. Instead we can concatenate them together.
Can you write Python code for it?

class CharLoopConcatModel(nn.Module):
    def __init__(self, vocab_size, n_fac):
        super().__init__()
        self.e = nn.Embedding(vocab_size, n_fac)
        self.l_in = nn.Linear(n_fac + n_hidden, n_hidden)
        self.l_hidden = nn.Linear(n_hidden, n_hidden)
        self.l_out = nn.Linear(n_hidden, vocab_size)

    def forward(self, *cs):
        bs = cs[0].size(0)
#         h  = torch.zeros(bs, n_hidden).cuda()
        h = V(torch.zeros(bs, n_hidden).cuda())
        for c in cs:
            inp = torch.cat((h, self.e(c)), 1)
            inp = F.relu(self.l_in(inp))
#             h   = torch.tanh(self.l_hidden(inp))
            h   = F.tanh(self.l_hidden(inp))

        return F.log_softmax(self.l_out(inp), dim=-1)

model = CharLoopConcatModel(vocab_size, n_fac).cuda()
optimizer = optim.Adam(model.parameters(), 1e-3)

it = iter(mdata.trn_dl)
*xs,yt = next(it)
# t = model(*xs)
t = model(*V(xs))

xs[0].size(0)

t

fit(model, mdata, 1, optimizer, F.nll_loss)

set_lrs(optimizer, 1e-4)
fit(model, mdata, 1, optimizer, F.nll_loss)


I am working on the file "FADL1/lesson6-rnn_codealong.ipynb".
The first task is:
The number of latent factors to create (ie: size of the embedding matrix):
Can you write Python code for it?

n_fac = 42 # about half the number of our characters

'0.3' in torch.__version__

class Char3Model(nn.Module):
    def __init__(self, vocab_size, n_fac):
        super().__init__()
        self.e = nn.Embedding(vocab_size, n_fac) # embedding

        # the 'green arrow' from our diagram – the layer operation from input to hidden
        self.l_in = nn.Linear(n_fac, n_hidden)

        # the 'orange arrow' from our diagram – the layer operation from hidden to hidden
        self.l_hidden = nn.Linear(n_hidden, n_hidden)

        # the 'blue arrow' from our diagram – the layer operation from hidden to output
        self.l_out = nn.Linear(n_hidden, vocab_size)

    def forward(self, c1, c2, c3):
        in1 = F.relu(self.l_in(self.e(c1)))
        in2 = F.relu(self.l_in(self.e(c2)))
        in3 = F.relu(self.l_in(self.e(c3)))

        if '0.3' in torch.__version__:
            h = V(torch.zeros(in1.size()).cuda())
            h = F.tanh(self.l_hidden(h+in1))
            h = F.tanh(self.l_hidden(h+in2))
            h = F.tanh(self.l_hidden(h+in3))
        else:
            h = torch.zeros(in1.size()).cuda() # I dont think I have to wrap as Variable since this is pytorch 0.4, no?
            h = torch.tanh(self.l_hidden(h + in1))
            h = torch.tanh(self.l_hidden(h + in2))
            h = torch.tanh(self.l_hidden(h + in3))

        return F.log_softmax(self.l_out(h))

mdata = ColumnarModelData.from_arrays('.', [-1], np.stack([x1,x2,x3], axis=1), y, bs=512)
model = Char3Model(vocab_size, n_fac).cuda()

it = iter(mdata.trn_dl)
*xs,yt = next(it)
# tensor = model(*xs)
tensor = model(*V(xs))

optimizer = optim.Adam(model.parameters(), 1e-2)

set_lrs(optimizer, 1e-3)
fit(model, mdata, 1, optimizer, F.nll_loss)

set_lrs(optimizer, 1e-3)
fit(model, mdata, 1, optimizer, F.nll_loss)


I am working on the file "FADL1/lesson6-rnn_codealong.ipynb".
The first task is:
Then create the exact same thing, offset by 1, as our labels.
Can you write Python code for it?

c_out_dat = [[idx[i+j] for i in range(cs)] for j in range(1, len(idx) - cs, cs)]

xs = np.stack(c_in_dat)
xs.shape

ys = np.stack(c_out_dat)
ys.shape

xs[:cs, :cs]

ys[:cs, :cs]


I am working on the file "FADL1/lesson6-rnn_codealong.ipynb".
The first task is:
idx will be the data we use form now on – it simply converts all characters to their index (based on the mapping above).
Can you write Python code for it?

idx = [char_indices[c] for c in text]
idx[:10]

''.join(indices_char[i] for i in idx[:70])


I am working on the file "FADL1/quick_dogscats.ipynb".
The first task is:
Another way to create the submission csv file:
Can you write Python code for it?

df = pd.DataFrame(test_preds)  # NOTE: pretty sure this has to be [np.argmax(pred) for pred in test_preds]
df.columns = data.classes

# '6:' skips the 'test1/' part of the fname, ':-4' skips the '.jpg' part
df.insert(0, 'id', [o[6:-4] for o in data.test_ds.fnames])

df.head()


I am working on the file "FADL1/quick_dogscats.ipynb".
The first task is:
Misc; looking at stuff
Can you write Python code for it?

submission.head()

data.classes

# 1st 10 filenames in test dataset: (I fucking love fastai)
data.test_ds.fnames[:10]


I am working on the file "FADL1/quick_dogscats.ipynb".
The first task is:
NOTE: THIS WILL NOT WORK FOR THIS DATASET. Only because I'm lazy. The format for this datset's submissions is:

id,label
<id_num>,<0 or 1>
...,...

But this 2nd method is generally how you'll go about it for multi-class data; and even this datset if you make sure the format is correct.
Saving the CSV File & Submitting
Can you write Python code for it?

os.mkdir(PATH + '/results/')

%ls $PATH

submission.to_csv(PATH+'results/' + 'submission_dogscats_quick_00.csv')

SUBM = f'{PATH}subm/'
os.makedirs(SUBM, exist_ok=True)
# df.to_csv(f'{PATH}subm.gz', compression='gzip', index=False)
submission.to_csv(f'{SUBM}subm.gz', compression='gzip', index=False)

FileLink(f'{SUBM}subm.gz')


I am working on the file "FADL1/quick_dogscats.ipynb".
The first task is:
One way to create the submission csv file:
Can you write Python code for it?

# See: http://forums.fast.ai/t/dog-breed-identification-challenge/7464/101
ids = [i.split('.jpg')[0].split('/')[-1] for i in data.test_dl.dataset.fnames]

preds = [np.argmax(pred) for pred in test_preds]

preds[:10]d

submission = pd.DataFrame()
submission['id'] = [i for i in ids]
submission['label'] = [p for p in preds]

submission.head()


I am working on the file "FADL1/quick_dogscats.ipynb".
The first task is:
Up above, TTA() has is_test set to False by default, so it won't use the test set. Iguess it uses the valid set? To use the test set, specify is_test=True
NOTE: running TTA() on the test set wasn't working ('NoneType' TypeError) because the test_name par to from_paths(..) is by default set to None. Also the test dir in this dataset (from fast.ai) is called test1/, so has to be specfd as such or renamed.
Can you write Python code for it?

# Reloading saved weights after re-openning notebook;
# Also specifying test dataset for final predictions:
tfms = tfms_from_model(resnet50, sz, aug_tfms=transforms_side_on, max_zoom=1.1)
data = ImageClassifierData.from_paths(PATH, tfms=tfms, bs=bs, test_name='test1')
learn = ConvLearner.pretrained(resnet50, data)
learn.load('ResNet50_01')

%time log_preds, y = learn.TTA(is_test=True)

test_preds = np.exp(log_preds)

len(test_preds)

# id,label
# 1,0.5
# 2,0.5

data.test_ds.get_x(0)

test_preds[:10]


I am working on the file "FADL1/quick_dogscats.ipynb".
The first task is:
We didn't specify precompute=True because it only makes it a little faster for this first set. So we skipped it. It's a shortcut that caches some of the intermediate steps that don't have to be recalculated each time. NOTE: when we use precomputed activations: data augmentation will not work.
So if you specfy data augmentation, such as w/ aug_tfms=..., and also precompute=True fastai will not actually do any data augmentation.
NOTE: if you're using big / deep model like ResNet50 or ResNeXt101 on a dataset v.similar to ImageNet (like this cats/dogs datset; ie: side-on photos of standard objects of a similar size to ImageNet: 200~500 pxls), you should probably run learn.bn_freeze(True) after learn.unfreeze().
What this is doing is causing the BatchNorm moving averages to stop updating.
(Not currently supported by any library besides fastai, and is v.important)
Needed to restart kernel & set batch-size to 16 to fit it in Gfx Memory
rerun lines: tfms = ...; data = ...; learn = ...;
We run one more epoch, training the entire network:
Can you write Python code for it?

# we can unfreeze and this will train the entire network
learn.unfreeze()
# bn_freeze stops updates of
learn.bn_freeze(True) # important for large models --> stops updates of BN moving avgs
                      # use for models w/ ~ > 34 layers, & larger imgs & reg.s of intrst & norml imgs
%time learn.fit([1e-5, 1e-4, 1e-2], 1, cycle_len=1)

# We use Test Time Augmentation to ensure we get the best predictions we can
%time log_preds, y = learn.TTA()

# Finally, this gives us about 99.6% accuracy
metrics.log_loss(y, np.exp(log_preds)), accuracy(log_preds, y)


I am working on the file "FADL1/sgd_lesson1.ipynb".
The first task is:
Components of Learning:

Representation: space of allowed models
Exmpl: (1) Logistic Regression, (2) 3-Layer Feedforward Neural Network

Evaluation: how do you judge or prefer one model over another (loss function, utility function)
Optimization: how do you search the space of models to obtain better evaluation

Linear Regression problem:
the goal of linear regression is to fit a line to a set of points:
Can you write Python code for it?

# Generating fake data
def lin(a,b,x): return a*x+b

def gen_fake_data(n, a, b):
    x = s = np.random.uniform(0,1,n)
    y = lin(a,b,x) + 0.1 * np.random.normal(0,3,n)
    return x, y
x, y = gen_fake_data(50, 3., 8.)

plt.scatter(x,y, s=8); plt.xlabel("x"); plt.ylabel("y");


I am working on the file "FADL1/sgd_lesson1.ipynb".
The first task is:
Gradient Descent
Can you write Python code for it?

# generate some more data
x, y = gen_fake_data(10000, 3., 8.)

print(x.shape, y.shape)

dtype = torch.cuda.FloatTensor

# from NumPy to PyTorch
x = torch.from_numpy(x)
y = torch.from_numpy(y)

from torch.autograd import Variable
# Create random Tensors to hold x and y, and wrap them in Variables.
# Setting requires_grad=False indicates that we don't need to compute gradients
# wrt to these Variables.
x = Variable(x.type(dtype), requires_grad=False)
y = Variable(y.type(dtype), requires_grad=False)

# Create random Tensors for weights a and b, ard wrap them in Variables.
# Setting requires_grad+True indicates that we want to compute gradients wrt
# these Variables.
a = Variable(torch.randn(1).type(dtype), requires_grad=True)
b = Variable(torch.randn(1).type(dtype), requires_grad=True)

a.grad = Variable(dtype(a.size()).zero_())
b.grad = Variable(dtype(b.size()).zero_())

a, b

learning_rate = 1e-3
for t in range(10000):
    # Forward pass: copmute predicted y using operatiosn on Variables
    y_pred = x * a + b
    loss = (y_pred - y).pow(2).mean()
    if t % 1000 == 0: print(loss.data)

    # Manually zero the gradients
    a.grad.data.zero_()
    b.grad.data.zero_()

    # Computes the gradient of loss wrt all Variables with requires_grad=True
    # After this call a.grad and b.grad will be Variables holding the gradient
    # of the loss wrt a and b respectively.
    loss.backward()

    # Update a and b using gradient descent; a.data and b.data are Tensors,
    # a.grad and b.grad are Variables and a.grad.data and b.grad.data are
    # Tensors.
    a.data -= learning_rate * a.grad.data
    b.data -= learning_rate * b.grad.data

print(a, b)


I am working on the file "FADL1/sgd_lesson1.ipynb".
The first task is:
Stochasting Gradient Descent (SGD)
Visualizing Gradient Descent
Can you write Python code for it?

x, y = gen_fake_data(50, 3., 8.)

def sse(y,y_pred): return ((y-y_pred)**2).sum()
def loss(y,a,b,x): return sse(y, lin(a,b,x))
def avg_loss(y,a,b,x): return np.sqrt(loss(y,a,b,x)/y.shape[0])

a_guess = -1.
b_guess = 1.
avg_loss(y, a_guess, b_guess, x)

lr = 0.01
def upd():
    global a_guess, b_guess
    y_pred = lin(a_guess, b_guess, x)
    dydb = 2 * (y_pred - y)
    dyda = x * dydb
    a_guess -= lr * dyda.mean()
    b_guess -= lr * dydb.mean()

fig = plt.figure(dpi=100, figsize=(5,4))
plt.scatter(x,y)
line, = plt.plot(x, lin(a_guess, b_guess, x))
plt.close()

def animate(i):
    line.set_ydata(lin(a_guess, b_guess, x))
    for i in range(30): upd()
    return line,

ani = animation.FuncAnimation(fig, animate, np.arange(0, 40), interval=100)

ani


I am working on the file "FADL1/sgd_lesson1.ipynb".
The first task is:
You want to find parameters (weights) a and b such that you minimize the error btwn the points and the line a * x + b. Note that a and b are unknown. For a regression problem, the most common error function or loss function is the mean squared error.
Can you write Python code for it?

def mean_square_error(y_hat, y):
    return np.power(y_hat - y, 2).mean()

# say our prediction is a=10, b=5; computing our error:
y_hat = lin(10,5,x)
mean_square_error(y_hat, y)


I am working on the file "FADL1/vgg16_lesson1.ipynb".
The first task is:

Can you write Python code for it?

%reload_ext autoreload
%autoreload 2
%matplotlib inline

from fastai.imports import *

from fastai.transforms import *
from fastai.conv_learner import *
from fastai.model import *
from fastai.dataset import *
from fastai.sgdr import *
from fastai.plots import *

PATH = "data/dogscats/"
sz=224
ARCH = vgg16
bs = 16

data = ImageClassifierData.from_paths(PATH, bs=bs, tfms=tfms_from_model(ARCH, sz),
                                      test_name = 'test1')

# No reason to precompute activations as I'm running a single prediction run on the test set
# Also, since I trained all ConvLayers earlier... where will it know how to compute
# the activations if I haven't loaded the weights yet?
learn = ConvLearner.pretrained(ARCH, data, precompute=False)

learn.load('vgg16_04')

log_preds, y = learn.TTA()
accuracy(log_preds, y)

df = pd.read_feather(PATH + 'results/' + 'log_preds')

df.head()

preds = np.array(list(zip(*(df['dog'],df['cat']))))

preds = np.exp(preds)

preds = [np.argmax(pred) for pred in preds]

new_preds = [int(i==0) for i in preds]

new_preds[:10]

ids = [f[6:-4] for f in learn.data.test_dl.dataset.fnames]
submission = pd.DataFrame({'id': ids, 'label': new_preds})

submission.to_csv(PATH + 'subm/' + 'submission_vgg16_04_wtf.gz', compression='gzip', index=False)
FileLink(PATH + 'subm/' + 'submission_vgg16_04_wtf.gz')

log_preds_df = np.array(list(zip(*(df['dog'],df['cat']))))

test_preds = np.exp(log_preds_df)

test_preds = np.clip(test_preds, 0.05, 0.95)

data.classes

labels = test_preds[:,1]

labels[:10]

learn.data.test_dl.dataset.fnames

ids = [f[6:-4] for f in learn.data.test_dl.dataset.fnames]
submission = pd.DataFrame({'id': ids, 'label': labels})

submission.to_csv(PATH + 'subm/' + 'submission_vgg16_04_omg.csv.gz', compression='gzip', index=False)
FileLink(PATH + 'subm/' + 'submission_vgg16_04_omg.csv.gz')


I am working on the file "FADL1/vgg16_lesson1.ipynb".
The first task is:
Another way to create submission file:
Can you write Python code for it?

submission = pd.DataFrame(preds)
submission.columns = ['label']

submission.insert(0, 'id', ids)

submission.head()


I am working on the file "FADL1/vgg16_lesson1.ipynb".
The first task is:
Experimenting with saving as Pandas.DataFrame.to_feather(.)
Can you write Python code for it?

log_preds_df = pd.DataFrame(log_preds, columns=['dog','cat'])

log_preds_df.to_feather(PATH + 'results/' + 'log_preds')

# log_preds_df = pd.read_feather(PATH + 'results/' + 'log_preds')

test_preds = np.exp(log_preds)

ids = [f[6:-4] for f in learn.data.test_dl.dataset.fnames]

preds = [np.argmax(pred) for pred in test_preds]

submission = pd.DataFrame({'id': ids, 'label': preds})


I am working on the file "FADL2/CIFAR-10-Copy-Test.ipynb".
The first task is:

Can you write Python code for it?

f_model = resnet18
sz = 32

tfms = tfms_from_model(f_model, sz, aug_tfms=transforms_side_on, pad=sz//8)
model_data = ImageClassifierData.from_paths(PATH, bs=bs, tfms=tfms,
                                            val_name='test', num_workers=2)

# learner = ConvLearner.from_model_data(f_model, model_data) # throws AttributeError: 'function' object has no attribute 'children'

learner = ConvLearner.pretrained(f_model, model_data)

learner.crit

# learner.summary()

# learner.model.children


I am working on the file "FADL2/CIFAR-10-Copy-Test.ipynb".
The first task is:
Could num_classes be used for loss-choice logic?
Can you write Python code for it?


I am working on the file "FADL2/CIFAR-10-Copy-Test.ipynb".
The first task is:
Learner Initialization Tests
From this fast.ai forum thread. Looking at how the loss function is determined when a Learner is initialized from scratch or from a pretrained model.
NOTE: fastai.conv_learner edited to print out a couple things.
Can you write Python code for it?

sz = 64
from fastai.models import darknet
darknet53 = darknet.darknet_53()

tfms = tfms_from_stats(imagenet_stats, sz, aug_tfms=transforms_side_on, pad=sz//8)
model_data = ImageClassifierData.from_paths(PATH, bs=bs, tfms=tfms, val_name='test', num_workers=2)
learner = ConvLearner.from_model_data(darknet53, model_data)

learner.crit

darknet.Darknet()


I am working on the file "FADL2/CIFAR-10-Copy-Test.ipynb".
The first task is:
Viewing Data:
Can you write Python code for it?

data = get_data(32, 4)

x, y = next(iter(data.trn_dl))

plt.imshow(data.trn_ds.denorm(x)[0]);

plt.imshow(data.trn_ds.denorm(x)[1]);


I am working on the file "FADL2/CIFAR-10-Copy-Test.ipynb".
The first task is:
earlier vsn Darknet
from: https://github.com/sgugger/Deep-Learning/blob/master/Resnet%2050%20and%20Darknet%2053.ipynb
Can you write Python code for it?

class alt_ConvBN(nn.Module):
    # convolutional layer then BatchNorm
    def __init__(self, ch_in, ch_out, kernel_size=3, stride=1, padding=0):
        super().__init__()
        self.conv = nn.Conv2d(ch_in, ch_out, kernel_size=kernel_size, stride=stride,
                              padding=padding, bias=False)
        self.bn = nn.BatchNorm2d(ch_out, momentum=0.01)

    def forward(self, x):
        return F.leaky_relu(self.bn(self.conv(x)), negative_slope=0.1)

class alt_DarknetBlock(nn.Module):
    # the basic blocks
    def __init__(self, ch_in):
        super().__init__()
        ch_hid = ch_in//2
        self.conv1 = alt_ConvBN(ch_in, ch_hid, kernel_size=1, stride=1, padding=0)
        self.conv2 = alt_ConvBN(ch_hid, ch_in, kernel_size=3, stride=1, padding=1)

    def forward(self, x):
        out = self.conv1(x)
        out = self.conv2(out)
        return out + x

class alt_Darknet(nn.Module):
    # Replicates table 1 from the YOLOv3 paper
    def __init__(self, num_blocks, num_classes=1000):
        super().__init__()
        self.conv = alt_ConvBN(3, 32, kernel_size=3, stride=1, padding=1)
        self.layer1 = self.make_group_layer(32, num_blocks[0])
        self.layer2 = self.make_group_layer(64, num_blocks[1], stride=2)
        self.layer3 = self.make_group_layer(128,num_blocks[2], stride=2)
        self.layer4 = self.make_group_layer(256,num_blocks[3], stride=2)
        self.layer5 = self.make_group_layer(512,num_blocks[4], stride=2)
        self.linear = nn.Linear(1024, num_classes)

    def make_group_layer(self, ch_in, num_blocks, stride=1):
        layers = [alt_ConvBN(ch_in, ch_in*2, stride=stride)]
        for i in range(num_blocks):
            layers.append(alt_DarknetBlock(ch_in*2))
        return nn.Sequential(*layers)

    def forward(self, x):
        out = self.conv(x)
        out = self.layer1(out)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)
        out = self.layer5(out)
        out =...


I am working on the file "FADL2/darknet_imagenet_sample.ipynb".
The first task is:
Setup
Can you write Python code for it?

def reset_valset(path):
    path_val = path/'valid'
    path_trn = path/'train'

    if not os.path.exists(path_val):
        print('No validation directory to reset.')
        return

    for folder in path_val.iterdir():
        for f in folder.iterdir():
            os.rename(f, path_trn / str(f).split('valid/')[-1])

def create_valset(path, p=0.15, seed=0):
    np.random.seed(seed=seed)

    path_val = path/'valid'
    path_trn = path/'train'
    reset_valset(path)

    # move random p-percent selection from train/ to valid/
    for folder in path_trn.iterdir():
        os.makedirs(path_val/str(folder).split('train/')[-1], exist_ok=True)
        flist = list(folder.iterdir())
        n_move = int(np.round(len(flist) * p))
        fmoves = np.random.choice(flist, n_move, replace=False)

        for f in fmoves:
            os.rename(f, path_val / str(f).split('train/')[-1])

def count_files(path):
    count = 0
    for folder in path.iterdir():
        count += len(list(folder.glob('*')))
    return count

create_valset(PATH, p=0.2)
count_files(PATH_TRAIN), count_files(PATH/'valid')


I am working on the file "FADL2/darknet_imagenet_sample.ipynb".
The first task is:
Train
Can you write Python code for it?

learner.fit(lrs=1e-2, n_cycle=1, wds=wd, cycle_len=3, use_clr=(40, 10))

# learner.save('darknet53_imagenet_sample_00')
learner.load('darknet53_imagenet_sample_00')

learner.fit(lrs=1e-2, n_cycle=1, wds=wd, cycle_len=3, use_clr=(40, 10))

learner.fit(lrs=1e-2, n_cycle=1, wds=wd, cycle_len=10, use_clr=(40, 10))

learner.save('darknet53_imagenet_sample_01')


I am working on the file "FADL2/darknet_loss_PR.ipynb".
The first task is:
'Strange/Normal' behavior:
ConvLearner.pretrained will only accept model definitions, not models themselves:
Can you write Python code for it?

learner = ConvLearner.pretrained(resnet18, model_data)

learner = ConvLearner.pretrained(resnet18(num_classes=10), model_data)


I am working on the file "FADL2/darknet_loss_PR.ipynb".
The first task is:
A version of darknet.py with the proposed changes above is used.
Can you write Python code for it?


I am working on the file "FADL2/darknet_loss_PR.ipynb".
The first task is:
Current Darknet, and Proposed Changes:
NOTE: from .layers import * changed to from fastai.layers import *, preventing ModuleNotFoundError.
Can you write Python code for it?

import torch
import torch.nn as nn
import torch.nn.functional as F
from fastai.layers import *  ###  <<<------

class ConvBN(nn.Module):
    "convolutional layer then batchnorm"

    def __init__(self, ch_in, ch_out, kernel_size = 3, stride=1, padding=0):
        super().__init__()
        self.conv = nn.Conv2d(ch_in, ch_out, kernel_size=kernel_size, stride=stride, padding=padding, bias=False)
        self.bn = nn.BatchNorm2d(ch_out, momentum=0.01)
        self.relu = nn.LeakyReLU(0.1, inplace=True)

    def forward(self, x): return self.relu(self.bn(self.conv(x)))

class DarknetBlock(nn.Module):
    def __init__(self, ch_in):
        super().__init__()
        ch_hid = ch_in//2
        self.conv1 = ConvBN(ch_in, ch_hid, kernel_size=1, stride=1, padding=0)
        self.conv2 = ConvBN(ch_hid, ch_in, kernel_size=3, stride=1, padding=1)

    def forward(self, x): return self.conv2(self.conv1(x)) + x

class Darknet(nn.Module):
    "Replicates the darknet classifier from the YOLOv3 paper (table 1)"

    def make_group_layer(self, ch_in, num_blocks, stride=1):
        layers = [ConvBN(ch_in,ch_in*2,stride=stride)]
        for i in range(num_blocks): layers.append(DarknetBlock(ch_in*2))
        return layers

    def __init__(self, num_blocks, num_classes=1000, start_nf=32):
        super().__init__()
        nf = start_nf
        layers = [ConvBN(3, nf, kernel_size=3, stride=1, padding=1)]
        for i,nb in enumerate(num_blocks):
            layers += self.make_group_layer(nf, nb, stride=(1 if i==1 else 2))
            nf *= 2
        layers += [nn.AdaptiveAvgPool2d(1), Flatten(), nn.Linear(nf, num_classes)]
        self.layers = nn.Sequential(*layers)

    def forward(self, x): return self.layers(x)

########################
### Proposed Version ###
class PR_Darknet(nn.Module):
    "Replicates the darknet classifier from the YOLOv3 paper (table 1)"

    def make_group_layer(self, ch_in, num_blocks, stride=1):
        layers = [ConvBN(ch_in,ch_in*2,stride=stride)]
        for i in range(num_blocks):...


I am working on the file "FADL2/darknet_loss_PR.ipynb".
The first task is:
Darknet with Cross Entropy loss
Can you write Python code for it?

f_model = darknet_53()

learner = ConvLearner.from_model_data(f_model, model_data, crit=F.cross_entropy)

learner.crit


I am working on the file "FADL2/darknet_loss_PR.ipynb".
The first task is:
However cross_entropy is NLL(LogSoftmax). The final layer is a LogSoftmax activation. The NLL criterion applied to its output will produce a Cross Entropy loss function.
Can you write Python code for it?

display_head(learner)

learner.lr_find()
learner.sched.plot()


I am working on the file "FADL2/darknet_loss_PR.ipynb".
The first task is:
However the current version of Darknet is not accepted by ConvLearner.pretrained at all. This makes sense, given that the model is not yet pretrained, but also suggests further work is needed to integrate the model into the library.
Can you write Python code for it?

learner = ConvLearner.pretrained(darknet.darknet_53, model_data)

learner = ConvLearner.pretrained(darknet.darknet_53(num_classes=10), model_data)


I am working on the file "FADL2/darknet_loss_PR.ipynb".
The first task is:
In this case the Learning Rate Finder 'fails' due to very small - often negative - loss values.
Can you write Python code for it?


I am working on the file "FADL2/darknet_loss_PR.ipynb".
The first task is:
Sometimes the LR Finder manages to produce a plot, but the results leave much to be desired:
Can you write Python code for it?


I am working on the file "FADL2/darknet_loss_PR.ipynb".
The first task is:
The from_model_data method works, as seen in section 1.

Misc
Can you write Python code for it?

# Use this version of `display_head` if the other is too finicky for you.
# NOTE: fastai learners other than darknet will have to be entered as:
#       [str(learner_or_model).split('\n')]

def display_head(fastai_learner, λ_name=None):
    """displays final conv block and network head."""
    n = len(fastai_learner[0])
    if λ_name == None: λ_name='DarknetBlock'
    j = 1
    # find final conv block
    for i in range(n):
        if λ_name in str(fastai_learner[0][-j]): break
        j += 1
    # print head & 'neck'
    for i in range(j):
        print(fastai_learner[0][i-j])

# display_head(learner, λ_name='BasicBlock')
display_head(learner1) #darknet learner
print('--------')
display_head([str(learner2).split('\n')], λ_name='BasicBlock') #resnet learner
print('--------')
display_head([str(f_model).split('\n')], λ_name='BasicBlock') #resnet model


I am working on the file "FADL2/darknet_loss_PR.ipynb".
The first task is:
There is no final activation layer. The criterion will be applied to the final layer's output:
Can you write Python code for it?


I am working on the file "FADL2/darknet_loss_PR.ipynb".
The first task is:
fastai.conv_learner logic sets criterion to torch.nn.functional.nll_loss:
Can you write Python code for it?


I am working on the file "FADL2/darknet_loss_PR.ipynb".
The first task is:
the fastai library adds the necessary LogSoftmax layer to the end of the model
NOTE: default constructor for resnet18 & darknet is 1000 classes (ImageNet). fastai lib finds the correct num_classes from the ModelData object. That's why the resnet18 model above has 1000 output features, and the resnet18 learner below it has the correct 10.
The criterion, the loss function, of the learner is still F.nll_loss:
Can you write Python code for it?


I am working on the file "FADL2/darknet_loss_troubleshooting.ipynb".
The first task is:
.from_model_data
Can you write Python code for it?

learner = ConvLearner.from_model_data(darknet53(), model_data)

# learner.summary()


I am working on the file "FADL2/darknet_loss_troubleshooting.ipynb".
The first task is:
1 linear layer at the end.
Can you write Python code for it?

from fastai.conv_learner import *

resnet18


I am working on the file "FADL2/darknet_loss_troubleshooting.ipynb".
The first task is:
1.
Can you write Python code for it?

from fastai.conv_learner import *
from fastai.models import darknet
from pathlib import Path

PATH = Path('data/imagenet')
PATH_TRAIN = PATH/'train'

# sz = 256
# bs = 32
# darknet53 = darknet.darknet_53()
# tfms = tfms_from_model(darknet53, sz)
# model_data = ImageClassifierData.from_paths(PATH, bs=bs, tfms=tfms, val_name='train')
# learner = ConvLearner.from_model_data(darknet53, model_data)

learner.crit

model_data

learner.crit

learner.crit = F.cross_entropy

learner.crit


I am working on the file "FADL2/darknet_loss_troubleshooting.ipynb".
The first task is:
2.
Checking behavior of .pretrained in fastai/conv_learner.py. Looking at number of FC layers at the end of a stock PyTorch model, compared to loading through FastAI:
Can you write Python code for it?

from torchvision.models import resnet18

resnet18

resnet18 = resnet18()

# resnet18


I am working on the file "FADL2/darknet_loss_troubleshooting.ipynb".
The first task is:
3.
Can you write Python code for it?

%matplotlib inline
%reload_ext autoreload
%autoreload 2
from fastai.conv_learner import *
from fastai.models import darknet
from pathlib import Path
PATH = Path('data/imagenet')
PATH_TRAIN = PATH/'train'

bs = 32
sz = 256

tfms = tfms_from_stats(imagenet_stats, sz)
model_data = ImageClassifierData.from_paths(PATH, bs=bs, tfms=tfms)

darknet53 = darknet.darknet_53


I am working on the file "FADL2/darknet_loss_troubleshooting.ipynb".
The first task is:
What fastai originally imports is the resnet18 constructor function from torchvision.models.resnet. Good to know.
Can you write Python code for it?

learner = ConvLearner.from_model_data(resnet18(), model_data)

# learner

from pathlib import Path
PATH = Path('data/cifar10')

sz = 32
bs = 64
tfms = tfms_from_model(resnet18, sz)
model_data = ImageClassifierData.from_paths(PATH, bs, tfms=tfms, val_name='test')

learner = ConvLearner.pretrained(resnet18, model_data)

# learner


I am working on the file "FADL2/darknet_loss_troubleshooting.ipynb".
The first task is:
head:
('AdaptiveAvgPool2d-232',
              OrderedDict([('input_shape', [-1, 1024, 14, 14]),
                           ('output_shape', [-1, 1024, 1, 1]),
                           ('nb_params', 0)])),
             ('Flatten-233',
              OrderedDict([('input_shape', [-1, 1024, 1, 1]),
                           ('output_shape', [-1, 1024]),
                           ('nb_params', 0)])),
             ('Linear-234',
              OrderedDict([('input_shape', [-1, 1024]),
                           ('output_shape', [-1, 1000]),
                           ('trainable', True),
                           ('nb_params', 1025000)]))])
learner with NLL loss fails Learning-Rate Finder phase due to negative losses:
Can you write Python code for it?

learner.lr_find()
learner.sched.plot()

learner.crit


I am working on the file "FAFT_64-points_R2C/nbFAFT128_2D.ipynb".
The first task is:
$W$ TRANSFORM FROM AXES-0
After the transfom, f_gpu[:32, :] contains real values and f_gpu[32:, :] contains imaginary values. g33_gpu contains the 33th. complex values
Can you write Python code for it?

f33 = np.zeros( [1 ,64], dtype = np.complex64 )

# One gpu array.

f_gpu = gpuarray.to_gpu( np.ascontiguousarray( f , dtype = np.float32 ) )
f33_gpu = gpuarray.to_gpu( np.ascontiguousarray( f33 , dtype = np.complex64 ) )


I am working on the file "FAFT_64-points_R2C/nbFAFT128_2D.ipynb".
The first task is:
$W$ TRANSFORM FROM AXES-1
After the transfom, f_gpu[:, :32] contains real values and f_gpu[:, 32:] contains imaginary values. f33_gpu contains the 33th. complex values
Can you write Python code for it?

f33 = np.zeros( [64, 1], dtype = np.complex64 )

# One gpu array.

f_gpu = gpuarray.to_gpu( np.ascontiguousarray( f , dtype = np.float32 ) )
f33_gpu = gpuarray.to_gpu( np.ascontiguousarray( f33 , dtype = np.complex64 ) )