I am working on the file "Seminar4/bonus/Bonus-advanced-theano.ipynb".
The first task is:
How does it work?
if you're currently in classroom, chances are i am explaining this text wall right now
* 1 You define inputs f your future function;
* 2 You write a recipe for some transformation of inputs;
* 3 You compile it;
* You have just got a function!
* The gobbledegooky version: you define a function as symbolic computation graph.

There are two main kinвs of entities: "Inputs" and "Transformations"
Both can be numbers, vectors, matrices, tensors, etc.

Both can be integers, floats of booleans (uint8) of various size.

An input is a placeholder for function parameters.

N from example above

Transformations are the recipes for computing something given inputs and transformation

(T.arange(N)^2).sum() are 3 sequential transformations of N
Doubles all functions of numpy vector syntax
You can almost always go with replacing "np.function" with "T.function" aka "theano.tensor.function"
np.mean -> T.mean
np.arange -> T.arange
np.cumsum -> T.cumsum
and so on.
builtin operations also work that way
np.arange(10).mean() -> T.arange(10).mean()
Once upon a blue moon the functions have different names or locations (e.g. T.extra_ops)
Ask us or google it

Still confused? We gonna fix that.
Can you write Python code for it?

#Inputs
example_input_integer = T.scalar("scalar input",dtype='float32')

example_input_tensor = T.tensor4("four dimensional tensor input") #dtype = theano.config.floatX by default
#не бойся, тензор нам не пригодится

input_vector = T.vector("", dtype='int32') # vector of integers

#Transformations

#transofrmation: elementwise multiplication
double_the_vector = input_vector*2

#elementwise cosine
elementwise_cosine = T.cos(input_vector)

#difference between squared vector and vector itself
vector_squares = input_vector**2 - input_vector

#Practice time:
#create two vectors of size float32
my_vector = student.init_float32_vector()
my_vector2 = student.init_one_more_such_vector()

#Write a transformation(recipe):
#(vec1)*(vec2) / (sin(vec1) +1)
my_transformation = student.implementwhatwaswrittenabove()

print my_transformation
#it's okay it aint a number


I am working on the file "Seminar4/bonus/Bonus-advanced-theano.ipynb".
The first task is:
Logistic regression example
[ 4 points max]
Implement the regular logistic regression training algorithm
Tips:
* Weights fit in as a shared variable
* X and y are potential inputs
* Compile 2 functions:
 * train_function(X,y) - returns error and computes weights' new values (through updates)
 * predict_fun(X) - just computes probabilities ("y") given data
We shall train on a two-class MNIST dataset
* please note that target y are {0,1} and not {-1,1} as in some formulae
Can you write Python code for it?

from sklearn.datasets import load_digits
mnist = load_digits(2)

X,y = mnist.data, mnist.target

print "y [shape - %s]:"%(str(y.shape)),y[:10]

print "X [shape - %s]:"%(str(X.shape))
print X[:3]
print y[:10]

# inputs and shareds
shared_weights = <student.code_me()>
input_X = <student.code_me()>
input_y = <student.code_me()>

predicted_y = <predicted probabilities for input_X>
loss = <logistic loss (scalar, mean over sample)>

grad = <gradient of loss over model weights>

updates = {
    shared_weights: <new weights after gradient step>
}

train_function = <compile function that takes X and y, returns log loss and updates weights>
predict_function = <compile function that takes X and computes probabilities of y>

from sklearn.cross_validation import train_test_split
X_train,X_test,y_train,y_test = train_test_split(X,y)

from sklearn.metrics import roc_auc_score

for i in range(5):
    loss_i = train_function(X_train,y_train)
    print "loss at iter %i:%.4f"%(i,loss_i)
    print "train auc:",roc_auc_score(y_train,predict_function(X_train))
    print "test auc:",roc_auc_score(y_test,predict_function(X_test))

print "resulting weights:"
plt.imshow(shared_weights.get_value().reshape(8,-1))
plt.colorbar()


I am working on the file "Seminar4/bonus/Bonus-advanced-theano.ipynb".
The first task is:
Shared variables

The inputs and transformations only exist when function is called

Shared variables always stay in memory like global variables

Shared variables can be included into a symbolic graph
They can be set and evaluated using special methods
but they can't change value arbitrarily during symbolic graph computation

we'll cover that later;

Hint: such variables are a perfect place to store network parameters

e.g. weights or some metadata
Can you write Python code for it?

#creating shared variable
shared_vector_1 = theano.shared(np.ones(10,dtype='float64'))

#evaluating shared variable (outside symbolicd graph)
print "initial value",shared_vector_1.get_value()

# within symbolic graph you use them just as any other inout or transformation, not "get value" needed

#setting new value
shared_vector_1.set_value( np.arange(5) )

#getting that new value
print "new value", shared_vector_1.get_value()

#Note that the vector changed shape
#This is entirely allowed... unless your graph is hard-wired to work with some fixed shape


I am working on the file "Seminar4/bonus/Bonus-advanced-theano.ipynb".
The first task is:
T.grad - why theano matters

Theano can compute derivatives and gradients automatically
Derivatives are computed symbolically, not numerically

Limitations:
* You can only compute a gradient of a scalar transformation over one or several scalar or vector (or tensor) transformations or inputs.
* A transformation has to have float32 or float64 dtype throughout the whole computation graph
 * derivative over an integer has no mathematical sense
Can you write Python code for it?

my_scalar = T.scalar(name='input',dtype='float64')

scalar_squared = T.sum(my_scalar**2)

#a derivative of v_squared by my_vector
derivative = T.grad(scalar_squared,my_scalar)

fun = theano.function([my_scalar],scalar_squared)
grad = theano.function([my_scalar],derivative)

import matplotlib.pyplot as plt
%matplotlib inline

x = np.linspace(-3,3)
x_squared = map(fun,x)
x_squared_der = map(grad,x)

plt.plot(x, x_squared,label="x^2")
plt.plot(x, x_squared_der, label="derivative")
plt.legend()


I am working on the file "Seminar4/bonus/Bonus-advanced-theano.ipynb".
The first task is:
When debugging, one would generally want to reduce the computation complexity. For example, if you are about to feed neural network with 1000 samples batch, consider taking first 2.
If you really want to debug graph of high computation complexity, you could just as well compile it (e.g. with optimizer='fast_compile')

Do It Yourself
[2 points max]
Can you write Python code for it?

# Quest #1 - implement a function that computes a mean squared error of two input vectors
# Your function has to take 2 vectors and return a single number

<student.define_inputs_and_transformations()>

compute_mse =<student.compile_function()>

# Tests
from sklearn.metrics import mean_squared_error

for n in [1,5,10,10**3]:

    elems = [np.arange(n),np.arange(n,0,-1), np.zeros(n),
             np.ones(n),np.random.random(n),np.random.randint(100,size=n)]

    for el in elems:
        for el_2 in elems:
            true_mse = np.array(mean_squared_error(el,el_2))
            my_mse = compute_mse(el,el_2)
            if not np.allclose(true_mse,my_mse):
                print 'Wrong result:'
                print 'mse(%s,%s)'%(el,el_2)
                print "should be: %f, but your function returned %f"%(true_mse,my_mse)
                raise ValueError,"Что-то не так"

print "All tests passed"


I am working on the file "Seminar4/bonus/Bonus-advanced-theano.ipynb".
The first task is:
Why that rocks
Can you write Python code for it?

my_vector = T.vector('float64')

#Compute the gradient of the next weird function over my_scalar and my_vector
#warning! Trying to understand the meaning of that function may result in permanent brain damage

weird_psychotic_function = ((my_vector+my_scalar)**(1+T.var(my_vector)) +1./T.arcsinh(my_scalar)).mean()/(my_scalar**2 +1) + 0.01*T.sin(2*my_scalar**1.5)*(T.sum(my_vector)* my_scalar**2)*T.exp((my_scalar-4)**2)/(1+T.exp((my_scalar-4)**2))*(1.-(T.exp(-(my_scalar-4)**2))/(1+T.exp(-(my_scalar-4)**2)))**2

der_by_scalar,der_by_vector = <student.compute_grad_over_scalar_and_vector()>

compute_weird_function = theano.function([my_scalar,my_vector],weird_psychotic_function)
compute_der_by_scalar = theano.function([my_scalar,my_vector],der_by_scalar)

#Plotting your derivative
vector_0 = [1,2,3]

scalar_space = np.linspace(0,7)

y = [compute_weird_function(x,vector_0) for x in scalar_space]
plt.plot(scalar_space,y,label='function')
y_der_by_scalar = [compute_der_by_scalar(x,vector_0) for x in scalar_space]
plt.plot(scalar_space,y_der_by_scalar,label='derivative')
plt.grid();plt.legend()


I am working on the file "Seminar4/bonus/Bonus-advanced-theano.ipynb".
The first task is:
Your turn
Can you write Python code for it?

# Write a recipe (transformation) that computes an elementwise transformation of shared_vector and input_scalar
#Compile as a function of input_scalar

input_scalar = T.scalar('coefficient',dtype='float32')

scalar_times_shared = <student.write_recipe()>

shared_times_n = <student.compile_function()>

print "shared:", shared_vector_1.get_value()

print "shared_times_n(5)",shared_times_n(5)

print "shared_times_n(-0.5)",shared_times_n(-0.5)

#Changing value of vector 1 (output should change)
shared_vector_1.set_value([-1,0,1])
print "shared:", shared_vector_1.get_value()

print "shared_times_n(5)",shared_times_n(5)

print "shared_times_n(-0.5)",shared_times_n(-0.5)


I am working on the file "Seminar4/bonus/Bonus-advanced-theano.ipynb".
The first task is:
my1stNN
[basic part 4 points max]
Your ultimate task for this week is to build your first neural network [almost] from scratch and pure theano.
This time you will same digit recognition problem, but at a larger scale
* images are now 28x28
* 10 different digits
* 50k samples
Note that you are not required to build 152-layer monsters here. A 2-layer (one hidden, one output) NN should already have ive you an edge over logistic regression.
[bonus score]
If you've already beaten logistic regression with a two-layer net, but enthusiasm still ain't gone, you can try improving the test accuracy even further! The milestones would be 95%/97.5%/98.5% accuraсy on test set.
SPOILER!
At the end of the notebook you will find a few tips and frequently made mistakes. If you feel enough might to shoot yourself in the foot without external assistance, we encourage you to do so, but if you encounter any unsurpassable issues, please do look there before mailing us.
Can you write Python code for it?

from mnist import load_dataset

#[down]loading the original MNIST dataset.
#Please note that you should only train your NN on _train sample,
# _val can be used to evaluate out-of-sample error, compare models or perform early-stopping
# _test should be hidden under a rock untill final evaluation... But we both know it is near impossible to catch you evaluating on it.
X_train,y_train,X_val,y_val,X_test,y_test = load_dataset()

print X_train.shape,y_train.shape

plt.imshow(X_train[0,0])

<here you could just as well create computation graph>

<this may or may not be a good place to evaluating loss and updates>

<here one could compile all the required functions>

<this may be a perfect cell to write a training&evaluation loop in>

<predict & evaluate on test here, right? No cheating pls.>


I am working on the file "Seminar4/bonus/Bonus-advanced-theano.ipynb".
The first task is:
theano teaser
Doing the very same thing
Can you write Python code for it?

import theano
import theano.tensor as T

#I gonna be function parameter
N = T.scalar("a dimension",dtype='int32')

#i am a recipe on how to produce sum of squares of arange of N given N
result = (T.arange(N)**2).sum()

#Compiling the recipe of computing "result" given N
sum_function = theano.function(inputs = [N],outputs=result)

%%time
sum_function(10**8)


I am working on the file "Seminar5/Seminar5.ipynb".
The first task is:
If your implementation is correct, the number above will be small, because deprocess function is the inverse of preprocess function
Can you write Python code for it?

#load model weights
#vgg19.npz is available for download at
#https://yadi.sk/d/UQPXeM_GqEmGg
net = build_model()
params = np.load('vgg19.npz')['params']
for i in range(32,len(params)):
    params[i] = params[i].T
lasagne.layers.set_all_param_values(net.values(), params)

input_image = T.tensor4('input')
output = lasagne.layers.get_output(net['prob'], input_image)
prob = theano.function([input_image], output)


I am working on the file "Seminar5/Seminar5.ipynb".
The first task is:
In the cell below, you can test your preprocessing function on some sample images. If it is implemented correctly, albatross.jpg will be classified as albatross with 99.9% certainty, and with other pictures the network will produce mostly meaningful result.
You can notice that network output varies from run to run. This behaviour can be supressed with help of "deterministic" keyword in get_output function in the cell above.
Can you write Python code for it?

img = imread('sample_images/albatross.jpg')
plt.imshow(img)
plt.show()

p = prob(preprocess(img))

labels = p.ravel().argsort()[-1:-6:-1]
print 'top-5 classes are:'
for l in labels:
    print '%3f\t%s' % (p.ravel()[l], classes[l].split(',')[0])


I am working on the file "Seminar5/Seminar5.ipynb".
The first task is:
Now, use vgg19 network and your knowledge of machine learning to classify cats and dogs!
data: https://yadi.sk/d/m6ZO4BvWqEmR9
catsvsdogs/val/ validation images
catsvsdogs/val_labels.pickle labels for validation images, sorted by filename
catsvsdogs/test/ test images
You have to implement classification algorithm, tune it on validation images, save output of your algorithm on test images in form of pickled file, as shown below. Your results, as well as this notebook, have to be attached to your letter to rdlclass@yandex.ru
I expect classification accuracy >95%, or >90% at least
Cheating is not allowed
Can you write Python code for it?

def classify(img):
    if np.random.rand() > 0.5:
        return 'cat'
    else:
        return 'dog'

path = 'catsvsdogs/test/'
files = sorted(os.listdir(path))

labels = []

for f in files:
    img = imread(path + f)
    label = classify(img)
    labels.append(label)

pickle.dump(labels, open('test_labels.pickle', 'wb'))


I am working on the file "Seminar5/Seminar5.ipynb".
The first task is:
Visualizations
It is easy to visualize the weights of the first convolutional layer:
Can you write Python code for it?

w = net['conv1_1'].W.eval().copy()
w -= w.min()
w /= w.max()
plt.figure(figsize=(10, 10))
for i in range(8):
    for j in range(8):
        n = 8*j + i
        if n < 64:
            plt.subplot(8,8,n)
            plt.axis('off')
            plt.imshow(w[n,:,:,:].transpose((1,2,0)), interpolation='none')
plt.show()


I am working on the file "Seminar6/Seminar6.ipynb".
The first task is:
Other option would be to use precomuted RPN proposals. Fetch them from here and unpack to the ./data folder. The advantage of using them is that you only need 300 entities per image (instead of 2000) to get a good performance.
Pretrained base models
Finally, you need to download one of the pretrained NNs that we will use a starting point for our object detection pipeline. As we are using Lasagne, it's a good idea to check out Lasagne's ModelZoo. We suggest not to go crazy and use a moderately-sized model like caffe_reference (or even shallower). We all want to set the new state-of-the-art for Object Detection, but let's start small.
Write your modules
The most interesting part of the assignment is, of course, the custom model injection. As we already mentioned, the original code uses Caffe. We (the TAs) did our best to get rid of this nasty dependency. Now, spend some time comparing the code provided by us and the Faster R-CNN repo. The code makes heavy use of the configuration files available by calling from fast_rcnn.config import cfg. The cfg contains various parameters that govern the behaviour of the whole system. The standard configuration is defined in ./lib/fast_rcnn/config.py. In order to override it use ./experiments/cfgs/rcnn.yml.
You'll have to
  1. Add missing pieces to ./lib/fast_rcnn/train.py (this is where the main training loop is).
  2. Implement the network (./custom/net.py) and the solver (./custom/solver.py) class respecting the interface described in the source files. You are NOT required to code up the bounding-box regression.
  3. Repeat the steps above for two kinds of training regimes:
    - Fine-tuning of the top layer.
    - Fine-tuning of the whole network.
Custom classes
The main purpose of the Net class is to hold a symbolic representation of your neural network. This is where you load a pretrained NN optionally modifying its architecture or weights.
The Solver is used to define actual training procedure:
  1. Create an instance of Net
  2. Define a loss function
  3. Define an...
Can you write Python code for it?

# Do NOT run this cell. This serves just as an illustration.

from roi_data_layer.layer import RoIDataLayer

# Create an instance and run initial setup.
roi_data_layer = RoIDataLayer()
roi_data_layer.setup()

# This is important! You need to supply the roidb. This one is available in ./lib/fast_rcnn/train.py via SolverWrapper (look at the __init__).
# By the way, SolverWrapper.__init__ is the place where you should instantiate your solver.
roi_data_layer.set_roidb(roidb)

# This is what you'd call at every training iteration.
# This populates roi_data_layer.top list with with tensors that you might need for conducting a step of you solver.
roi_data_layer.forward()
# First three tensors are:
#   1. data (ndarray): (cfg.TRAIN.IMS_PER_BATCH, 3, image_height, image_width) tensor containing the whole scenes
#   2. rois (ndarray): (cfg.TRAIN.BATCH_SIZE, 5) tensor containg ROIs; rois[:, 0] are indices of scenes in data, the rest
#                      are (left, top, right, bottom) coordinates
#   3. labels (ndarray): (cfg.TRAIN.BATCH_SIZE,) tensor contaning correct labels (those are float32, convert if needed)
data, rois, labels = roi_data_layer.top[: 3]


I am working on the file "Seminar6/Seminar6.ipynb".
The first task is:
PASCAL VOC 2007
In our experiments, we are going to use PASCAL VOC 2007 so we download training, validation and test data as well as VOCdevkit (this may take a while!). We are assuming you have the wget utility installed on your computer.
Can you write Python code for it?

# If you already have the dataset on your machine just put the absolute path (root path - w/o VOCdevkit)
# into the following variable:
PASCALVOC2007_PATH=''

if not PASCALVOC2007_PATH:
    # You can change the download target path.
    PASCALVOC2007_PATH='./downloads'

    !mkdir -p {PASCALVOC2007_PATH}
    !wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar {PASCALVOC2007_PATH}
    !wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar {PASCALVOC2007_PATH}
    !wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCdevkit_08-Jun-2007.tar {PASCALVOC2007_PATH}

    !cd {PASCALVOC2007_PATH}; tar xvf VOCtrainval_06-Nov-2007.tar
    !cd {PASCALVOC2007_PATH}; tar xvf VOCtest_06-Nov-2007.tar
    !cd {PASCALVOC2007_PATH}; tar xvf VOCdevkit_08-Jun-2007.tar

# We symlink the dataset path to the folder that is expected by the R-CNN code.
VOCDEVKIT_PATH = os.path.join(PASCALVOC2007_PATH, 'VOCdevkit')
!ln -f -s {VOCDEVKIT_PATH} ./data/VOCdevkit2007


I am working on the file "Seminar6/Seminar6.ipynb".
The first task is:
R-CNN introduction
In a nutshell, R-CNN is just a common image recognition neural network applied to patches that obtained by rescaling rectangular regions of a bigger scene. Those regions are usually computed externally using, for example, Selective Search. The overall pipeline is best described by the following slide from the Girshick's awesome presentation:

You are encouraged to read the relevant paper to get deeper understanding of how the system works.
Setup the code
Cython modules
First, we need to build Cython modules that come with the original Faster R-CNN package.
Can you write Python code for it?


I am working on the file "Seminar6/Seminar6.ipynb".
The first task is:
Region proposals
Next, you need to fetch some external region proposals. The original code uses Selective Search by default. You can download and setup this data by running the cell below. It is very time-consuming, so go have some coffee.
Can you write Python code for it?

# Alternatively, you can ask people who have already downloaded the data to share it.
# Then you can just put it in the ./data folder and skip this step.
!data/scripts/fetch_selective_search_data.sh


I am working on the file "Seminar6/Seminar6.ipynb".
The first task is:
Some tips

Please be careful with the input format. Make sure that your network receives what it expects, i.e. check the order of color channels (RGB/BGR), range of values ($ [0, 1] $/ $ [0, 255] $), etc.
Try Dropout to reduce overfitting.
Try different optimization algorithms. Adam seems to require less tuning but may give slightly worse results than SGD.

Train model
Now it's time to train your freshly written model. First, edit ./experiments/cfgs/rcnn.yml to override the standard settings.
The training procedure is launched by invoking the following shell-command:
Can you write Python code for it?

# Change this to adjust the number of training iterations.
NUM_ITERS=40000

!./experiments/scripts/train_rcnn.sh {NUM_ITERS}


I am working on the file "Seminar6/Seminar6.ipynb".
The first task is:
Test model
Let's asses the quality of trained model on the test set of PASCAL VOC 2007. In order to do this, you have to implement the Tester encapsulating a trained network. The Tester's sole purpose is to produce confidence scores for bounding-boxes at, well, test time.
After the coding is done, the evaluation can be run by invoking:
Can you write Python code for it?

# Put the path to the network snapshot here:
SNAPSHOT=''

!./experiments/scripts/test_rcnn.sh {SNAPSHOT}


I am working on the file "Seminar8/VAE_homework.ipynb".
The first task is:
Autoencoder
Why to use all this complicated formulaes and regularizations, what is the need for variational inference? To analyze the difference, let's first train just an autoencoder on the data:
<img src="Autoencoder_structure.png" alt="Autoencoder">
Can you write Python code for it?

import lasagne

input_X = T.matrix("X")

input_shape = [None,image_h*image_w*3]

HU_encoder = 2000 #you can play with this values
HU_decoder = 2000
dimZ = 1000 #considering face reconstruction task, which size of representation seems reasonable?

# define the network
# use ReLU for hidden layers' activations
# GlorotUniform initialization for W
# zero initialization for biases
# it's also convenient to put sigmoid activation on output layer to get nice normalized pics

#l_input =
#l_enc =
#l_z =
#l_dec =
#l_out =

# create prediction variable
prediction = lasagne.layers.get_output(l_out)

# create loss function
loss = lasagne.objectives.squared_error(prediction, input_X).mean()

# create parameter update expressions
params = lasagne.layers.get_all_params(l_out, trainable=True)
updates = lasagne.updates.adam(loss, params, learning_rate=0.001)

# compile training function that updates parameters and returns training loss
# this will take a while
train_fn = theano.function([input_X], loss, updates=updates)
test_fn = theano.function([input_X], prediction)

def iterate_minibatches(inputs, batchsize, shuffle=True):
    if shuffle:
        indices = np.arange(len(inputs))
        np.random.shuffle(indices)
    for start_idx in range(0, len(inputs) - batchsize + 1, batchsize):
        if shuffle:
            excerpt = indices[start_idx:start_idx + batchsize]
        else:
            excerpt = slice(start_idx, start_idx + batchsize)
        yield inputs[excerpt]

# train your autoencoder
# visualize progress in reconstruction and loss decay

for batch in iterate_minibatches(X_val[:10], 1):
    pred = test_fn(batch)
    plot_gallery([batch[0],pred], image_h, image_w, n_row=1, n_col=2)


import lasagne

input_X = T.matrix("X")

input_shape = [None,image_h*image_w*3]

HU_encoder = 2000 #you can play with this values
HU_decoder = 2000
dimZ = 100 #considering face reconstruction task, which size of representation seems reasonable?

# define the network
# use ReLU for hidden layers' activations
# GlorotUniform initialization for W
# zero initialization for biases
# it's also convenient to put sigmoid activation on output layer to get nice normalized pics

l_input = lasagne.layers.InputLayer(input_shape, input_X)
l_enc = lasagne.layers.DenseLayer(l_input, HU_encoder, b=lasagne.init.Constant(0))
l_z = lasagne.layers.DenseLayer(l_enc, dimZ, b=lasagne.init.Constant(0))
l_dec = lasagne.layers.DenseLayer(l_z, HU_decoder, b=lasagne.init.Constant(0))
l_out = lasagne.layers.DenseLayer(l_dec, input_shape[1], b=lasagne.init.Constant(0),
                                  nonlinearity=lasagne.nonlinearities.sigmoid)

# create prediction variable
prediction = lasagne.layers.get_output(l_out)

# create loss function
loss = lasagne.objectives.squared_error(prediction, input_X).mean()

# create parameter update expressions
params = lasagne.layers.get_all_params(l_out, trainable=True)
updates = lasagne.updates.adam(loss, params, learning_rate=0.001)

# compile training function that updates parameters and returns training loss
# this will take a while
train_fn = theano.function([input_X], loss, updates=updates)
test_fn = theano.function([input_X], prediction)
test_loss = theano.function([input_X], loss)

def iterate_minibatches(inputs, batchsize, shuffle=True):
    if shuffle:
        indices = np.arange(len(inputs))
        np.random.shuffle(indices)
    for start_idx in range(0, len(inputs) - batchsize + 1, batchsize):
        if shuffle:
            excerpt = indices[start_idx:start_idx + batchsize]
        else:
            excerpt = slice(start_idx, start_idx + batchsize)
        yield inputs[excerpt]

from IPython.display import clear_output

val_log = []
train_log = []

# train your autoencoder
# visualize...


I am working on the file "Seminar8/VAE_homework.ipynb".
The first task is:
Can you visualize how the distribution of $z$ looks like? Is it dense? What properties would we expect from it? Can we perform interpolation in $z$ space?
Variational Autoencoder
Bayesian approach in deep learning considers everything in terms of distributions. Now our encoder generates not just a vector $z$ but posterior ditribution q(z|x). Technically, the first difference is that you need to split bottleneck layer in two. One dense layer will generate vector $\mu$, and another will generate vector $\sigma$. Reparametrization trick is implemented via the GaussianSampler layer, that generates random vetor $\epsilon$ and returns $\mu+\sigma\epsilon$
The code for this layer taken from "recipes" folder of Lasagne github repo:
Can you write Python code for it?


I am working on the file "Seminar8/VAE_homework.ipynb".
The first task is:
Congrats!
If you managed to tune your autoencoders to converge and learn something about the world, now it's time to make fun out of it. As you may have noticed, there are face attributes in dataset. We're interesting in "Smiling" column, but feel free to try others as well! Here is the first task:
1) Extract the "Smilling" column as a separate numpy vector and sort this vector.
Can you write Python code for it?

attrs[:10]

smiling = attrs.Smiling.values

ind = np.argsort(smiling)

plot_gallery(np.vstack((data[ind[:10]], data[ind[-10:]])), image_h, image_w, n_row=2, n_col=10)


I am working on the file "Seminar8/VAE_homework.ipynb".
The first task is:
Congrats!
If you managed to tune your autoencoders to converge and learn something about the world, now it's time to make fun out of it. As you may have noticed, there are labels in dataset, some unknown real values, face attribute. Here is the first task:
1) Sort by the attribute and find out what it is about.
Can you write Python code for it?

attrs[:10]
#You're after "Smiling" column, but feel free to try others as well!

#show top- and bottom-10 faces of sorted


I am working on the file "Seminar8/VAE_homework.ipynb".
The first task is:
For simplicity we want all values of the data to lie in the interval $[0,1]$:
Can you write Python code for it?

X_train = np.float32(X_train)
X_train = X_train/255
X_val = np.float32(X_val)
X_val = X_val/255

%matplotlib inline
import matplotlib.pyplot as plt

def plot_gallery(images, h, w, n_row=3, n_col=6):
    """Helper function to plot a gallery of portraits"""
    plt.figure(figsize=(1.5 * n_col, 1.7 * n_row))
    plt.subplots_adjust(bottom=0, left=.01, right=.99, top=.90, hspace=.35)
    for i in range(n_row * n_col):
        plt.subplot(n_row, n_col, i + 1)
        plt.imshow(images[i].reshape((h, w, 3)), cmap=plt.cm.gray, vmin=-1, vmax=1, interpolation='nearest')
        plt.xticks(())
        plt.yticks(())

plot_gallery(X_train, image_h, image_w)

import theano
import theano.tensor as T


I am working on the file "Seminar8/VAE_homework.ipynb".
The first task is:
Now build the loss and training function:
Can you write Python code for it?

# create prediction variable
# prediction =

# create loss function
# ...
# loss = KL_divergence(..., ...) + log_likelihood(..., ..., ...)

# create parameter update expressions
# params =
# updates =

# compile training and testing functions
# train_fn =
# test_fn =


I am working on the file "Seminar8/VAE_homework.ipynb".
The first task is:
Sampling
This task requires deeper Lasagne knowledge. You need to perform inference from $z$, reconstruct an image given some random $z$ representations.
Can you write Python code for it?

z_sample = T.matrix()

# Your code goes here:
# generated_x =

hidden_state = lasagne.layers.InputLayer([None, 100], z_sample)
decoder = lasagne.layers.DenseLayer(hidden_state, HU_decoder, W=l_dec.W, b=l_dec.b)
output_layer = lasagne.layers.DenseLayer(decoder, HU_decoder, W=l_out.W, b=l_out.b,
                                        nonlinearity=lasagne.nonlinearities.sigmoid)

generated_x = lasagne.layers.get_output(output_layer)

gen_fn = theano.function([z_sample], generated_x)

z = np.random.randn(25, dimZ)*0.5
output = gen_fn(np.asarray(z, dtype=theano.config.floatX))
plot_gallery(output, image_h, image_w, n_row=5, n_col=5)


I am working on the file "Seminar8/VAE_homework.ipynb".
The first task is:
Since our decoder is also a function that generates distribution, we need to do the same splitting for output layer. When testing the model we will look only on mean values, so one of the output will be actual autoencoder output.
In this homework we only ask for implementation of the simplest version of VAE - one $z$ sample per input. You can consider to sample several outputs from one input and average like it is in Lasagne recipes.
Can you write Python code for it?

# to compare with conventional AE, keep these hyperparameters
# or change them for the values that you used before

HU_encoder = 2000
HU_decoder = 2000
dimZ = 1000

# define the network
# you can start from https://github.com/Lasagne/Recipes/blob/master/examples/variational_autoencoder/variational_autoencoder.py
# or another example https://github.com/y0ast/Variational-Autoencoder/blob/master/VAE.py
# but remember that this is not your ground truth since the data is not MNIST


I am working on the file "Seminar9/Bonus-seminar.ipynb".
The first task is:
As you can see, our preprocessing is somewhat crude. Let us see if that is enough for our network
Non-sequences
Some data features are categorical data. E.g. location, contract type, company
They require a separate preprocessing step.
Can you write Python code for it?

#One-hot-encoded category and subcategory

from sklearn.feature_extraction import DictVectorizer

categories = []
data_cat = df[["Category","LocationNormalized","ContractType","ContractTime"]]

categories = [A list of dictionaries {"category":category_name, "subcategory":subcategory_name} for each data sample]

vectorizer = DictVectorizer(sparse=False)
df_non_text = vectorizer.fit_transform(categories)
df_non_text = pd.DataFrame(df_non_text,columns=vectorizer.feature_names_)


I am working on the file "Seminar9/Bonus-seminar.ipynb".
The first task is:
Coffee-lation
Can you write Python code for it?

train_fun = theano.function([desc_token_ids,title_token_ids,categories,target_y],[loss,prediction],updates = updates)
eval_fun = theano.function([desc_token_ids,title_token_ids,categories,target_y],[det_loss,det_prediction])


I am working on the file "Seminar9/Bonus-seminar.ipynb".
The first task is:
Determinitic prediction

In case we use stochastic elements, e.g. dropout or noize
Compile a separate set of functions with deterministic prediction (deterministic = True)
Unless you think there's no neet for dropout there ofc. Btw is there?
Can you write Python code for it?

#deterministic version
det_prediction = lasagne.layers.get_output(nn,deterministic=True)[:,0]

#equivalent loss function
det_loss = <an excercise in copy-pasting and editing>


I am working on the file "Seminar9/Bonus-seminar.ipynb".
The first task is:
Final evaluation
Evaluate network over the entire test set
Can you write Python code for it?

#evaluation
epoch_y_true = []
epoch_y_pred = []

b_c = b_loss = 0
for j, (b_desc,b_title,b_cat, b_y) in enumerate(
    iterate_minibatches(desc_ts,title_ts,nontext_ts,target_ts,batchsize=batch_size,shuffle=True)):
    loss,pred_probas = eval_fun(b_desc,b_title,b_cat,b_y)

    b_loss += loss
    b_c +=1

    epoch_y_true.append(b_y)
    epoch_y_pred.append(pred_probas)

epoch_y_true = np.concatenate(epoch_y_true)
epoch_y_pred = np.concatenate(epoch_y_pred)

print "Scores:"
print '\tloss:',b_loss/b_c
print '\trmse:',mean_squared_error(epoch_y_true,epoch_y_pred)**.5
print '\tmae:',mean_absolute_error(epoch_y_true,epoch_y_pred)


I am working on the file "Seminar9/Bonus-seminar.ipynb".
The first task is:
Loss function

The standard way:
prediction
loss
updates
training and evaluation functions
Can you write Python code for it?

#All trainable params
weights = lasagne.layers.get_all_params(nn,trainable=True)

#Simple NN prediction
prediction = lasagne.layers.get_output(nn)[:,0]

#loss function
loss = lasagne.objectives.squared_error(prediction,target_y).mean()

#Weight optimization step
updates = <your favorite optimizer>


I am working on the file "Seminar9/Bonus-seminar.ipynb".
The first task is:
Remove rare tokens
We are unlikely to make use of words that are only seen a few times throughout the corpora.
Again, if you want to beat Kaggle competition metrics, consider doing something better.
Can you write Python code for it?

#Word frequency distribution, just for kicks
_=plt.hist(token_counts.values(),range=[0,50],bins=50)

#Select only the tokens that had at least 10 occurences in the corpora.
#Use token_counts.

min_count = 5
tokens = <tokens from token_counts keys that had at least min_count occurences throughout the dataset>

token_to_id = {t:i+1 for i,t in enumerate(tokens)}
null_token = "NULL"
token_to_id[null_token] = 0

print "# Tokens:",len(token_to_id)
if len(token_to_id) < 10000:
    print "Alarm! It seems like there are too few tokens. Make sure you updated NLTK and applied correct thresholds -- unless you now what you're doing, ofc"
if len(token_to_id) > 100000:
    print "Alarm! Too many tokens. You might have messed up when pruning rare ones -- unless you know what you're doin' ofc"


I am working on the file "Seminar9/Bonus-seminar.ipynb".
The first task is:
Replace words with IDs
Set a maximum length for titles and descriptions.
 * If string is longer that that limit - crop it, if less - pad with zeros.
 * Thus we obtain a matrix of size [n_samples]x[max_length]
 * Element at i,j - is an identifier of word j within sample i
Can you write Python code for it?

def vectorize(strings, token_to_id, max_len=150):
    token_matrix = []
    for s in strings:
        if type(s) is not str:
            token_matrix.append([0]*max_len)
            continue
        s = s.decode('utf8').lower()
        tokens = tokenizer.tokenize(s)
        token_ids = map(lambda token: token_to_id.get(token,0), tokens)[:max_len]
        token_ids += [0]*(max_len - len(token_ids))
        token_matrix.append(token_ids)

    return np.array(token_matrix)

desc_tokens = vectorize(df.FullDescription.values,token_to_id,max_len = 500)
title_tokens = vectorize(df.Title.values,token_to_id,max_len = 15)


I am working on the file "Seminar9/Bonus-seminar.ipynb".
The first task is:
Save preprocessed data [optional]

The next tab can be used to stash all the essential data matrices and get rid of the rest of the data.
Highly recommended if you have less than 1.5GB RAM left
To do that, you need to first run it with save_prepared_data=True, then restart the notebook and only run this tab with read_prepared_data=True.
Can you write Python code for it?

save_prepared_data = True #save
read_prepared_data = False #load

#but not both at once
assert not (save_prepared_data and read_prepared_data)

if save_prepared_data:
    print "Saving preprocessed data (may take up to 3 minutes)"

    import pickle
    with open("preprocessed_data.pcl",'w') as fout:
        pickle.dump(data_tuple,fout)
    with open("token_to_id.pcl",'w') as fout:
        pickle.dump(token_to_id,fout)

    print "done"

elif read_prepared_data:
    print "Reading saved data..."

    import pickle

    with open("preprocessed_data.pcl",'r') as fin:
        data_tuple = pickle.load(fin)
    title_tr,title_ts,desc_tr,desc_ts,nontext_tr,nontext_ts,target_tr,target_ts = data_tuple
    with open("token_to_id.pcl",'r') as fin:
        token_to_id = pickle.load(fin)

    #Re-importing libraries to allow staring noteboook from here
    import pandas as pd
    import numpy as np
    import matplotlib.pyplot as plt
    %matplotlib inline

    print "done"


I am working on the file "Seminar9/Bonus-seminar.ipynb".
The first task is:
Tokenizing
First, we create a dictionary of all existing words.
Assign each word a number - it's Id
Can you write Python code for it?

from nltk.tokenize import RegexpTokenizer
from collections import Counter,defaultdict
tokenizer = RegexpTokenizer(r"\w+")

#Dictionary of tokens
token_counts = Counter()

#All texts
all_texts = np.hstack([df.FullDescription.values,df.Title.values])

#Compute token frequencies
for s in all_texts:
    if type(s) is not str:
        continue
    s = s.decode('utf8').lower()
    tokens = tokenizer.tokenize(s)
    for token in tokens:
        token_counts[token] +=1


I am working on the file "Seminar9/Bonus-seminar.ipynb".
The first task is:
Train the monster
Since we have several data sources, our neural network may differ from what you used to work with.

Separate input for titles
cnn+global max or RNN
Separate input for description
cnn+global max or RNN
Separate input for categorical features
Few dense layers + some black magic if you want

These three inputs must be blended somehow - concatenated or added.

Output: a simple regression task
Can you write Python code for it?

#libraries
import lasagne
from theano import tensor as T
import theano

#3 inputs and a refere output
title_token_ids = T.matrix("title_token_ids",dtype='int32')
desc_token_ids = T.matrix("desc_token_ids",dtype='int32')
categories = T.matrix("categories",dtype='float32')
target_y = T.vector("is_blocked",dtype='float32')


I am working on the file "Seminar9/Bonus-seminar.ipynb".
The first task is:
Training loop

The regular way with loops over minibatches
Since the dataset is huge, we define epoch as some fixed amount of samples isntead of all dataset
Can you write Python code for it?

# Out good old minibatch iterator now supports arbitrary amount of arrays (X,y,z)

def iterate_minibatches(*arrays,**kwargs):

    batchsize=kwargs.get("batchsize",100)
    shuffle = kwargs.get("shuffle",True)

    if shuffle:
        indices = np.arange(len(arrays[0]))
        np.random.shuffle(indices)
    for start_idx in range(0, len(arrays[0]) - batchsize + 1, batchsize):
        if shuffle:
            excerpt = indices[start_idx:start_idx + batchsize]
        else:
            excerpt = slice(start_idx, start_idx + batchsize)
        yield [arr[excerpt] for arr in arrays]


I am working on the file "Seminar9/Bonus-seminar.ipynb".
The first task is:
Tweaking guide

batch_size - how many samples are processed per function call
optimization gets slower, but more stable, as you increase it.
May consider increasing it halfway through training
minibatches_per_epoch - max amount of minibatches per epoch
Does not affect training. Lesser value means more frequent and less stable printing
Setting it to less than 10 is only meaningfull if you want to make sure your NN does not break down after one epoch
n_epochs - total amount of epochs to train for
n_epochs = 10**10 and manual interrupting is still an option

Tips:

With small minibatches_per_epoch, network quality may jump up and down for several epochs

Plotting metrics over training time may be a good way to analyze which architectures work better.

Once you are sure your network aint gonna crash, it's worth letting it train for a few hours of an average laptop's time to see it's true potential
Can you write Python code for it?

from sklearn.metrics import mean_squared_error,mean_absolute_error

n_epochs = 100
batch_size = 100
minibatches_per_epoch = 100

for i in range(n_epochs):

    #training
    epoch_y_true = []
    epoch_y_pred = []

    b_c = b_loss = 0
    for j, (b_desc,b_title,b_cat, b_y) in enumerate(
        iterate_minibatches(desc_tr,title_tr,nontext_tr,target_tr,batchsize=batch_size,shuffle=True)):
        if j > minibatches_per_epoch:break

        loss,pred_probas = train_fun(b_desc,b_title,b_cat,b_y)

        b_loss += loss
        b_c +=1

        epoch_y_true.append(b_y)
        epoch_y_pred.append(pred_probas)

    epoch_y_true = np.concatenate(epoch_y_true)
    epoch_y_pred = np.concatenate(epoch_y_pred)

    print "Train:"
    print '\tloss:',b_loss/b_c
    print '\trmse:',mean_squared_error(epoch_y_true,epoch_y_pred)**.5
    print '\tmae:',mean_absolute_error(epoch_y_true,epoch_y_pred)

    #evaluation
    epoch_y_true = []
    epoch_y_pred = []
    b_c = b_loss = 0
    for j, (b_desc,b_title,b_cat, b_y) in enumerate(
        iterate_minibatches(desc_ts,title_ts,nontext_ts,target_ts,batchsize=batch_size,shuffle=True)):
        if j > minibatches_per_epoch: break
        loss,pred_probas = eval_fun(b_desc,b_title,b_cat,b_y)

        b_loss += loss
        b_c +=1

        epoch_y_true.append(b_y)
        epoch_y_pred.append(pred_probas)

    epoch_y_true = np.concatenate(epoch_y_true)
    epoch_y_pred = np.concatenate(epoch_y_pred)

    print "Val:"
    print '\tloss:',b_loss/b_c
    print '\trmse:',mean_squared_error(epoch_y_true,epoch_y_pred)**.5
    print '\tmae:',mean_absolute_error(epoch_y_true,epoch_y_pred)

print "If you are seeing this, it's time to backup your notebook. No, really, 'tis too easy to mess up everything without noticing. "


I am working on the file "Seminar9/Seminar9-en-Avito.ipynb".
The first task is:

Can you write Python code for it?

print ("Blocked ratio",df.is_blocked.mean())
print ("Count:",len(df))


I am working on the file "Seminar9/Seminar9-en-Avito.ipynb".
The first task is:
As you can see, our preprocessing is somewhat crude. Let us see if that is enough for our network
Non-sequences
Some data features are not text samples. E.g. price, # urls, category, etc
They require a separate preprocessing.
Can you write Python code for it?

#All numeric features
df_numerical_features = df[["phones_cnt","emails_cnt","urls_cnt","price"]]

#One-hot-encoded category and subcategory

from sklearn.feature_extraction import DictVectorizer

categories = []
data_cat_subcat = df[["category","subcategory"]].values

categories = [{"category":cat_name, "subcategory":subcat_name} for cat_name, subcat_name in data_cat_subcat]

vectorizer = DictVectorizer(sparse=False)
cat_one_hot = vectorizer.fit_transform(categories)
cat_one_hot = pd.DataFrame(cat_one_hot,columns=vectorizer.feature_names_)

df_non_text = pd.merge(
    df_numerical_features,cat_one_hot,on = np.arange(len(cat_one_hot))
)
del df_non_text["key_0"]


I am working on the file "Seminar9/Seminar9-en-Avito.ipynb".
The first task is:
Balance-out the classes

Vast majority of data samples are non-prohibited
250k banned out of 4kk
Let's just downsample random 250k legal samples to make further steps less computationally demanding
If you aim for high Kaggle score, consider a smarter approach to that.
Can you write Python code for it?

df.groupby(['category'])['is_blocked'].mean(),

df.category.value_counts()

#downsample
data = df.copy()
df = data[data.is_blocked == 1]
data = data[data.is_blocked == 0]
df.shape

for category in data.category.value_counts().keys():
    to_add = data[data.category == category]
    n_samples = df.category.value_counts()[category]
    indexes = np.random.choice(np.arange(to_add.shape[0]), size=n_samples)
    df = df.append(to_add.iloc[indexes])

print ("Blocked ratio:",df.is_blocked.mean())
print ("Count:",len(df))

df.groupby('category')['is_blocked'].mean()

assert df.is_blocked.mean() < 0.51
assert df.is_blocked.mean() > 0.49
assert len(df) <= 560000

print ("All tests passed")

#In case your RAM-o-meter is in the red
if very_low_ram:
    data = data[::2]


I am working on the file "Seminar9/Seminar9-en-Avito.ipynb".
The first task is:
Coffee-lation
Can you write Python code for it?


I am working on the file "Seminar9/Seminar9-en-Avito.ipynb".
The first task is:
Data format examples
Can you write Python code for it?

print ("Размер матрицы:",title_tokens.shape)
for title, tokens in zip(df.title.values[:3],title_tokens[:3]):
    print (title,'->', tokens[:10],'...')


I am working on the file "Seminar9/Seminar9-en-Avito.ipynb".
The first task is:
Determinitic prediction

In case we use stochastic elements, e.g. dropout or noize
Compile a separate set of functions with deterministic prediction (deterministic = True)
Unless you think there's no neet for dropout there ofc. Btw is there?
Can you write Python code for it?

#deterministic version
det_prediction = lasagne.layers.get_output(nn,deterministic=True)[:,0]

#equivalent loss function
det_loss = lasagne.objectives.binary_hinge_loss(det_prediction, target_y, log_odds=True).mean()


I am working on the file "Seminar9/Seminar9-en-Avito.ipynb".
The first task is:
Remove rare tokens
We are unlikely to make use of words that are only seen a few times throughout the corpora.
Again, if you want to beat Kaggle competition metrics, consider doing something better.
Can you write Python code for it?

#Word frequency distribution, just for kicks
_ = plt.hist(list(token_counts.values()),range=[0,50],bins=50)

#Select only the tokens that had at least 10 occurences in the corpora.
#Use token_counts.

min_count = 10
tokens = dict(token for token in list(token_counts.items()) if token[1] > min_count)

token_to_id = {t:i+1 for i,t in enumerate(tokens)}
null_token = "NULL"
token_to_id[null_token] = 0

print ("# Tokens:",len(token_to_id))
if len(token_to_id) < 30000:
    print ("Alarm! It seems like there are too few tokens. Make sure you updated NLTK and applied correct thresholds -- unless you now what you're doing, ofc")
if len(token_to_id) > 1000000:
    print ("Alarm! Too many tokens. You might have messed up when pruning rare ones -- unless you know what you're doin' ofc")


I am working on the file "Seminar9/Seminar9-en-Avito.ipynb".
The first task is:
Replace words with IDs
Set a maximum length for titles and descriptions.
 * If string is longer that that limit - crop it, if less - pad with zeros.
 * Thus we obtain a matrix of size [n_samples]x[max_length]
 * Element at i,j - is an identifier of word j within sample i
Can you write Python code for it?

def vectorize(strings, token_to_id, max_len=150):
    token_matrix = []
    for s in strings:
        if type(s) is not str:
            token_matrix.append([0]*max_len)
            continue
        s = s.lower()
        tokens = tokenizer.tokenize(s)
        token_ids = list(map(lambda token: token_to_id.get(token,0), tokens))[:max_len]
        token_ids += [0]*(max_len - len(token_ids))
        token_matrix.append(token_ids)

    return np.array(token_matrix)

desc_tokens = vectorize(df.description.values,token_to_id,max_len = 150)
title_tokens = vectorize(df.title.values,token_to_id,max_len = 15)


I am working on the file "Seminar9/Seminar9-en-Avito.ipynb".
The first task is:
Save preprocessed data [optional]

The next tab can be used to stash all the essential data matrices and get rid of the rest of the data.
Highly recommended if you have less than 1.5GB RAM left
To do that, you need to first run it with save_prepared_data=True, then restart the notebook and only run this tab with read_prepared_data=True.
Can you write Python code for it?

save_prepared_data = False #save
read_prepared_data = True #load

#but not both at once
assert not (save_prepared_data and read_prepared_data)

if save_prepared_data:
    print ("Saving preprocessed data (may take up to 3 minutes)")

    import pickle
    with open("preprocessed_data.pcl",'wb') as fout:
        pickle.dump(data_tuple, fout)
    with open("token_to_id.pcl",'wb') as fout:
        pickle.dump(token_to_id,fout)

    print ("готово")

elif read_prepared_data:
    print ("Reading saved data...")

    import pickle

    with open("preprocessed_data.pcl",'rb') as fin:
        data_tuple = pickle.load(fin)
    title_tr,title_ts,desc_tr,desc_ts,nontext_tr,nontext_ts,target_tr,target_ts = data_tuple
    with open("token_to_id.pcl",'rb') as fin:
        token_to_id = pickle.load(fin)

    #Re-importing libraries to allow staring noteboook from here
    import pandas as pd
    import numpy as np
    import matplotlib.pyplot as plt
    %matplotlib inline

    print ("done")


I am working on the file "Seminar9/Seminar9-en-Avito.ipynb".
The first task is:
Split data into training and test
Can you write Python code for it?

#Target variable - whether or not sample contains prohibited material
target = df.is_blocked.values.astype('int32')
#Preprocessed titles
title_tokens = title_tokens.astype('int32')
#Preprocessed tokens
desc_tokens = desc_tokens.astype('int32')

#Non-sequences
df_non_text = df_non_text.astype('float32')

from sklearn.model_selection import train_test_split

#Split into training and test set.

#Difficulty selector:
#Easy: split randomly
#Medium: select test set items that have item_ids strictly above that of training set
#Hard: do whatever you want, but score yourself using kaggle private leaderboard

data_tuple = train_test_split(title_tokens, desc_tokens, df_non_text, target, test_size=0.1, random_state=123)
title_tr,title_ts,desc_tr,desc_ts,nontext_tr,nontext_ts,target_tr,target_ts = data_tuple


I am working on the file "Seminar9/Seminar9-en-Avito.ipynb".
The first task is:
Tokenizing
First, we create a dictionary of all existing words.
Assign each word a number - it's Id
Can you write Python code for it?

from nltk.tokenize import RegexpTokenizer
from collections import Counter,defaultdict
tokenizer = RegexpTokenizer(r"\w+")

#Dictionary of tokens
token_counts = Counter()

#All texts
all_texts = np.hstack([df.description.values,df.title.values])

#Compute token frequencies
for s in all_texts:
    if type(s) is not str:
        continue
    s = s.lower()
    tokens = tokenizer.tokenize(s)
    for token in tokens:
        token_counts[token] +=1


I am working on the file "Seminar9/Seminar9-en-Avito.ipynb".
The first task is:
Training loop

The regular way with loops over minibatches
Since the dataset is huge, we define epoch as some fixed amount of samples isntead of all dataset
Can you write Python code for it?

#average precision at K

from oracle import APatK, score

# Out good old minibatch iterator now supports arbitrary amount of arrays (X,y,z)

def iterate_minibatches(*arrays,**kwargs):

    batchsize=kwargs.get("batchsize",100)
    shuffle = kwargs.get("shuffle",True)

    if shuffle:
        indices = np.arange(len(arrays[0]))
        np.random.shuffle(indices)
    for start_idx in range(0, len(arrays[0]) - batchsize + 1, batchsize):
        if shuffle:
            excerpt = indices[start_idx:start_idx + batchsize]
        else:
            excerpt = slice(start_idx, start_idx + batchsize)
        yield [np.array(arr)[excerpt] for arr in arrays]


I am working on the file "Seminar9/Seminar9-en-Avito.ipynb".
The first task is:
Tweaking guide

batch_size - how many samples are processed per function call
optimization gets slower, but more stable, as you increase it.
May consider increasing it halfway through training
minibatches_per_epoch - max amount of minibatches per epoch
Does not affect training. Lesser value means more frequent and less stable printing
Setting it to less than 10 is only meaningfull if you want to make sure your NN does not break down after one epoch
n_epochs - total amount of epochs to train for
n_epochs = 10**10 and manual interrupting is still an option

Tips:

With small minibatches_per_epoch, network quality may jump around 0.5 for several epochs

AUC is the most stable of all three metrics

Average Precision at top 2.5% (APatK) - is the least stable. If batch_size*minibatches_per_epoch < 10k, it behaves as a uniform random variable.

Plotting metrics over training time may be a good way to analyze which architectures work better.

Once you are sure your network aint gonna crash, it's worth letting it train for a few hours of an average laptop's time to see it's true potential
Can you write Python code for it?

from sklearn.metrics import roc_auc_score, accuracy_score

n_epochs = 5
batch_size = 500
minibatches_per_epoch = 100

for i in range(n_epochs):
    #training
    epoch_y_true = []
    epoch_y_pred = []

    b_c = b_loss = 0
    for j, (b_desc,b_title,b_cat, b_y) in enumerate(
        iterate_minibatches(desc_tr,title_tr,nontext_tr,target_tr,batchsize=batch_size,shuffle=True)):
        if j > minibatches_per_epoch:break

        loss,pred_probas = train_fun(b_desc,b_title,b_cat,b_y)

        b_loss += loss
        b_c +=1

        epoch_y_true.append(b_y)
        epoch_y_pred.append(pred_probas)

    epoch_y_true = np.concatenate(epoch_y_true)
    epoch_y_pred = np.concatenate(epoch_y_pred)

    print ("Train:")
    print ('\tloss:',b_loss/b_c)
    print ('\tacc:',accuracy_score(epoch_y_true,epoch_y_pred>0.))
    print ('\tauc:',roc_auc_score(epoch_y_true,epoch_y_pred))
    print ('\tap@k:',APatK(epoch_y_true,epoch_y_pred,K = int(len(epoch_y_pred)*0.025)+1))

    #evaluation
    epoch_y_true = []
    epoch_y_pred = []
    b_c = b_loss = 0
    for j, (b_desc,b_title,b_cat, b_y) in enumerate(
        iterate_minibatches(desc_ts,title_ts,nontext_tr,target_ts,batchsize=batch_size,shuffle=True)):
        if j > minibatches_per_epoch: break
        loss,pred_probas = eval_fun(b_desc,b_title,b_cat,b_y)

        b_loss += loss
        b_c +=1

        epoch_y_true.append(b_y)
        epoch_y_pred.append(pred_probas)

    epoch_y_true = np.concatenate(epoch_y_true)
    epoch_y_pred = np.concatenate(epoch_y_pred)

    print ("Val:")
    print ('\tloss:',b_loss/b_c)
    print ('\tacc:',accuracy_score(epoch_y_true,epoch_y_pred>0.))
    print ('\tauc:',roc_auc_score(epoch_y_true,epoch_y_pred))
    print ('\tap@k:',APatK(epoch_y_true,epoch_y_pred,K = int(len(epoch_y_pred)*0.025)+1))

print ("If you are seeing this, it's time to backup your notebook. No, really, 'tis too easy to mess up everything without noticing. ")


I am working on the file "Seminar9/Seminar9_en.ipynb".
The first task is:

Can you write Python code for it?

print "Blocked ratio",df.is_blocked.mean()
print "Count:",len(df)


I am working on the file "Seminar9/Seminar9_en.ipynb".
The first task is:
Balance-out the classes

Vast majority of data samples are non-prohibited
250k banned out of 4kk
Let's just downsample random 250k legal samples to make further steps less computationally demanding
If you aim for high Kaggle score, consider a smarter approach to that.
Can you write Python code for it?

#downsample

< downsample data so that both classes have approximately equal ratios>

df = <downsampled dataset>

print "Blocked ratio:",df.is_blocked.mean()
print "Count:",len(df)

assert df.is_blocked.mean() < 0.51
assert df.is_blocked.mean() > 0.49
assert len(df) <= 560000

print "All tests passed"

#In case your RAM-o-meter is in the red
if very_low_ram:
    data = data[::2]


I am working on the file "Seminar9/Seminar9_en.ipynb".
The first task is:
Coffee-lation
Can you write Python code for it?

train_fun = theano.function([desc_token_ids,title_token_ids,categories,target_y],[loss,prediction],updates = updates)
eval_fun = theano.function([desc_token_ids,title_token_ids,categories,target_y],[det_loss,det_prediction])


I am working on the file "Seminar9/Seminar9_en.ipynb".
The first task is:
Data format examples
Can you write Python code for it?

print "Размер матрицы:",title_tokens.shape
for title, tokens in zip(df.title.values[:3],title_tokens[:3]):
    print title,'->', tokens[:10],'...'


I am working on the file "Seminar9/Seminar9_en.ipynb".
The first task is:
Dataset
Ex-kaggle-competition on prohibited content detection
There goes the description - https://www.kaggle.com/c/avito-prohibited-content
Download
High-RAM mode,
 * Download avito_train.tsv from competition data files
Low-RAM-mode,
 * Download downsampled dataset from here
     * archive https://yadi.sk/d/l0p4lameqw3W8
     * raw https://yadi.sk/d/I1v7mZ6Sqw2WK (in case you feel masochistic)
What's inside
Different kinds of features:
* 2 text fields - title and description
* Special features - price, number of e-mails, phones, etc
* Category and subcategory - unsurprisingly, categorical features
* Attributes - more factors
Only 1 binary target whether or not such advertisement contains prohibited materials
* criminal, misleading, human reproduction-related, etc
* diving into the data may result in prolonged sleep disorders
Can you write Python code for it?

if not low_RAM_mode:
    # a lot of ram
    df = pd.read_csv("avito_train.tsv",sep='\t')
else:
    #aroung 4GB ram
    df = pd.read_csv("avito_train_1kk.tsv",sep='\t')

print df.shape, df.is_blocked.mean()
df[:5]


I am working on the file "Seminar9/Seminar9_en.ipynb".
The first task is:
Determinitic prediction

In case we use stochastic elements, e.g. dropout or noize
Compile a separate set of functions with deterministic prediction (deterministic = True)
Unless you think there's no neet for dropout there ofc. Btw is there?
Can you write Python code for it?

#deterministic version
det_prediction = lasagne.layers.get_output(nn,deterministic=True)[:,0]

#equivalent loss function
det_loss = <an excercise in copy-pasting and editing>


I am working on the file "Seminar9/Seminar9_en.ipynb".
The first task is:
Loss function

The standard way:
prediction
loss
updates

training and evaluation functions

Hinge loss

$ L_i = \max(0, \delta - t_i p_i) $
delta is a tunable parameter: how far should a neuron be in the positive margin area for us to stop bothering about it
Function description may mention some +-1  limitations - this is not neccessary, at least as long as hinge loss has a default flag binary = True
Can you write Python code for it?

#All trainable params
weights = lasagne.layers.get_all_params(nn,trainable=True)

#Simple NN prediction
prediction = lasagne.layers.get_output(nn)[:,0]

#Hinge loss
loss = lasagne.objectives.binary_hinge_loss(prediction,target_y,delta = what_do_you_think).mean()

#Weight optimization step
updates = <your favorite optimizer>


I am working on the file "Seminar9/Seminar9_en.ipynb".
The first task is:
Remove rare tokens
We are unlikely to make use of words that are only seen a few times throughout the corpora.
Again, if you want to beat Kaggle competition metrics, consider doing something better.
Can you write Python code for it?

#Word frequency distribution, just for kicks
_=plt.hist(token_counts.values(),range=[0,50],bins=50)

#Select only the tokens that had at least 10 occurences in the corpora.
#Use token_counts.

min_count = 10
tokens = <tokens from token_counts keys that had at least min_count occurences throughout the dataset>

token_to_id = {t:i+1 for i,t in enumerate(tokens)}
null_token = "NULL"
token_to_id[null_token] = 0

print "# Tokens:",len(token_to_id)
if len(token_to_id) < 30000:
    print "Alarm! It seems like there are too few tokens. Make sure you updated NLTK and applied correct thresholds -- unless you now what you're doing, ofc"
if len(token_to_id) < 1000000:
    print "Alarm! Too many tokens. You might have messed up when pruning rare ones -- unless you know what you're doin' ofc"


I am working on the file "Seminar9/Seminar9_en.ipynb".
The first task is:
Save preprocessed data [optional]

The next tab can be used to stash all the essential data matrices and get rid of the rest of the data.
Highly recommended if you have less than 1.5GB RAM left
To do that, you need to first run it with save_prepared_data=True, then restart the notebook and only run this tab with read_prepared_data=True.
Can you write Python code for it?

save_prepared_data = True #save
read_prepared_data = False #load

#but not both at once
assert not (save_prepared_data and read_prepared_data)

if save_prepared_data:
    print "Saving preprocessed data (may take up to 3 minutes)"

    import pickle
    with open("preprocessed_data.pcl",'w') as fout:
        pickle.dump(data_tuple,fout)
    with open("token_to_id.pcl",'w') as fout:
        pickle.dump(token_to_id,fout)

    print "готово"

elif read_prepared_data:
    print "Reading saved data..."

    import pickle

    with open("preprocessed_data.pcl",'r') as fin:
        data_tuple = pickle.load(fin)
    title_tr,title_ts,desc_tr,desc_ts,nontext_tr,nontext_ts,target_tr,target_ts = data_tuple
    with open("token_to_id.pcl",'r') as fin:
        token_to_id = pickle.load(fin)

    #Re-importing libraries to allow staring noteboook from here
    import pandas as pd
    import numpy as np
    import matplotlib.pyplot as plt
    %matplotlib inline

    print "done"


I am working on the file "Seminar9/Seminar9_en.ipynb".
The first task is:
Split data into training and test
Can you write Python code for it?

#Target variable - whether or not sample contains prohibited material
target = df.is_blocked.values.astype('int32')
#Preprocessed titles
title_tokens = title_tokens.astype('int32')
#Preprocessed tokens
desc_tokens = desc_tokens.astype('int32')

#Non-sequences
df_non_text = df_non_text.astype('float32')

#Split into training and test set.

#Difficulty selector:
#Easy: split randomly
#Medium: select test set items that have item_ids strictly above that of training set
#Hard: do whatever you want, but score yourself using kaggle private leaderboard

title_tr,title_ts,desc_tr,desc_ts,nontext_tr,nontext_ts,target_tr,target_ts = <define_these_variables>


I am working on the file "Seminar9/Seminar9_en.ipynb".
The first task is:
Train the monster
Since we have several data sources, our neural network may differ from what you used to work with.

Separate input for titles
cnn+global max or RNN
Separate input for description
cnn+global max or RNN
Separate input for categorical features
обычные полносвязные слои или какие-нибудь трюки

These three inputs must be blended somehow - concatenated or added.

Output: a simple binary classification
1 sigmoidal with binary_crossentropy
2 softmax with categorical_crossentropy - essentially the same as previous one
1 neuron without nonlinearity (lambda x: x) +  hinge loss
Can you write Python code for it?

#libraries
import lasagne
from theano import tensor as T
import theano

#3 inputs and a refere output
title_token_ids = T.matrix("title_token_ids",dtype='int32')
desc_token_ids = T.matrix("desc_token_ids",dtype='int32')
categories = T.matrix("categories",dtype='float32')
target_y = T.ivector("is_blocked")


I am working on the file "Seminar9/Seminar9_en.ipynb".
The first task is:
Training loop

The regular way with loops over minibatches
Since the dataset is huge, we define epoch as some fixed amount of samples isntead of all dataset
Can you write Python code for it?

#average precision at K

from oracle import APatK, score

# Out good old minibatch iterator now supports arbitrary amount of arrays (X,y,z)

def iterate_minibatches(*arrays,**kwargs):

    batchsize=kwargs.get("batchsize",100)
    shuffle = kwargs.get("shuffle",True)

    if shuffle:
        indices = np.arange(len(arrays[0]))
        np.random.shuffle(indices)
    for start_idx in range(0, len(arrays[0]) - batchsize + 1, batchsize):
        if shuffle:
            excerpt = indices[start_idx:start_idx + batchsize]
        else:
            excerpt = slice(start_idx, start_idx + batchsize)
        yield [arr[excerpt] for arr in arrays]


I am working on the file "Seminar9/Seminar9_ru.ipynb".
The first task is:

Can you write Python code for it?

print "Доля заблокированных объявлений",df.is_blocked.mean()
print "Всего объявлений:",len(df)


I am working on the file "Seminar9/Seminar9_ru.ipynb".
The first task is:
Final evaluation
Оценим качество модели по всей тестовой выборке.
Can you write Python code for it?

#evaluation
epoch_y_true = []
epoch_y_pred = []

b_c = b_loss = 0
for j, (b_desc,b_title,b_cat, b_y) in enumerate(
    iterate_minibatches(desc_ts,title_ts,nontext_tr,target_ts,batchsize=batch_size,shuffle=True)):
    loss,pred_probas = eval_fun(b_desc,b_title,b_cat,b_y)

    b_loss += loss
    b_c +=1

    epoch_y_true.append(b_y)
    epoch_y_pred.append(pred_probas)

epoch_y_true = np.concatenate(epoch_y_true)
epoch_y_pred = np.concatenate(epoch_y_pred)

final_accuracy = accuracy_score(epoch_y_true,epoch_y_pred>0)
final_auc = roc_auc_score(epoch_y_true,epoch_y_pred)
final_apatk = APatK(epoch_y_true,epoch_y_pred,K = int(len(epoch_y_pred)*0.025)+1)

print "Scores:"
print '\tloss:',b_loss/b_c
print '\tacc:',final_accuracy
print '\tauc:',final_auc
print '\tap@k:',final_apatk
score(final_accuracy,final_auc,final_apatk)


I am working on the file "Seminar9/Seminar9_ru.ipynb".
The first task is:
Вырежем редкие токены
Can you write Python code for it?

#распределение частот слов - большинство слов встречаются очень редко - для нас это мусор
_=plt.hist(token_counts.values(),range=[0,50],bins=50)

#возьмём только те токены, которые встретились хотя бы 10 раз в обучающей выборке
#информацию о том, сколько раз встретился каждый токен, можно найти в словаре token_counts

min_count = 10
tokens = <список слов(ключей) из token_counts, которые встретились в выборке не менее min_count раз>

token_to_id = {t:i+1 for i,t in enumerate(tokens)}
null_token = "NULL"
token_to_id[null_token] = 0

print "Всего токенов:",len(token_to_id)
if len(token_to_id) < 30000:
    print "Алярм! Мало токенов. Проверьте, есть ли в token_to_id юникодные символы, если нет - обновите nltk или возьмите другой токенизатор"
if len(token_to_id) < 1000000:
    print "Алярм! Много токенов. Если вы знаете, что делаете - всё ок, если нет - возможно, вы слижком слабо обрезали токены по количеству"


I am working on the file "Seminar9/Seminar9_ru.ipynb".
The first task is:
Главный цикл обучения

Всё как обычно - в цикле по минибатчам запускаем функцию обновления весов.
Поскольку выборка огромна, а чашки чая хватает в среднем на  100к примеров, будем на каждой эпохе пробегать только часть примеров.
Can you write Python code for it?

#average precision at K

from oracle import APatK, score

# наш старый знакомый - итератор по корзинкам - теперь умеет работать с произвольным числом каналов (название, описание, категории, таргет)

def iterate_minibatches(*arrays,**kwargs):

    batchsize=kwargs.get("batchsize",100)
    shuffle = kwargs.get("shuffle",True)

    if shuffle:
        indices = np.arange(len(arrays[0]))
        np.random.shuffle(indices)
    for start_idx in range(0, len(arrays[0]) - batchsize + 1, batchsize):
        if shuffle:
            excerpt = indices[start_idx:start_idx + batchsize]
        else:
            excerpt = slice(start_idx, start_idx + batchsize)
        yield [arr[excerpt] for arr in arrays]


I am working on the file "Seminar9/Seminar9_ru.ipynb".
The first task is:
Заменим слова на их id
Для каждого описания установим максимальную длину.
 * Если описание больше длины - обрежем, если меньше - дополним нулями.
 * Таким образом, у нас получится матрица размера (число объявлений)x(максимальная длина)
 * Элемент под индексами i,j - номер j-того слова i-того объявления
Can you write Python code for it?

def vectorize(strings, token_to_id, max_len=150):
    token_matrix = []
    for s in strings:
        if type(s) is not str:
            token_matrix.append([0]*max_len)
            continue
        s = s.decode('utf8').lower()
        tokens = tokenizer.tokenize(s)
        token_ids = map(lambda token: token_to_id.get(token,0), tokens)[:max_len]
        token_ids += [0]*(max_len - len(token_ids))
        token_matrix.append(token_ids)

    return np.array(token_matrix)

desc_tokens = vectorize(df.description.values,token_to_id,max_len = 150)
title_tokens = vectorize(df.title.values,token_to_id,max_len = 15)


I am working on the file "Seminar9/Seminar9_ru.ipynb".
The first task is:
Как вы видите, всё довольно грязно. Посмотрим, сожрёт ли это нейронка
Нетекстовые признаки
Часть признаков не являются строками текста: цена, количество телефонов, категория товара.
Их можно обработать отдельно.
Can you write Python code for it?

#Возьмём числовые признаки
df_numerical_features = df[["phones_cnt","emails_cnt","urls_cnt","price"]]

#Возьмём one-hot encoding категорий товара.
#Для этого можно использовать DictVectorizer (или другой ваш любимый препроцессор)

from sklearn.feature_extraction import DictVectorizer

categories = []
for cat_str, subcat_str in df[["category","subcategory"]].values:

    cat_dict = {"category":cat_str,"subcategory":subcat_str}
    categories.append(cat_dict)

vectorizer = DictVectorizer(sparse=False)
cat_one_hot = vectorizer.fit_transform(categories)
cat_one_hot = pd.DataFrame(cat_one_hot,columns=vectorizer.feature_names_)

df_non_text = pd.merge(
    df_numerical_features,cat_one_hot,on = np.arange(len(cat_one_hot))
)
del df_non_text["key_0"]


I am working on the file "Seminar9/Seminar9_ru.ipynb".
The first task is:
Поделим данные на обучение и тест
Can you write Python code for it?

#целевая переменная - есть заблокирован ли контент
target = df.is_blocked.values.astype('int32')
#закодированное название
title_tokens = title_tokens.astype('int32')
#закодированное описание
desc_tokens = desc_tokens.astype('int32')

#все нетекстовые признаки
df_non_text = df_non_text.astype('float32')

#поделим всё это на обучение и тест
from sklearn.cross_validation import train_test_split
data_tuple = train_test_split(title_tokens,desc_tokens,df_non_text.values,target)

title_tr,title_ts,desc_tr,desc_ts,nontext_tr,nontext_ts,target_tr,target_ts = data_tuple


I am working on the file "Seminar9/Seminar9_ru.ipynb".
The first task is:
Познакомимся с данными
Бывший kaggle-конкурс про выявление нежелательного контента.
Описание конкурса есть тут - https://www.kaggle.com/c/avito-prohibited-content
Скачать
Если много RAM,
 * Из данных конкурса (вкладка Data) нужно скачать avito_train.tsv и распаковать в папку с тетрадкой
Если мало RAM,
 * Cкачайте прореженную выборку отсюда
     * Пожатая https://yadi.sk/d/l0p4lameqw3W8
     * Непожатая https://yadi.sk/d/I1v7mZ6Sqw2WK
Много разных признаков:
* 2 вида текста - заголовок и описание
* Много специальных фичей - цена, количество телефонов/ссылок/e-mail адресов
* Категория и субкатегория - как ни странно, категориальные фичи
* Аттрибуты - много категориальных признаков
Нужно предсказать всего 1 бинарный признак - есть ли в рекламе нежелательный контент.
* Под нежелательным контентом понимается криминал, прон, афера, треска и прочие любимые нами темы.
* Да, если присмотреться к заблокированным объявлениям, можно потерять аппетит и сон на пару дней.
* Однако профессия аналитика данных обязывает вас смотреть на данные.
 * А кто сказал, что будет легко? Data Science - опасная профессия.
Can you write Python code for it?

if not low_RAM_mode:
    # Если у вас много оперативки
    df = pd.read_csv("avito_train.tsv",sep='\t')
else:
    #Если у вас меньше 4gb оперативки
    df = pd.read_csv("avito_train_1kk.tsv",sep='\t')

print df.shape, df.is_blocked.mean()
df[:5]


I am working on the file "Seminar9/Seminar9_ru.ipynb".
The first task is:
Поучим нейронку
Поскольку у нас есть несколько источников данных, наша нейронная сеть будет немного отличаться от тех, что вы тренировали раньше.

Отдельный вход для заголовка
свёртка + global max pool или RNN
Отдельный вход для описания
свёртка + global max pool или RNN
Отдельный вход для категориальных признаков
обычные полносвязные слои или какие-нибудь трюки

Всё это нужно как-то смешать - например, сконкатенировать

Выход - обычный двухклассовый выход
1 сигмоидальный нейрон и binary_crossentropy
2 нейрона с softmax и categorical_crossentropy - то же самое, что 1 сигмоидальный
1 нейрон без нелинейности (lambda x: x) и hinge loss
Can you write Python code for it?

#загрузим библиотеки
import lasagne
from theano import tensor as T
import theano

#3 входа и 1 выход
title_token_ids = T.matrix("title_token_ids",dtype='int32')
desc_token_ids = T.matrix("desc_token_ids",dtype='int32')
categories = T.matrix("categories",dtype='float32')
target_y = T.ivector("is_blocked")


I am working on the file "Seminar9/Seminar9_ru.ipynb".
The first task is:
Сбалансируем выборку

Выборка смещена в сторону незаблокированных объявлений
4 миллиона объявлений и только 250 тысяч заблокированы.
Давайте просто выберем случайные 250 тысяч незаблокированных объявлений и сократим выборку до полумилиона.
В последствии можно испоьзовать более умные способы сбалансировать выборку

Если у вас слабый ПК и вы видите OutOfMemory, попробуйте уменьшить размер выборки до 100 000 примеров
Алсо если вы не хотите ждать чтения всех данных каждый раз - сохраните уменьшенную выборку и читайте её
Can you write Python code for it?

#downsample

< выдели подвыборку, в которой отрицательных примеров примерно столько же, сколько положительных>

df = <уменьшенная подвыборка>

print "Доля заблокированных объявлений:",df.is_blocked.mean()
print "Всего объявлений:",len(df)

assert df.is_blocked.mean() < 0.51
assert df.is_blocked.mean() > 0.49
assert len(df) <= 560000

print "All tests passed"

#прореживаем данные ещё в 2 раза, если памяти не хватает
if very_low_ram:
    data = data[::2]


I am working on the file "Seminar9/Seminar9_ru.ipynb".
The first task is:
Скомпилируем функции обучения и оценки качества
Can you write Python code for it?

train_fun = theano.function([desc_token_ids,title_token_ids,categories,target_y],[loss,prediction],updates = updates)
eval_fun = theano.function([desc_token_ids,title_token_ids,categories,target_y],[det_loss,det_prediction])


I am working on the file "Seminar9/Seminar9_ru.ipynb".
The first task is:
Токенизируем примеры
Сначала соберём словарь всех возможных слов.
Поставим каждому слову в соответствие целое число - его id
Can you write Python code for it?

from nltk.tokenize import RegexpTokenizer
from collections import Counter,defaultdict
tokenizer = RegexpTokenizer(r"\w+")

#словарь для всех токенов
token_counts = Counter()

#все заголовки и описания
all_texts = np.hstack([df.description.values,df.title.values])

#считаем частоты слов
for s in all_texts:
    if type(s) is not str:
        continue
    s = s.decode('utf8').lower()
    tokens = tokenizer.tokenize(s)
    for token in tokens:
        token_counts[token] +=1


I am working on the file "Seminar9/Seminar9_ru.ipynb".
The first task is:
Целевая функция и обновления весов

Делаем всё стандартно:
получаем предсказание
считаем функцию потерь
вычисляем обновления весов

компилируем итерацию обучения и оценки весов

Hinge loss

$ L_i = \max(0, \delta - t_i p_i) $
Важный параметр - delta - насколько глубоко пример должен быть в правильном классе, чтобы перестать нас волновать
В описании функции в документации может быть что-то про ограничения на +-1 - не верьте этому - главное, чтобы в функции по умолчанию стоял флаг binary = True
Can you write Python code for it?

#Все обучаемые параметры сети
weights = lasagne.layers.get_all_params(nn,trainable=True)

#Обычное предсказание нейронки
prediction = lasagne.layers.get_output(nn)[:,0]

#функция потерь для prediction
loss = lasagne.objectives.binary_hinge_loss(prediction,target_y,delta = 1.0).mean()

#Шаг оптимизации весов
updates = <Ваш любимый метод оптимизации весов>


I am working on the file "Seminar9/Seminar9_ru.ipynb".
The first task is:
Чтобы оценивать качество сети, в которой есть элемент случайности

Dropout, например,
Нужно отдельно вычислить ошибку для случая, когда dropout выключен (deterministic = True)
К слову, неплохо бы убедиться, что droput нам вообще нужен
Can you write Python code for it?

#Предсказание нейронки без учёта dropout и прочего шума - если он есть
det_prediction = lasagne.layers.get_output(nn,deterministic=True)[:,0]

#функция потерь для det_prediction
det_loss = lasagne.objectives.binary_hinge_loss(det_prediction,target_y,delta = 1.0).mean()


I am working on the file "Send mail.ipynb".
The first task is:
Send mail function.
This function takes 5 argument. 1. Login Data. 2. To Email 3. From Email  4. HTML format massage 5. Normal text
The HTML message, is best and preferred.
Can you write Python code for it?

# text = "Hi!\n5633222222222222222http://www.python.org"
# html = """\
#
#
#
#
Hi!

#            How are you?

#            Here is the link you wanted.
#

#
#
#     """
def Send_Mail(smtpserver,TO_EMAIL,text=None,html=None,subject='Subject missing',FROM_EMAIL='Shahariar'):
# Create message container - the correct MIME type is multipart/alternative.
    msg = MIMEMultipart('alternative') # In turn, use text/plain and text/html parts within the multipart/alternative part.
    msg['Subject'] = subject #Subject of the message
    msg['From'] = formataddr((str(Header(FROM_EMAIL, 'utf-8')), FROM_EMAIL)) #Adding custom Sender Name
    msg['To'] = TO_EMAIL #Assining Reciver email

    part1 = MIMEText(text, 'plain') #adding text part of mail
    part2 = MIMEText(html, 'html') #Adding HTMLpart of mail

    # Attach parts into message container.
    # According to RFC 2046, the last part of a multipart message, in this case
    # the HTML message, is best and preferred.
    msg.attach(part1) #attach Plain text
    msg.attach(part2) #attach HTML text

    # sendmail function takes 3 arguments: sender's address, recipient's address
    # and message to send - here it is sent as one string.
    try:
        smtpserver.sendmail(FROM_EMAIL, TO_EMAIL, msg.as_string())
        print " Message Send"
        smtpserver.quit() #stopping server
    except Exception:
        print Exception


I am working on the file "SensorMLparsing_IOOSSOS.ipynb".
The first task is:
Export to GeoPackage (gpkg) file
Can you write Python code for it?

stations_gdf.dtypes

for d in stations_gdf:
    print(d, type(stations_gdf['{}'.format(d)][0]))

# Remove attributes that lead to puking when exporting to file
stations_gdf_out = stations_gdf.drop(['starting', 'ending'], axis=1)

fpth = '~/ipythonnotebooks/%s_sossml_stations.gpkg' % sos_label
stations_gdf_out.to_file(fpth, driver='GPKG')


I am working on the file "SensorMLparsing_IOOSSOS.ipynb".
The first task is:
Request and parse SensorML documents (and possibly SOS GetCapabilities)
Can you write Python code for it?

stations_df = get_stations_df(sos_endpoints[sos_label], station_urns)

# Assign EPSG:4326 CRS, retrieved from epsg.io
# The OGC WKT crs string is available directly at http://epsg.io/4326.wkt
# or http://spatialreference.org/ref/epsg/4326/ogcwkt/
crs = '''GEOGCS["WGS 84",
           DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],
             AUTHORITY["EPSG","6326"]],
           PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],
           UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],
         AUTHORITY["EPSG","4326"]]'
'''
geometry = [Point(xy) for xy in zip(stations_df.lon, stations_df.lat)]
stations_gdf = gpd.GeoDataFrame(stations_df, geometry=geometry, crs=crs)

stations_gdf.head()


I am working on the file "SensorMLparsing_IOOSSOS.ipynb".
The first task is:
Select SOS (sos_label) and optionally a list of station urn's. Using station_urns = None will result in querying the SOS GetCapabilities to extract all non-network, station offerings. Otherwise, pass a list of station_urns to process only those stations.
The 2nd of these two cells will be run, naturally
Can you write Python code for it?

sos_label = 'CeNCOOS'

station_urns = [
    'urn:ioos:station:cencoos:Tiburon',
    'urn:ioos:station:cencoos:Carquinez',
    'urn:ioos:station:cencoos:Trinidad',
    'urn:ioos:station:cencoos:Humboldt',
    'urn:ioos:station:cencoos:Monterey',
    'urn:ioos:station:cencoos:SantaCruz',
    'urn:ioos:station:mlml:mlml-sea'
    ]

#station_urns = ['urn:ioos:station:cencoos:Trinidad']
#station_urns = None

sos_label = 'NANOOS'

# station_urns = ['urn:ioos:station:nanoos:nerrs_sosecwq']
station_urns = None


I am working on the file "SensorMLparsing_IOOSSOS.ipynb".
The first task is:
Select SOS and optionally station urn's, the issue DescribeSensor requests
Can you write Python code for it?

sos_endpoints = {
    'CeNCOOS': 'http://sos.cencoos.org/sos/sos/kvp',
    'NANOOS': 'http://data.nanoos.org/52nsos/sos/kvp'
}


I am working on the file "SensorMLparsing_IOOSSOS.ipynb".
The first task is:
Workhorse function to issue requests and parse the GetCapabilities and SensorML responses
Can you write Python code for it?

def get_stations_df(sos_url, station_urns_sel=None):
    """ Returns a GeoDataFrame
    """
    # LATER: ADD ERROR TEST/CATCH AFTER EACH WEB REQUEST
    oFrmt = 'text/xml; subtype="sensorML/1.0.1/profiles/ioos_sos/1.0"'

    if station_urns_sel is not None:
        params = {'service': 'SOS', 'request': 'GetCapabilities', 'acceptVersions': '1.0.0'}
        sosgc = SensorObservationService(sos_url + '?' + urlencode(params))
        station_urns = station_urns_sel
    else:
        sos_collector = IoosSweSos(sos_url)
        station_urns = [urn.name for urn in sos_collector.server.offerings
                        if 'network' not in urn.name.split(':')]
        sos_collector.features = station_urns
        sml_lst = sos_collector.metadata(timeout=200)

    station_recs = []
    for station_idx, station_urn in enumerate(station_urns):
        if station_urns_sel is not None:
            sml_str = sosgc.describe_sensor(procedure=station_urn, outputFormat=oFrmt,
                                            timeout=200)
            sml = SensorML(sml_str)
        else:
            sml = sml_lst[station_idx]

        ds = IoosDescribeSensor(sml._root)

        pos = testXMLValue(ds.system.location.find(nsp('gml:Point/gml:pos')))

        system_el = sml._root.findall(nsp('sml:member'))[0].find(nsp('sml:System'))

        # Assume there's a single DocumentList/member; will read the first one only.
        # Assume that member corresponds to xlink:arcrole="urn:ogc:def:role:webPage"
        documents = system_el.findall(nsp('sml:documentation/sml:DocumentList/sml:member'))
        if len(documents) > 0:
            document = Documentation(documents[0])
            webpage_url = document.documents[0].url
        else:
            webpage_url = None

        contacts = system_el.findall(nsp('sml:contact/sml:ContactList/sml:member'))
        contacts_dct = {}
        for c in contacts:
            contact = Contact(c)
            role = contact.role.split('/')[-1]
            contacts_dct[role] =...


I am working on the file "SentiWordNet.ipynb".
The first task is:
First, we take a sentence and split it into tokens: "This red car is horrible. It is a very broken car. I want to punch it."
Can you write Python code for it?

article = "This red car is horrible. It is a very broken car. I want to punch it."
tokens = nltk.word_tokenize(article)
tokens


I am working on the file "SentiWordNet.ipynb".
The first task is:
For some reason this seems to eat up some words. For example, where is "horrible"? It turns out that the disambiguate() function doesn't return anything if the word does not have more than one meaning. We can try to remedy that:
Can you write Python code for it?

disambiguated_sentence = disambiguate(sentence)
for word, synset in disambiguated_sentence:
    if synset:
        senti_synset = swn.senti_synset(synset._name)
        print word, senti_synset, senti_synset.obj_score()
    else:
        senti_synsets = swn.senti_synsets(word)
        if len(senti_synsets) == 1:
            print word, senti_synsets[0], senti_synsets[0].obj_score()


I am working on the file "SentiWordNet.ipynb".
The first task is:
Now let's try it on one word, "plant". Specifically, "plant.n.01":
Can you write Python code for it?

plant01 = swn.senti_synset("plant.n.01")

plant01.neg_score(), plant01.pos_score(), plant01.obj_score()


I am working on the file "SentiWordNet.ipynb".
The first task is:
Now that we have that, we can try looking up the sentiments for each token in the sentence in SentiWordNet, and print the top results for meanings that we find, along with the definition, the positive score, the negative score and the objectivity score. These scores are normalized such that they add up to 1.
Can you write Python code for it?

for word, pos in tagged_tokens:

    swn_pos = nltk_to_sentiwordnet.get(pos[:2], None)

    if swn_pos == None:
        continue

    synsets = swn.senti_synsets(word.lower(), pos=swn_pos)

    if len(synsets) == 0:
        continue

    print("{}:".format(word))
    for synset in synsets[:3]:
        print(synset, synset.pos_score(), synset.neg_score(), synset.obj_score())
        print "    ", synset.synset.definition()
    print("------")


I am working on the file "SentiWordNet.ipynb".
The first task is:
Still some stuff is missing. Where is "red"? Since this doesn't seem to be giving great results, and our initial prediction is that it'll have a marginal benefit anyway, I'll opt to not use WSD.
A full implementation of SentiWordNet scoring
Can you write Python code for it?

import nltk
from nltk.corpus import stopwords
from nltk.corpus import sentiwordnet as swn
import numpy as np

#nltk.download("sentiwordnet")
#nltk.download("stopwords")

flatten = lambda l: [item for sublist in l for item in sublist]
english_stopwords = set(stopwords.words("english"))

nltk_to_sentiwordnet = {
    "NN": "n",
    "VB": "v",
    "JJ": "a",
    "RB": "r",
}

def get_sentiment(article):

    sentences = nltk.sent_tokenize(article)
    sentence_words = [nltk.word_tokenize(sentence) for sentence in sentences]
    tagged_sentence_words = flatten(nltk.pos_tag_sents(sentence_words))

    # Filter stopwords
    tagged_sentence_words = [word for word in tagged_sentence_words if word[1] not in english_stopwords]

    pos_scores = []
    neg_scores = []
    subj_scores = []

    for word, pos in tagged_sentence_words:

        swn_pos = nltk_to_sentiwordnet.get(pos[:2], None)

        if swn_pos == None:
            continue

        synsets = swn.senti_synsets(word.lower(), pos=swn_pos)

        if len(synsets) == 0:
            continue

        #print("{}:".format(word))
        for synset in synsets[:1]:
            pos_scores.append(synset.pos_score())
            neg_scores.append(synset.neg_score())
            subj_scores.append(1 - synset.obj_score())

    return np.average(pos_scores, weights=subj_scores) , np.average(neg_scores, weights=subj_scores), np.mean(subj_scores)

get_sentiment(article)


I am working on the file "SentiWordNet.ipynb".
The first task is:
The results are approximately sorted in frequency of usage. Now, we can use the SWN dictionary like so, when parsing actual sentences:
Can you write Python code for it?

from nltk.tag.perceptron import PerceptronTagger
from nltk.tokenize import word_tokenize


I am working on the file "SentiWordNet.ipynb".
The first task is:
Then we tag these tokens according to what parts of speech they might be (noun, verb, etc):
Can you write Python code for it?

tagged_tokens = nltk.pos_tag(tokens)
tagged_tokens


I am working on the file "SentiWordNet.ipynb".
The first task is:
This notebook requires not only NLTK, but also additional corpora that come with NLTK to perform various functions:
Can you write Python code for it?

# To download the SentiWordNet and stopword corpora, run the commands below
nltk.download("sentiwordnet")
nltk.download("stopwords")


I am working on the file "SentiWordNet.ipynb".
The first task is:
You'll notice that this is a very specific "sense" of a word. "plant.n.01" means "the first meaning of the word "plant", that is a noun. Wordnet provides many meanings of words. You can search for multiple, like so.
Can you write Python code for it?


I am working on the file "Sentiment Analysis on Movie Reviews/Sentiment-Analysis-on-Movie-Reviews-Detail-using-multiple-models.ipynb".
The first task is:
Building a Pipeline
In order to make the vectorizer => transformer => classifier easier to work with, scikit-learn provides a Pipeline class that behaves like a compound classifier.
You can compare the above accuracy result of the classifier without using Pipeline and the below accuracy result of the classifier while using Pipeline class. It's the same. Hence, Pipeline class highly simplifies our task of tokenizing and tfidf conversion.
Can you write Python code for it?

text_clf = Pipeline([
    ('vect', CountVectorizer()),
    ('tfidf', TfidfTransformer()),
    ('clf', MultinomialNB()),
])

text_clf.fit(X_train, y_train)

predicted = text_clf.predict(X_train)

np.mean(predicted == y_train)


I am working on the file "Sentiment Analysis on Movie Reviews/Sentiment-Analysis-on-Movie-Reviews-Detail-using-multiple-models.ipynb".
The first task is:
Create Submission
Can you write Python code for it?

test.info()

X_test = test['Phrase']
phraseIds = test['PhraseId']
predicted = text_clf.predict(X_test)
output = pd.DataFrame( data={"PhraseId":phraseIds, "Sentiment":predicted} )
#output.to_csv( "submission.csv", index=False, quoting=3 )


I am working on the file "Sentiment Analysis on Movie Reviews/Sentiment-Analysis-on-Movie-Reviews-Detail-using-multiple-models.ipynb".
The first task is:
Extracting features
In order to perform machine learning on text documents, we first need to turn the text content into numerical feature vectors.
Bags of words
The most intuitive way to do so is the bags of words representation:

assign a fixed integer id to each word occurring in any document of the training set (for instance by building a dictionary from words to integer indices).

for each document $#i$, count the number of occurrences of each word $w$ and store it in $X[i, j]$ as the value of feature $#j$ where $j$ is the index of word $w$ in the dictionary

Reference: http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html
The Bag of Words model learns a vocabulary from all of the documents, then models each document by counting the number of times each word appears.
We'll be using the CountVectorizer feature extractor module from scikit-learn to create bag-of-words features.
Can you write Python code for it?

X_train = train['Phrase']
y_train = train['Sentiment']

# Convert a collection of text documents to a matrix of token counts
count_vect = CountVectorizer()

# Fit followed by Transform
# Learn the vocabulary dictionary and return term-document matrix
X_train_counts = count_vect.fit_transform(X_train)

#X_train_count = X_train_count.toarray()

# 156060 rows of train data & 15240 features (one for each vocabulary word)
X_train_counts.shape

# get all words in the vocabulary
vocab = count_vect.get_feature_names()
print (vocab)

# get index of any word
count_vect.vocabulary_.get(u'100')

# Sum up the counts of each vocabulary word
dist = np.sum(X_train_counts, axis=0)
# print (dist) # matrix

dist = np.squeeze(np.asarray(dist))
print (dist) # array

zipped = zip(vocab, dist)
zipped.sort(key = lambda t: t[1], reverse=True) # sort words by highest number of occurrence

# For each, print the vocabulary word and the number of times it
# appears in the training set
for tag, count in zipped:
    print (count, tag)


I am working on the file "Sentiment Analysis on Movie Reviews/Sentiment-Analysis-on-Movie-Reviews-Detail-using-multiple-models.ipynb".
The first task is:
In the above code, we first used the fit() method to fit our estimator and then the transform() method to transform our count-matrix to a tf-idf representation.
These two steps can be combined using fit_transform() method.
Can you write Python code for it?

tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
X_train_tfidf.shape


I am working on the file "Sentiment Analysis on Movie Reviews/Sentiment-Analysis-on-Movie-Reviews-Detail-using-multiple-models.ipynb".
The first task is:
Logistic Regression Classifier
Can you write Python code for it?

text_clf = Pipeline([
    ('vect', CountVectorizer(stop_words='english', max_features=5000)),
    ('tfidf', TfidfTransformer()),
    ('clf', LogisticRegression())
])

text_clf.fit(X_train, y_train)
predicted = text_clf.predict(X_train)
np.mean(predicted == y_train)


I am working on the file "Sentiment Analysis on Movie Reviews/Sentiment-Analysis-on-Movie-Reviews-Detail-using-multiple-models.ipynb".
The first task is:
Train Classifier
We train our classifier by inputing our features and expecting our classifier to output/predict the sentiment value for each phrase in test dataset.
Naive Bayes Classifier
Can you write Python code for it?

clf = MultinomialNB().fit(X_train_tfidf, y_train)

predicted = clf.predict(X_train_tfidf)

np.mean(predicted == y_train)


I am working on the file "Sentiment Analysis on Movie Reviews/Sentiment-Analysis-on-Movie-Reviews-Logistic-Regression.ipynb".
The first task is:
Create Submission
Can you write Python code for it?

X_test = test['Phrase']
phraseIds = test['PhraseId']
predicted = text_clf.predict(X_test)
output = pd.DataFrame( data={"PhraseId":phraseIds, "Sentiment":predicted} )
#output.to_csv( "submission_logistic_regression.csv", index=False, quoting=3 )


I am working on the file "Sentiment Analysis on Movie Reviews/Sentiment-Analysis-on-Movie-Reviews-Logistic-Regression.ipynb".
The first task is:
Train Classifier
Can you write Python code for it?

X_train = train['Phrase']
y_train = train['Sentiment']

text_clf = Pipeline([
    ('vect', CountVectorizer()),
    ('tfidf', TfidfTransformer()),
    ('clf', LogisticRegression())
])

text_clf = text_clf.fit(X_train, y_train)

X_test = train['Phrase']
predicted = text_clf.predict(X_test)

print (np.mean(predicted == y_train))

test.info()


I am working on the file "Sentiment Analysis on Movie Reviews/Sentiment-Analysis-on-Movie-Reviews-RNN-LSTM-Kaggle.ipynb".
The first task is:
Creating Submission
Can you write Python code for it?

test_pred = model.predict_classes(word_id_test)

test_pred

#make a submission
test['Sentiment'] = test_pred.reshape(-1,1)
header = ['PhraseId', 'Sentiment']
test.to_csv('./submission_lstm_cnn.csv', columns=header, index=False, header=True)


I am working on the file "Sentiment Analysis on Movie Reviews/Sentiment-Analysis-on-Movie-Reviews-RNN-LSTM-Kaggle.ipynb".
The first task is:
Padding Sequences
Can you write Python code for it?

#pad sequences
word_id_train = sequence.pad_sequences(np.array(word_id_train), maxlen=seq_len)
word_id_test = sequence.pad_sequences(np.array(word_id_test), maxlen=seq_len)
y_train_enc = np_utils.to_categorical(sentiment_train, num_labels)

print (word_id_train)

print (y_train_enc)


I am working on the file "Sentiment Analysis on Movie Reviews/Sentiment-Analysis-on-Movie-Reviews-RNN-LSTM-Kaggle.ipynb".
The first task is:
Training LSTM RNN Model
Long short-term memory (LSTM) is a recurrent neural network (RNN) architecture that remembers values over arbitrary intervals. Stored values are not modified as learning proceeds. RNNs allow forward and backward connections between neurons.
An LSTM network contains LSTM units instead of, or in addition to, other network units. An LSTM unit remembers values for either long or short time periods. The key to this ability is that it uses no activation function within its recurrent components. Thus, the stored value is not iteratively modified and the gradient does not tend to vanish when trained with backpropagation through time.
keras.layers.recurrent.LSTM(units, activation='tanh', recurrent_activation='hard_sigmoid', use_bias=True, kernel_initializer='glorot_uniform', recurrent_initializer='orthogonal', bias_initializer='zeros', unit_forget_bias=True, kernel_regularizer=None, recurrent_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, recurrent_constraint=None, bias_constraint=None, dropout=0.0, recurrent_dropout=0.0)

units: Positive integer, dimensionality of the output space.
dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the inputs.
recurrent_dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the recurrent state.
Source: https://keras.io/layers/recurrent/#lstm

Embedding Layer
Embedding Layer is used to:

One-hot encoded vectors are high-dimensional and sparse. Let's assume that we are doing Natural Language Processing (NLP) and have a dictionary of 2000 words. This means that, when using one-hot encoding, each word will be represented by a vector containing 2000 integers. And 1999 of these integers are zeros. In a big dataset this approach is not computationally efficient.

The vectors of each embedding get updated while training the neural network. This allows us to visualize relationships between words, but also between everything that can be...
Can you write Python code for it?

#LSTM
print "fitting LSTM ..."
model = Sequential()
model.add(Embedding(dictionary_size, 128))
model.add(Dropout(0.2))
model.add(LSTM(128))
model.add(Dropout(0.2))
model.add(Dense(num_labels))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

model.fit(word_id_train, y_train_enc, epochs=3, batch_size=256, verbose=1)


I am working on the file "Sentiment Analysis.ipynb".
The first task is:
Get some information about the predictions
Can you write Python code for it?

print ("Number of tweets of classfied for each party")
tweets_clean.Party.value_counts().head()

print ("Number of tweets of classfied for each party")
tweets_clean[tweets_clean.Sentiment=='Positive'].Party.value_counts().head()

print ("Number of tweets of classfied for each party")
tweets_clean[tweets_clean.Sentiment=='Negative'].Party.value_counts().head()


I am working on the file "Sentiment Analysis.ipynb".
The first task is:
Import training data
Can you write Python code for it?

with open("/resources/data/classified_tweets.txt", "r",encoding="utf8") as myfile:
     data = myfile.readlines()