I am working on the file "2_Classification/Classification II-student.ipynb".
The first task is:
Advanced work
Complete the following code to implement the CV process without using GridSearchCV().
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

from sklearn import neighbors
from sklearn.cross_validation import StratifiedKFold

# Parameters
K_max = 20
rang_K = np.arange(1, K_max+1)
nfold = 10
N_tr, N_dim = X_train.shape

# Create data partitions (check the structure of folds)
folds = StratifiedKFold(Y_train, n_folds=nfold, shuffle=True)

# Create variable (acc_val) to save the validation results
acc_val = np.zeros([nfold, K_max])
for f, (train, val) in enumerate(folds):
    # f is an index with the number of fold
    # train has the training data positions for fold f
    # val has the validation data positions for fold f

    # Create training and validation partitions for fold f
    X_tr_f = # <FILL IN>
    Y_tr_f = # <FILL IN>
    X_val_f = # <FILL IN>
    Y_val_f = # <FILL IN>

    # Explore the values of K
    for K, n_neighbors in enumerate(rang_K):
        # Train a K-NN with the training data of this fold data using n_neighbors
        clf = # <FILL IN>
        # Evaluate its accuaracy over the validation data of this fold and save it in acc_val
        acc_val[f, K] = # <FILL IN>

# Compute the average validation error (average over the different folds)
acc_val_med = # <FILL IN>
# Select the optimum value of K as the value which achieves the maximum average validation error
K_opt = # <FILL IN>

# Train the final K-NN classfier (with K_opt) and obtain its test error
clf = # <FILL IN>
acc_test = # <FILL IN>

print "K_NN with optimum K = " + np.array_str(K_opt) + \
    ". Test error = " + np.array_str(acc_test)

###########################################################
# TEST CELL
###########################################################
from test_helper import Test

# TEST accuracy values
Test.assertEquals(K_opt, 6, 'incorrect result: The value of K_opt is uncorrect')
Test.assertEquals(np.round(KNN_acc_test,4), 0.9444, 'incorrect result: Test error of K-NN after CV...


I am working on the file "2_Classification/Classification II-student.ipynb".
The first task is:
After running the CV process, the classifier object  contains the information of the CV process (next cell explore the parameter ".grid_scores_" to obtain this information).
Can you write Python code for it?

# Printing results
print("Cross validation results:")
for params, mean_score, scores in clf.grid_scores_:
    print("For K = %d, validation accuracy is %2.2f (+/-%1.3f)%%"
          % (params['n_neighbors'], 100*mean_score, 100*scores.std() / 2))

# Selecting validation error (mean values)
vect_val=[CV_result[1] for CV_result in clf.grid_scores_]

# Ploting results
plt.figure()
plt.plot(rang_K,vect_tr,'b', label='Training accuracy')
plt.plot(rang_K,vect_test,'r', label='Test accuracy')
plt.plot(rang_K,vect_val,'g', label='Validation accuracy')
plt.legend()
plt.xlabel('K value')
plt.ylabel('Accuracy')
plt.title('Evolution of K-NN accuracy (including validation result)')
plt.show()


I am working on the file "2_Classification/Classification II-student.ipynb".
The first task is:
Examine the fields ".best_estimator_" and ".best_params_" of the classifier generated by the CV process:
* ".best_estimator_" contains  the final classifier trained with this select value.
* ".best_params_" is a dictionary with the selected parameters. In our example, "best_params_['n_neighbors']" would provide the selected value of K.
Save the selected value of K in variable denoted "K_opt" and compute the test error of the final classifier.
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

# Assign to K_opt the value of K selected by CV
K_opt = # <FILL IN>
print("The value optimum of K is %d" %(K_opt))

# Select the final classifier  and compute its test error
KNN_acc_test = # <FILL IN>
print("The test accuracy is %2.2f" %(100*KNN_acc_test))


I am working on the file "2_Classification/Classification II-student.ipynb".
The first task is:
Include the necessary code to plot their classification boundaries
Can you write Python code for it?


I am working on the file "2_Classification/Classification II-student.ipynb".
The first task is:
Next code, let you plot the evolution of above computed train and test accuracies.
Can you write Python code for it?

#Plot acc vs K
import matplotlib.pyplot as plt

plt.figure()
plt.plot(rang_K,vect_tr,'b', label='Training accuracy')
plt.plot(rang_K,vect_test,'r', label='Test accuracy')
plt.legend()
plt.xlabel('K value')
plt.ylabel('Accuracy')
plt.title('Evolution of K-NN accuracy')
plt.show()


I am working on the file "2_Classification/Classification II-student.ipynb".
The first task is:
Part 1: K-NN
A K-NN approach classifies each new data searching its K nearest neighbors (among the training data) and assigning the majority class among these neighbors. As expected, its performance depends on the number of neighbors (K) used.
 1.1: Training a K-NN classifier
To start to work, let's analyze for different values of K the K-NN performance, both over training and test sets. Use the KNeighborsClassifier() method to complete the below code.
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

from sklearn import neighbors

K_max=20
rang_K = np.arange(1, K_max+1)
vect_tr=[]
vect_test=[]

for n_neighbors in rang_K:
    # Create a KNN classifier, train it and compute training and error accuracies.
    clf = #<FILL IN>
    acc_tr = #<FILL IN>
    acc_test = #<FILL IN>

    # Saving accuracies in a list
    vect_tr#<FILL IN>
    vect_test#<FILL IN>

    print("For K = %d, train accuracy is %2.2f%% and test accuracy is %2.2f%%"
          % (n_neighbors, 100*acc_tr, 100*acc_test))

###########################################################
# TEST CELL
###########################################################
from test_helper import Test

# TEST accuracy values
Test.assertEquals(np.round(np.sum(np.array(vect_tr)),2), 18.90, 'incorrect result: Training error of K-NN is uncorrect')
Test.assertEquals(np.round(np.sum(np.array(vect_test)),2), 18.37, 'incorrect result: Test error of K-NN is uncorrect')


I am working on the file "2_Classification/Classification II-student.ipynb".
The first task is:
Part 2: SVM
SVM is one of the most well-known classifiers due to its good generalization properties in many different applications. Besides, by means of the kernel trick, its linear formulation can easily extended to a non linear fashion.
Here, we will test its performance when different kernel functions are used. For this purpose, we can use the SCV( ) method, which let you select the kernel function to be used, and the method GridSearchCV( ) to adjust the different free parameters (C and kernel parameter).
Complete the following cells, when it is required, to train in each case a linear SVM (defining kernel='linear' in the method SCV( )), an SVM with gaussian kernel (kernel='rbf') and an SVM with polynomial kernel (kernel='poly').
For each method, adjust the corresponding free parameters with a 10 fold CV process (the ranges to explore are defined at the beginning of each cell). Return the values of selected parameters and the accuracy of the final SVM.
 2.1: Linear SVM
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

from sklearn import svm
rang_C = np.logspace(-3, 3, 10)
tuned_parameters = [{'C': rang_C}]

nfold = 10

# Train a liner SVM and adjust by CV the parameter C
clf_base = svm.SVC(kernel='linear')
lin_svc  = # <FILL IN>
lin_svc.# <FILL IN>

# Save the value of C selected and compute the final accuracy
C_opt = # <FILL IN>
acc_lin_svc = # <FILL IN>

print "The C value selected is " + str(C_opt)
print("The test accuracy of the linear SVM is %2.2f" %(100*acc_lin_svc))

###########################################################
# TEST CELL
###########################################################
from test_helper import Test

# TEST accuracy values
Test.assertEquals(np.round(C_opt,2), 0.46, 'incorrect result: The value of C_opt is uncorrect')
Test.assertEquals(np.round(acc_lin_svc,4), 0.9556, 'incorrect result: Test accuracy of the linear SVM after CV process is uncorrect')


I am working on the file "2_Classification/Classification III-student.ipynb".
The first task is:
Part 0: Load and preprocess data
In the following sections, we are going to use all input features of the Iris dataset. So, let's start running the following cell to load the complete Iris data.
Can you write Python code for it?

import numpy as np
from sklearn import datasets
from sklearn.cross_validation import train_test_split
from sklearn.preprocessing import StandardScaler

# Initialize the random generator seed to compare results
np.random.seed(0)

iris = datasets.load_iris()
X = iris.data # All input features are used
Y = iris.target

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=.4)

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)


I am working on the file "2_Classification/Classification III-student.ipynb".
The first task is:
Part 1: Trees
 1.1: Training a decision Tree
Decision Trees learn simple decision rules selecting iteratively a input feature and setting a threshold over it, so the are simple tool to understand and to interpret.
Use the DecisionTreeClassifier( ) function to train a decision tree. Although the tree depth is usually a parameter to select, here we are working with only for input features, so you can use all default parameter and obtain a good performance. Complete the following code to return in the variable acc_tree the tree accuracy.
Can you write Python code for it?


I am working on the file "2_Classification/Classification III-student.ipynb".
The first task is:
Part 3: Ensembles
The goal of ensemble methods is to combine the predictions of several base estimators or learners to obtain a classifier of improved performance. We are going to work with two ensemble methods:

Bagging methods: their driving principle is to build several estimators with diversity among them and then to average their predictions.
Boosting methods: in this case, base estimators are built sequentially forcing new learners to pay more attention to samples missclassified by previous learners.

 3.1. Bagging methods
Here, to implement bagged classifiers, we are going to use BaggingClassifier( ) object which includes different degrees of freedom in the learners design: with or without samples replacement, selecting random subsets of features instead of samples or selecting subsets of both samples and features.
For the sake of simplicity, we are going to use as base learner a decision stump (i.e., a decision tree with one depth level). Note that in the case of using decision trees as learners, the resulting ensemble results in a random forest.
Complete the following code to train a ensemble of bagged decision stumps. Set max_samples (percentage of training data used to train each learner) and max_features parameters (percentage of input features used to train each learner) to 0.5, and fix to 10 the number of learners used.
Can you write Python code for it?

from sklearn.ensemble import BaggingClassifier
from sklearn import tree
base_learner = tree.DecisionTreeClassifier(max_depth=1)
bagging = BaggingClassifier(base_learner, n_estimators = 10, max_samples=0.5, max_features = 0.5)
bagging.fit(X_train, Y_train)
acc_test = bagging.score(X_test, Y_test)

print('Accuracy of bagged ensemble is %2.2f '%(100*acc_test))


I am working on the file "2_Classification/Classification III-student.ipynb".
The first task is:
Run the above code again, do you obtain the same accuracy?
 2.2: Obtaining results statistically significant
Random forest have a random component when the training data are subsampled, so you can obtain a different result for different runnings of the algorithm. In this case, to be able to provide a statistically significant measurement of the performance of the classifier, we need to average the result over a large number of runs.
Complete the following code, to train again the RF classifier, but averaging its test accuracies over 50 runs. Provide its average accuracy and the average number of selected trees (include their standard deviations).
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################
# Initialize the random generator seed to compare results
np.random.seed(0)

print 'This can take a some minutes, be patient'
# Create RF classifier object with CV
clf_RF  = # <FILL IN>

acc_RF_vector=[]
n_trees_vector=[]
for run in np.arange(50):
    # For each run, train it, compute its accuracy and examine the number of optimal trees
    clf_RF.# <FILL IN>
    acc = # <FILL IN>
    acc_RF_vector.append(acc)
    n_trees = # <FILL IN>
    n_trees_vector.append(n_trees)

# Compute averaged accuracies and number of used trees
mean_acc_RF = # <FILL IN>
std_acc_RF = # <FILL IN>

mean_n_trees = # <FILL IN>
std_n_trees = # <FILL IN>

# Print the results
print('Averaged accuracy for RF classifier is %2.2f +/- %2.2f '%(100*mean_acc_RF, 100*std_acc_RF))
print('Averaged number of selected trees is %2.2f +/- %2.2f '%(mean_n_trees, std_n_trees))

###########################################################
# TEST CELL
###########################################################
from test_helper import Test

Test.assertEquals(np.round(mean_acc_RF, 1), 0.9 , 'incorrect result: The value of mean_acc_RF is uncorrect')
Test.assertEquals(np.round(std_acc_RF, 2), 0.03 , 'incorrect result: The value of std_acc_RF is uncorrect')
Test.assertEquals(np.round(mean_n_trees, 1), 4.2 , 'incorrect result: The value of mean_n_trees is uncorrect')
Test.assertEquals(np.round(std_n_trees, 1), 2.0 , 'incorrect result: The value of std_n_trees is uncorrect')


I am working on the file "2_Classification/Classification III-student.ipynb".
The first task is:
Try to use the following example of the scikit-learn help, to plot the classification regions for different pairs of input features. Modify the necessary code line to plot our training data over the decision regions.
Be careful, this examples retrains different classifiers for each pair of input features; therefore, its solution differs from the above one that we have just computed.
 Part 2: Random Forest
 2.1: Training a Random Forest
A Random Forest (RF) trains several decision tree classifiers, where each one is trained with different sub-samples of the training data, and averages their outputs to improve the final accuracy.
Use the RandomForestClassifier( ) function to train a RF classifier and select by cross validation the number of trees to use. The remaining parameters, such as the number of subsampled data or features, can be used with their default values. Return the optimal number of trees to be used and the final accuracy of the RF classifier.
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################
from sklearn.grid_search import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

rang_n_trees=np.arange(1,10)
tuned_parameters = [{'n_estimators': rang_n_trees}]
nfold = 10

clf_RF  = #<FILL IN>
n_trees_opt = #<FILL IN>
acc_RF = #<FILL IN>

print "The number of selected trees is " + str(n_trees_opt)
print("The test accuracy of the RF is %2.2f" %(100*acc_RF))


I am working on the file "2_Classification/Classification III-student.ipynb".
The first task is:
Unlike BaggingClassifier() method, AdaBoostClassifier() let you analyze the evolution of error without having to train the ensemble for different number of learners. For this task, you can use the classifier method .staged_score() which returns the evolution of the ensemble accuracy. Note that it returns this information with a generator object, so you have to iterate over it to access to each element.
The following code lines let you plot the evolution of the ensemble accuracy (over the test data) for both discrete and real Adaboost approaches.
Can you write Python code for it?

acc_AB_D_evol=[acc for acc in AB_D.staged_score(X_test, Y_test)]
acc_AB_R_evol=[acc for acc in AB_R.staged_score(X_test, Y_test)]

# Ploting results
rang_n_learners=np.arange(50)+1
plt.figure()
plt.subplot(211)
plt.plot(rang_n_learners,acc_AB_D_evol)
plt.xlabel('Number of learners')
plt.ylabel('Accuracy')
plt.title('Discrete AB accuracy')
plt.subplot(212)
plt.plot(rang_n_learners,acc_AB_R_evol)
plt.xlabel('Number of learners')
plt.ylabel('Accuracy')
plt.title('Real AB accuracy')
plt.show()


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
2. Let's create a function to compute the output of a single data
Can you write Python code for it?

#################################################
# TODO: Replace <FILL IN> with appropriate code
#################################################

import numpy as np
def output_1vsall_SVM(data, list_models):
    """Compute the output of a list of 1 vs. all SVM models for a test data

    Args:
        data (labeled point): data to be evaluated over the 1 vs. all SVM model
        list_models: a list of length number of classes where each element is a tuple (lab, model). Variable
            lab indicates the label 1 of the 1 vs. all problem and model is the SVM model solving the problem
    Returns:
        Output_label: The label estimated by the 1 vs. all model for the data
    """
    # Split the tuples of list_models into a list of labels and a list of models (you can use zip() method)
    labels, models = # FILL IN

    outputs = []
    # For each model...
    for model in models:
        # Compute the output over the test data
        out = # FILL IN
        # Add this output to outputs list
        # FILL IN

    # Get the test output label as the label associated to the model with the maximum output value
    pos = # FILL IN
    Output_label = # FILL IN
    return Output_label


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
2. Model training
MLLIb includes several classification methods. Most of them are only implemented for solving binary problems. As we intend to solve a multiclass problem, let's start using the classifiers with multiclass implementations:
* Decision trees
* Random Forest
Decision trees
As we already know, a decision tree works by selecting the most discriminative features and setting different threholds over them, in such a way, that each tree node splits the training data in different subsets with the aim to find the node purity (the data partitions belongs to a single class). For this purpose, a purity measure, such as the gini index, is used to select both the most discriminative features and the threshold to apply.
Review http://spark.apache.org/docs/latest/mllib-decision-tree.html for implementation details.
The following cell contains the necessary code to train a DecisionTree and creates in the variable model a DecisionTreeModel. Note that all free parameters (such as the maximum  depth of the tree) have been set by default.
Can you write Python code for it?

from pyspark.mllib.tree import DecisionTree

#  Train a DecisionTree model
#  Empty categoricalFeaturesInfo indicates all features are continuous.
model_tree = DecisionTree.trainClassifier(trainingData, numClasses=10, categoricalFeaturesInfo={},
                                     impurity='gini', maxDepth=5, maxBins=32)


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
3. Let's create a function to compute the error over a set of data
Note that the SVM final output is a real value, so you can compute the number of errors as the number of products label$\times$output lower than zero.
Can you write Python code for it?

#################################################
# TODO: Replace <FILL IN> with appropriate code
#################################################

from pyspark.mllib.util import MLUtils

def compute_1vsall_SVM_error(list_models, Data):

    """ Compute the classification error of the 1 vs. all SVM model over the samples given in Data.

    Args:
       list_models: a list of length number of classes where each element is a tuple (lab, model). Variable
            lab indicates the label 1 of the 1 vs. all problem and model is the SVM model solving the problem
       Data: an RDD with a data set of LabelPoints

    Returns:
        Int: A single value, between 0 and 1, indicating the classification error.
        A value of 1 indicates that all the samples are missclassfied and a value
        of 0 that all the samples are correctly classified.
    """

    # Evaluate model on test instances and compute test error
    # 1. Compute the model output
    predictions = # FILL IN
    # 2. Create an RDD of tuples (label, output)
    labelsAndPredictions = # FILL IN
    # 3. Compute test error
    testErr = # FILL IN

    return testErr


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
4. Evaluate the classification performance after removing useless pixels
Here, let's use a tree with default parameters. For comparison purposes, remember that the test error using all the features is around 30%.
Can you write Python code for it?

from pyspark.mllib.tree import DecisionTree
#  Train a DecisionTree model with selected features
model_tree = DecisionTree.trainClassifier(trainingData_sel, numClasses=10, categoricalFeaturesInfo={},
                                     impurity='gini', maxDepth=5, maxBins=32)

# Test error of the decision tree
tree_testErr = compute_classifier_error(model_tree, testData_sel)
print('Tree test error = ' + str(tree_testErr))

###########################################################
# TEST CELL
###########################################################

import numpy as np
from test_helper import Test

Test.assertEquals(np.round(100*tree_testErr,0), 31, 'incorrect result: test error is uncorrect')


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
4. Selecting the model parameters by cross-validation
Until now, both tree and RF model have used predefined parameter values. Here, we are going to adjust the free parameters of these models by cross validation.
Decision trees: adjusting the tree depth
The most critical parameter of a decision tree is its maximum depth. Note that deeper trees are more expressive (potentially allowing higher accuracy), but they are also more costly to train and are more likely to overfit.
The following cell explores different tree depths and evaluate its over the validation data to, finally, select the optimum tree depth as the depth value which provides the minimum classification error over the validation data. Complete the missing code lines.
Can you write Python code for it?

#################################################
# TODO: Replace <FILL IN> with appropriate code
#################################################

from pyspark.mllib.tree import DecisionTree
import numpy as np

# Initialize variables
bestModel = # FILL IN
best_error = # FILL IN
best_depth = # FILL IN
# Range of depth values to explore
depth_params = [5, 10, 15]

for depth_value in depth_params:
    # Train a decision tree fixing maxDepth to depth_value
    model_tree = DecisionTree.trainClassifier(trainingData, numClasses=10, categoricalFeaturesInfo={},
                                     impurity='gini', maxDepth=#FILL IN, maxBins=32)

    # Compute the model error over the validation data (use compute_classifier_error function)
    tree_valErr = # FILL IN

    print 'Tree depth is ' + str(depth_value) + ' and the validation error is '+ str(tree_valErr)

    # If the error has reduced, save the model, the optimum depth and error in bestModel, best_depth and best_error variables
    if (tree_valErr < best_error):
            bestModel = # FILL IN
            best_depth = # FILL IN
            best_error = # FILL IN

print 'Optimum tree depth: ' + str(best_depth)


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
5. Creating a multiclass classifier from a binary Support Vector Machine
MLLib includes a distributed SVM implementation, but it is only avaliable for binary problems (see MLLIb help at: http://spark.apache.org/docs/latest/mllib-linear-methods.html#linear-support-vector-machines-svms)
As we are working with a multiclass problem, let's adapt this implementation to be used in a  1vs. all fashion and aplly it to our multiclass problem.
Solving  a single 1 vs. all problem
Let's start considering a single 1 vs. all problem, for instance, let's consider that we want to classify the digit '0' from the remaining digits.
Then, we will procced as follows:
1. Convert the training labels to the 1 vs. all scheme
2. Train a binary SVM model with the new labels
3. Evaluate the model
1. Convert the training labels to the 1 vs. all scheme
Create a convert_label() funtion to generate the labels associated to the 1 vs. all problem
Can you write Python code for it?

#################################################
# TODO: Replace <FILL IN> with appropriate code
#################################################

def convert_label(label, label1):
    """Produce a 1 vs. all label encoding for a single label and the label to be included in the class 1.

    Args:
        label (int, str): the label to be coded
        label1 (int, str): the label to be included in the class 1.

    Returns:
        Int: A single value indicating the label (0 or 1) of the 1 vs. all problem.
    """
    # <FILL IN> : use all code lines that you need


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
6. Interpretability analysis: feature selection approaches
The MNIST dataset has a lot of features (pixels) which are useless for the classification, for instance, some of the are constant overall the data, so they haven't any discriminatory capability.
In this last section, let's implement some easy (but efficient) distribuited feature selection approaches, so that we can extract the most relevant features.
6.1 Remove features with zero variance
Let's start removing all pixels which are zero over all the images (background pixels), that is, its variance is zero. For this purpose, follow these steps:
1. Comupute the number of non zeros in each pixel
2. Select variables to keep
3. Create a function which removes the no desired features from the data
4. Evaluate the classification performance after removing useless pixels.
 1. Compute the number of zeros in each pixel
Use Statistics.colStats() funtion to compute the number of non zeros in each pixel (review first section of this notebook).
Note: we will compute this over the training data  and, later, apply the selection over trainign, validation and test data sets.
Can you write Python code for it?

#################################################
# TODO: Replace <FILL IN> with appropriate code
#################################################

from pyspark.mllib.stat import Statistics
features = trainingData.map(lambda x: x.features)

# Compute column summary statistics.
stats = # FILL IN
plot_data([stats.numNonzeros()], h, w)


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
6.2 Remove features with L1 regularization: L1-SVM
As we know, L1 regularization provides sparsity over the vector weights. If we have a linear model, this can provide a feature selection.
So, in this section, let's use the above multiclass SVM implemention using an L1 regularization. In this way, we will obtain both a classifier and a feature selection.
Can you write Python code for it?

#################################################
# TODO: Replace <FILL IN> with appropriate code
#################################################

# Initialize variables
C_value = 0.1

# Train the 1 vs. all SVM models (set regularization parameter to C_value)
# Note that we have added the parameter regType='l1'
multiclass_SVM = train_1vsall_SVM(trainingData, regParam=C_value, regType='l1')

# Compute the model error over the test data
error_1vsall = # FILL IN

# Analyze the number of zero weigths: compute the positions where all the vectors are zero
# multiclass_SVM is a list of tuples (class, model), getting the parameter
# model.weights you can access to the vector weights of each SVM

pos_sel = # FILL IN (You may need several lines to compute this)

print 'El número de variables seleccionadas son: ' + str(len(pos_sel))

###########################################################
# TEST CELL
###########################################################
import numpy as np
from test_helper import Test

Test.assertEquals(sum(pos_sel), 178643, 'incorrect result: test error is uncorrect')


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
6.2 Remove features with a low variance
Now, let's modify the code of the previous section to remove the features which variance is lower than a given threhold.
Complete the next cell following the indications.
Can you write Python code for it?

#################################################
# TODO: Replace <FILL IN> with appropriate code
#################################################

import numpy as np
from pyspark.mllib.regression import LabeledPoint
from pyspark.mllib.stat import Statistics
from pyspark.mllib.tree import DecisionTree

# Set a variance threshold
th_var = 0.5

# 1. Compute the variance with Statistics.colStats( )
variance = # FILL IN (you may need several code lines to compute this)

# 2. Get the positions of the features to keep
idx_keep = # FILL IN

# 3. Remove features from training, validation and test data
trainingData_sel = # FILL IN
valData_sel = # FILL IN
testData_sel = # FILL IN

# 4. Evaluate performance with a decision tree
# Train the model
model_tree = DecisionTree.trainClassifier(trainingData_sel, numClasses=10, categoricalFeaturesInfo={},
                                     impurity='gini', maxDepth=5, maxBins=32)

# Compute its test error
tree_testErr = # FILL IN
print('Tree test error = ' + str(tree_testErr))
print('Number of selected features = ' + str(len(idx_keep)))

###########################################################
# TEST CELL
###########################################################

import numpy as np
from test_helper import Test

Test.assertEquals(np.round(100*tree_testErr,0), 31, 'incorrect result: test error is uncorrect')
Test.assertEquals(len(idx_keep), 686, 'incorrect result: test error is uncorrect')


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
Analyze the validation error behaviour with the number of trees. Is this the expected behaviour?
Taking into account the trade-off computational cost vs. accuracy, which number of trees will you select?
Can you write Python code for it?

###########################################################
# TEST CELL
###########################################################
import numpy as np
from test_helper import Test

Test.assertEquals(best_ntrees, 100, 'incorrect result: best_ntrees is uncorrect')
Test.assertEquals(np.round(100*best_error,0), 15, 'incorrect result: best_error is uncorrect')


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
Analyze the validation error behaviour with the tree depth. Is this the expected behaviour?
Taking into account the trade-off computational cost vs. accuracy, which tree depth will you select?
Can you write Python code for it?

###########################################################
# TEST CELL
###########################################################
import numpy as np
from test_helper import Test

Test.assertEquals(best_depth, 15, 'incorrect result: best_depth is uncorrect')
Test.assertEquals(np.round(100*best_error,0), 15, 'incorrect result: best_error is uncorrect')


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
Create training, validation and test partitions
In this subsection, let’s split the normalized dataset into training and validation data. We will use 40% of the data for training a model and 60% for validating the hyperparameters of the different learning algorithms.
To save computational time, cache both normalized training and validation RDDs, sicen we will use them several times.
You can use the randomSplit() method for this purpose.
Note: when you call to randomSplit, please, set seed=0 for comparison purposes:
    randomSplit([....], seed=0)
Can you write Python code for it?

#################################################
# TODO: Replace <FILL IN> with appropriate code
#################################################
# Create trainign and validation partitions
(trainingData, valData) = # FILL

# Our learning algorithms will make several passes over these datasets, so let’s cache these RDD in memory
trainingData.cache()
valData.cache()

###########################################################
# TEST CELL
###########################################################
import numpy as np
from test_helper import Test

Test.assertEquals(np.round(sum(trainingData.first().features),0), 355, 'incorrect result: training data are uncorrect')
Test.assertEquals(np.round(sum(valData.first().features),0), 319, 'incorrect result: validation data are uncorrect')


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
Data analysis
To analyze the data, let's start plotting each digit image. For this purpose, you can use the code provided in the following cell.
Can you write Python code for it?

import matplotlib.pyplot as plt
from pyspark.mllib.linalg import Vectors

def plot_data(images, h, w, n_row=1, n_col=10):
    """Plots the set of images provided in images

    Args:
        images (list of sparse vectors or numpy arrays): list of images where each image contains the
            features corresponding to the pixels of an image.
        h: heigth of the image (in number of pixels).
        w: width of the image (in number of pixels).
        n_row: Number of rows to use when plotting all the images
        n_col: Number of columns to use when plotting all the images

    """
    plt.figure(figsize=(1.8 * n_col, 2.4 * n_row))
    plt.subplots_adjust(bottom=0, left=.01, right=.99, top=.90, hspace=.35)

    for i in range(len(images)):
        plt.subplot(n_row, n_col, i + 1)
        try:
            img = images[i].toArray()
        except:
            img = images[i]

        plt.imshow(img.reshape((h, w)), cmap=plt.cm.jet)
        plt.xticks(())
        plt.yticks(())

#################################################
# TODO: Replace <FILL IN> with appropriate code
#################################################

# Define the height and width of the images
h= 28
w =28

# From the data RDD, create a new RDD where each element only has the features (pixel values) of each image
features = #FILL

# Pick up 10 images and plot them with plot_data() function
images= #FILL
plot_data(images, h, w)


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
Finally, let's load the test data from the "mnist.t" file. To be able to use it for testing purposes, we will also have to normalize them using the normalization parameters learned with the training data.
Can you write Python code for it?

#################################################
# TODO: Replace <FILL IN> with appropriate code
#################################################

# Load test data
data = MLUtils.loadLibSVMFile(sc, "mnist.t", numFeatures=784)

# Normalize test data:

# 1. Create two new RDD by extracting the labels and features of the data
label = # FILL IN
features = # FILL IN

# 2. Normalize the data features (use the scaler object fitted with the training data)
features_norm = # FILL IN

# 3. Create a new RDD of LabeledPoint data using the normalized features and cache it!
# 3.1 RDD with tuples (label, features)
test = # FILL IN
# 3.2 RDD with Label Point data
testData = # FILL IN

testData.cache()
testData.first()

###########################################################
# TEST CELL
###########################################################
import numpy as np
from test_helper import Test

Test.assertEquals(np.round(sum(testData.first().features),0), 199, 'incorrect result: training data are uncorrect')


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
Implementing a 1 vs. all multiclass SVM
Starting from the previous function, let's create a 1 vs. all multiclass SVM. For these purpose, we need to implement:
1. A function to train the set of 1 vs. all SVM
2. A function to compute the output of the 1 vs. all set of classifiers.
3. A function to compute the error over a set of data
Note: The function includes the model.clearThreshold() command to transform the model outputs from discrete values (labels) to real values (smooth or continuous outputs). This will be necessary to later combine the output of different SVM in a 1 vs. all fashion.
1. Let's create a training function to build all 1 vs. all SVMs
Can you write Python code for it?

#################################################
# TODO: Replace <FILL IN> with appropriate code
#################################################

def train_1vsall_SVM(trainingData, regParam=0.01, regType='l2'):
    """Produce a list of SVM models solving all the 1 vs. all problems

    Args:
        trainingData (RDD of labeled points): the training data to adjust the model
        regParam: The regularizer parameter (default: 0.01).
        regType: The type of regularizer used for training our model (default: “l2”). Allowed values:
                “l1” for using L1 regularization
                “l2” for using L2 regularization
                None for no regularization

    Returns:
        List of SVM models: A list of length number of classes where each element is a tuple (lab, model). Variable
            lab indicates the label 1 of the 1 vs. all problem and model is the SVM model solving the problem
    """
    # Get all possible labels from training data
    labels = # FILL IN
    # Initialize the list of models to be returned
    list_models = []
    for lab in labels:
        # Convert labels of trainingData to 1 vs. all format (use convert_label function)
        trainingData_1vsall=# FILL IN
        # Train the SVM model with 1 vs. all data
        model = # FILL IN

        # Modify the model to get smooth outputs
        model.clearThreshold()
        # Create the tuple (lab, model) and add it to the list of models
        # FILL IN

    return list_models


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
In this case, the number of selected features depends on the C value. Complete the next cell to select this value by cross validation.
Can you write Python code for it?

#################################################
# TODO: Replace <FILL IN> with appropriate code
#################################################

# Initialize variables
bestModel = # FILL IN
best_error = # FILL IN
best_C = # FILL IN
# Range of depth values to explore
C_params = [0.01, 0.1, 1]

for C_value in C_params:
    # Train the 1 vs. all SVM models (set regularization parameter to C_value)
    multiclass_SVM = train_1vsall_SVM(trainingData, regParam=C_value, regType='l1')

    # Compute the model error over the validation data
    error_1vsall = # FILL IN

    # Compute the number of selected features
    num_var_sel = # FILL IN

    print 'C value is ' + str(C_value) + ', the number of selected features is ' + str(num_var_sel) + ' and the validation error is '+ str(error_1vsall)

    # If the error has reduced, save the model, the optimum depth and error in bestModel, best_depth and best_error variables
    if (error_1vsall < best_error):
            bestModel = # FILL IN
            best_C = # FILL IN
            best_error = # FILL IN

print 'Optimum C value is: ' + str(best_C)


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
Let's select as the class 1 the digit '0' and transform the labels of both training and test data
Can you write Python code for it?

#################################################
# TODO: Replace <FILL IN> with appropriate code
#################################################

label_1 = 0
# Transform the labels of training data
trainingData_1vsall= # FILL IN
# Transform the labels of test data
testData_1vsall= # FILL IN

###########################################################
# TEST CELL
###########################################################
import numpy as np
from test_helper import Test

Test.assertEquals(np.round(sum(trainingData_1vsall.first().features),0), 355, 'incorrect result: trainingData_1vsall are uncorrect')
Test.assertEquals(np.round(sum(testData_1vsall.first().features),0), 199, 'incorrect result: testData_1vsall are uncorrect')


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
MLlib includes different data types (see http://spark.apache.org/docs/latest/mllib-data-types.html), such as, Local vectors, Labeled points, Local matrices or Distributed matrices. In supervised learning algorthms, the default data type is the  “labeled point” (see http://spark.apache.org/docs/latest/mllib-data-types.html#labeled-point).
So we need to transform our data file to an RDD of LabeledPoint. Both features and label fields in a LabeledPoint are of type Double; however, the input dataset has both the features and label in string format. So, as you can imagine, the proccess to transform input data to numeric values can be a bit thedious...
However, this data file has an especific format, known as LIBSVM text file format, where each line is:
label index1:value1 index2:value2 ...

That is, it is representing a labeled sparse feature vector.
Furthermore, MLLib includes a specific funtion:
loadLibSVMFile(sc, path, numFeatures=-1, minPartitions=None, multiclass=None)

which directly reads this format data file and returns an RDD of LabeledPoint elements.
Run the following cell to load the MNIST data as LabelPoint elements.
Note: we have explicitely defined the number of features to be sure that the sparse feature vectors are created with all the dimensions
Can you write Python code for it?

from pyspark.mllib.util import MLUtils
data = MLUtils.loadLibSVMFile(sc, "mnist", numFeatures= 784)


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
Now, evaluate the error the selected model (bestModel) over the test data
Can you write Python code for it?

#################################################
# TODO: Replace <FILL IN> with appropriate code
#################################################

# Finally, evaluate the test error over the best model
tree_finalErr =  # FILL IN

print 'Final test error of the validated tree: ' + str(tree_finalErr)

###########################################################
# TEST CELL
###########################################################
import numpy as np
from test_helper import Test

Test.assertEquals(np.round(100*tree_finalErr,0), 14, 'incorrect result: best_error is uncorrect')


I am working on the file "2_Classification/MLLib_classification-students.ipynb".
The first task is:
Use the function to compute the test error of the tree classifier and the Random Forest model.
Can you write Python code for it?

#################################################
# TODO: Replace <FILL IN> with appropriate code
#################################################
# Test error of the decision tree
tree_testErr = # FILL IN
print('Tree test error = ' + str(tree_testErr))

# Test error of the random forest
RF_testErr = # FILL IN
print('Random Forest test error = ' + str(RF_testErr))

###########################################################
# TEST CELL
###########################################################
import numpy as np
from test_helper import Test

Test.assertEquals(np.round(100*tree_testErr,0), 31, 'incorrect result: decision tree error is uncorrect')
Test.assertEquals(np.round(100*RF_testErr,0), 22, 'incorrect result: RF error is uncorrect')


I am working on the file "2_Complex_Queries_Basic_Visualization_py3.ipynb".
The first task is:
Aggregations
While searching for individual results is useful, sharepa also lets you make aggregation queries that give you results across the entirety of the SHARE dataset at once. This is useful if you're curious about the completeness of data sets.
For example, we can find the number of documents per source that are missing tags.
Can you write Python code for it?

# this aggregates the number of documents that have no tags, per source, using query boolean (not tags) while also
    # grabbing all sources in the aggregations: sources below
missing_tags_aggregation = {
    "query": {
        "bool": {
            "must_not": {
                "exists": {
                    "field": "tags"
                  }
            }
        }
    },
    "aggregations": {
        "sources": {
            "terms": {
                "field": "sources", # A field where the SHARE source is stored
                "min_doc_count": 0,
                "size": 0  # Will return all sources, regardless if there are results
            }
        }
    }
}

# puts all the items without tags into a list
results_without_tags = query_share(SHARE_API, missing_tags_aggregation)

# counts the number of items without tags
missing_tags_counts = results_without_tags['aggregations']['sources']['buckets']

# this prints out the number of documents with missing tags, separated by sources
for source in missing_tags_counts:
    print('{} has {} documents without tags'.format(source['key'], source['doc_count'], ))


I am working on the file "2_Complex_Queries_Basic_Visualization_py3.ipynb".
The first task is:
Aggregations with sharepa
You can also use sharepa to do aggregations.
Let's make a sharepa search object that will give us the number of documents per sourcethat don't have language specified.
Can you write Python code for it?

# yay! creating another new search!
no_language_search = ShareSearch()

# this sets up our search query: all documents with no language field
no_language_search = no_language_search.query(
    'bool',
    must_not={"exists": {"field": "language"}}
)

no_language_search.aggs.bucket(
    'sources',  # Every aggregation needs a name
    'significant_terms',  # There are many kinds of aggregations
    field='sources',  # We store the source of a document in its type, so this will aggregate by source
    min_doc_count=1,
    percentage={},
    size=0
)


I am working on the file "2_Complex_Queries_Basic_Visualization_py3.ipynb".
The first task is:
Complex Queries
Pagination
One request to the SHARE API will return just 10 results by default. To get more results, you can use the "size" parameter in your request, or paginate through the results you get back.
Can you write Python code for it?

# We are setting up a query for items in the SHARE dataset that have the keyword "frogs"
basic_query = {
    "query": {
        "query_string": {
            "query": "frogs"
        }
    }
}

#this puts the results of querying SHARE_API with what we outlined in basic_query (frogs)
query_results = query_share(SHARE_API, basic_query)

#print out the numbered list of the results
print_numbered_results(query_results)


I am working on the file "2_Complex_Queries_Basic_Visualization_py3.ipynb".
The first task is:
Complex Queries and Dataframes
Let's plot the number of document that each source has. We'll limit it to the top 30 sources to make sure that the graph is readable. Here we'll use the sharepa helper function bucket_to_dataframe()
Can you write Python code for it?

# from the SHAREPA library, we import this function to transform a bucket into a dataframe so we can plot it!
from sharepa import bucket_to_dataframe

# creating a new search!
all_results = ShareSearch()

# creating our search query!
all_results = all_results.query(
    'query_string', # Type of query, will accept a lucene query string
    query='*', # This lucene query string will find all documents that don't have tags
    analyze_wildcard=True  # This will make elasticsearch pay attention to the asterisk (which matches anything)
)

# Lucene supports fielded data. When performing a search you can either specify a field, or use the default field.
all_results.aggs.bucket(
    'sources',  # Every aggregation needs a name
    'terms',  # There are many kinds of aggregations, terms is a pretty useful one though
    field='sources',  # We store the source of a document in its type, so this will aggregate by source
    size=0,  # These are just to make sure we get numbers for all the sources, to make it easier to combine graphs
    min_doc_count=0
)

# this executes our search!
all_results = all_results.execute()

# this uses that function we imported above to transform our aggregated search into a dataframe so we can plot it!
all_results_frame = bucket_to_dataframe('# documents by source', all_results.aggregations.sources.buckets)

# this sorts the dataframe by the number of documents by source (descending order)
all_results_frame_sorted = all_results_frame.sort(ascending=False,  columns='# documents by source')

# this creates a bar chart that displays the first 30 results
all_results_frame_sorted[:30].plot(kind='bar')


I am working on the file "2_Complex_Queries_Basic_Visualization_py3.ipynb".
The first task is:
Pagination with sharepa
You can also use sharepa to paginate through all of the results in your query, and to access slices of your query at any time. The ShareSearch object returns a generator that you can use to access all results, using slices.
First, we'll redefine our helper function for nicer printing with data returned from sharepa.
Can you write Python code for it?

# this is a function that helps us print the lists with pagination between results (10, next 10, etc.)
def print_numbered_sharepa_results(search_obj):
    results = search_obj.execute()
    print(
        'There are {} total results and {} results on this page'.format(
            search_obj.count(),
            len(results.hits)
        )
    )
    print('---------------')
    for result in enumerate(results.hits):
        print('{}. {}'.format(result[0] + 1, result[1]['title']))

# Sharepa is a python client for  browsing and analyzing SHARE data specifically using elasticsearch querying.
    # We can use this to aggregate, graph, and analyze the data.
    # Helpful Links:
        # https://github.com/CenterForOpenScience/sharepa
        # https://pypi.python.org/pypi/sharepa
    # here, we import the specific function from Sharepa called ShareSearch and pretty_print
from sharepa import ShareSearch
from sharepa.helpers import pretty_print

# we are creating a new local search!
frogs_search = ShareSearch()

# this sets up what we will actually search for -- keyword "frogs"
frogs_search = frogs_search.query(
    'query_string',
    query='frogs'
)

# print the results of the search we made for "frogs" keyword in a numbered list
print_numbered_sharepa_results(frogs_search)

# print the 10th - 20th results of the search we made for "frogs" keyword in a numbered list
print_numbered_sharepa_results(frogs_search[10:20])


I am working on the file "2_Complex_Queries_Basic_Visualization_py3.ipynb".
The first task is:
Top tags
Let's do an elasticsearch query to find out what the most used tags are used in the dataset across all sources.
Can you write Python code for it?

# creating another new search!
top_tag_search = ShareSearch()

# this sets up our search query with the aggregations
top_tag_search.aggs.bucket(
    'tagsTermFilter',  # Every aggregation needs a name
    'terms',  # There are many kinds of aggregations
    field='tags',  # We store the source of a document in its type, so this will aggregate by source
    min_doc_count=1,
    exclude= "of|and|or",
    size=10
)

# pretty_print(top_tag_search.to_dict())

# this executes the search as we've outlined it above
top_tag_results_executed = top_tag_search.execute()

# this places the results of the search into this dictionary
top_tag_results = top_tag_results_executed.aggregations.tagsTermFilter.to_dict()['buckets']

# this prints out our search results (prettily)
pretty_print(top_tag_results)


I am working on the file "2_Complex_Queries_Basic_Visualization_py3.ipynb".
The first task is:
We can choose different types of plots to generate. Here, we'll make a pie chart of the data sources with the top 10 most results.
Can you write Python code for it?

# Creating a pie graph using the first 10 items in the data frame with no legend
all_results_frame_sorted[:10].plot(kind='pie', y="# documents by source", legend=False)


I am working on the file "2_Data_analysis.ipynb".
The first task is:
(Step7)
we can display which types of queries the classifiers failed to classify. These are interesting to examine for further work on how to improve the classifiers and the quality of the data set
Can you write Python code for it?

pipe = Pipeline([('vect', vectorizer), ('clf',LogisticRegression(C=10))])
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=42)

cv = ShuffleSplit(n=len(X_train), n_iter=1, test_size=0.2, random_state=0) #DEBUG: n_iter=10

random_grid_search = RandomizedSearchCV(
    pipe,
    param_distributions={
        'clf__C':[10]
    },
    cv=cv,
    scoring='roc_auc',
    n_iter=1,
    random_state=5,
    refit=True
)
random_grid_search.fit(X_train, Y_train)

#Evaluate the best model on the test data
Y_test_predicted = random_grid_search.best_estimator_.predict(X_test)

#Payloads classified incorrectly
pd.options.display.max_colwidth = 200
print('False positives')
print(X_test[(Y_test == 0) & (Y_test_predicted == 1)])
print('False negatives')
print(X_test[(Y_test == 1) & (Y_test_predicted == 0)])


I am working on the file "2_Data_analysis.ipynb".
The first task is:
2-Grams features
create a Countvectorizer and TF-IDFvectorizer that uses 2-grams.
Can you write Python code for it?

def get2Grams(payload_obj):
    '''Divides a string into 2-grams

    Example: input - payload: "<script>"
             output- ["<s","sc","cr","ri","ip","pt","t>"]
    '''
    payload = str(payload_obj)
    ngrams = []
    for i in range(0,len(payload)-2):
        ngrams.append(payload[i:i+2])
    return ngrams

tfidf_vectorizer_2grams = TfidfVectorizer(tokenizer=get2Grams)
count_vectorizer_2grams = CountVectorizer(min_df=1, tokenizer=get2Grams)


I am working on the file "2_Data_analysis.ipynb".
The first task is:
3-Grams TFIDFVectorizer feature space visualization
Can you write Python code for it?

X = tfidf_vectorizer_3grams.fit_transform(payloads['payload'])
Y = payloads['is_malicious']
visualize_feature_space_by_projection(X,Y,title='PCA visualization of 3-grams TFIDFVectorizer feature space')


I am working on the file "2_Data_analysis.ipynb".
The first task is:
3-Grams features
Create a Countvectorizer and TF-IDFvectorizer that uses 3-grams
Can you write Python code for it?

def get3Grams(payload_obj):
    '''Divides a string into 3-grams

    Example: input - payload: "<script>"
             output- ["<sc","scr","cri","rip","ipt","pt>"]
    '''
    payload = str(payload_obj)
    ngrams = []
    for i in range(0,len(payload)-3):
        ngrams.append(payload[i:i+3])
    return ngrams

tfidf_vectorizer_3grams = TfidfVectorizer(tokenizer=get3Grams)
count_vectorizer_3grams = CountVectorizer(min_df=1, tokenizer=get3Grams)


I am working on the file "2_Data_analysis.ipynb".
The first task is:
Classifier results
Can you write Python code for it?

#Display the results for the classifiers that were trained using our custom feature space
custom_features_classifiers = pickle.load( open("data/trained_classifier_custom_all_features.p", "rb"))
display(custom_features_classifiers)

#Display the results for the classifiers that were using bag of words feature spaces
classifier_results = pickle.load( open( "data/trained_classifiers.p", "rb" ) )
display(classifier_results)

#Combine the two tables into one table
classifier_results = classifier_results.append(custom_features_classifiers)
classifier_results = classifier_results.sort_values(['sensitivity','accuracy'], ascending=[False,False])
display(classifier_results)


I am working on the file "2_Data_analysis.ipynb".
The first task is:
Custom feature space visualization
Can you write Python code for it?

X = create_features(pd.DataFrame(payloads['payload'].copy()))
Y = payloads['is_malicious']
visualize_feature_space_by_projection(X,Y,title='PCA visualization of custom feature space')


I am working on the file "2_Data_analysis.ipynb".
The first task is:
Export classifiers
First, export full list of trained classifiers for later use
Second, pick one classifier to save in a separate pickle, used later to implement in a dummy server
Can you write Python code for it?

#save complete list of classifiers to 'trained_classifiers'
pickle.dump( classifier_results, open( "data/trained_classifiers.p", "wb" ) )

#In this case, we are going to implement tfidf 2grams RandomForest in our dummy server
classifier = (custom_features_classifiers['model'].iloc[0])
print(classifier)

#Save classifiers in a pickle file to be able to re-use them without re-training
pickle.dump( classifier, open( "data/tfidf_2grams_randomforest.p", "wb" ) )


I am working on the file "2_Data_analysis.ipynb".
The first task is:
F1-score
Calculate F1-score of each classifier and add to classifiers table
(We didn't implement this in the train_model function as with the other performance metrics because we've already done a 82 hour training session before this and didn't want to re-run the entire training just to add F1-score from inside train_model)
Can you write Python code for it?

def f1_score(conf_matrix):
    precision = conf_matrix[0][0] / (conf_matrix[0][0] + conf_matrix[0][1] )
    recall = conf_matrix[0][0] / (conf_matrix[0][0] + conf_matrix[1][0] )

    return (2 * precision * recall) / (precision + recall)

#load classifier table if not yet loaded
classifier_results = pickle.load( open( "data/trained_classifiers.p", "rb" ) )

#Calculate F1-scores
classifier_results['F1-score'] = [ f1_score(conf_matrix) for conf_matrix in classifier_results['conf_matrix']]

#Re-arrange columns
classifier_results = classifier_results[['F1-score','accuracy','sensitivity','specificity','auc','conf_matrix','params','model','roc']]

#re-sort on F1-score
classifier_results = classifier_results.sort_values(['F1-score','accuracy'], ascending=[False,False])

display(classifier_results)


I am working on the file "2_Data_analysis.ipynb".
The first task is:
Final formating
Convert numeric columns to float
Round numeric columns to 4 decimals
Can you write Python code for it?


I am working on the file "2_Data_analysis.ipynb".
The first task is:
Scoring custom features
Score the custom features using the SelectKBest function, then visualize the scores in a graph
to see which features are less significant
Can you write Python code for it?

Y = payloads['is_malicious']
X = create_features(pd.DataFrame(payloads['payload'].copy()))

test = SelectKBest(score_func=chi2, k='all')
fit = test.fit(X, Y)
# summarize scores
print(fit.scores_)
features = fit.transform(X)
# summarize selected features
# summarize scores
np.set_printoptions(precision=2)
print(fit.scores_)

# Get the indices sorted by most important to least important
indices = np.argsort(fit.scores_)

# To get your top 10 feature names
featuress = []
for i in range(10):
    featuress.append(X.columns[indices[i]])

display(featuress)
display([featuress[i] + ' ' + str(fit.scores_[i]) for i in indices[range(10)]])

plt.rcdefaults()
fig, ax = plt.subplots()

y_pos = np.arange(len(featuress))
performance = 3 + 10 * np.random.rand(len(featuress))
error = np.random.rand(len(featuress))

ax.barh(y_pos, fit.scores_[indices[range(10)]],  align='center',
        color='green', ecolor='black')
ax.set_yticks(y_pos)
ax.set_yticklabels(featuress)
ax.set_xscale('log')

#ax.invert_yaxis()  # labels read top-to-bottom
ax.set_xlabel('Points')
ax.set_title('SelectKBest()')

plt.show()


I am working on the file "2_Data_analysis.ipynb".
The first task is:
Step2
load the payloads into memory
Can you write Python code for it?

payloads = pd.read_csv("data/payloads.csv",index_col='index')
display(payloads.head(30))


I am working on the file "2_Data_analysis.ipynb".
The first task is:
Step3A - feature engineering custom features
We will create our own feature space with features that might be important for this task, this includes:
 - length of payload
 - number of non-printable characters in payload
 - number of punctuation characters in payload
 - the minimum byte value of payload
 - the maximum byte value of payload
 - the mean byte value of payload
 - the standard deviation of payload byte values
 - number of distinct bytes in payload
 - number of SQL keywords in payload
 - number of javascript keywords in payload
Can you write Python code for it?

def plot_feature_distribution(features):
    print('Properties of feature: ' + features.name)
    print(features.describe())
    f, ax = plt.subplots(1, figsize=(10, 6))
    ax.hist(features, bins=features.max()-features.min()+1, normed=1)
    ax.set_xlabel('value')
    ax.set_ylabel('fraction')

    plt.show()

def create_feature_length(payloads):
    '''
        Feature describing the lengh of the input
    '''

    payloads['length'] = [len(str(row)) for row in payloads['payload']]
    return payloads

payloads = create_feature_length(payloads)
display(payloads.head())

plot_feature_distribution(payloads['length'])

def create_feature_non_printable_characters(payloads):
    '''
    Feature
    Number of non printable characthers within payload
    '''

    payloads['non-printable'] = [ len([1 for letter in str(row) if letter not in string.printable]) for row in payloads['payload']]
    return payloads

create_feature_non_printable_characters(payloads)
display(payloads.head())

plot_feature_distribution(payloads['non-printable'])

def create_feature_punctuation_characters(payloads):
    '''
    Feature
    Number of punctuation characthers within payload
    '''

    payloads['punctuation'] = [ len([1 for letter in str(row) if letter in string.punctuation]) for row in payloads['payload']]
    return payloads

create_feature_punctuation_characters(payloads)
display(payloads.head())

plot_feature_distribution(payloads['punctuation'])

def create_feature_min_byte_value(payloads):
    '''
    Feature
    Minimum byte value in payload
    '''

    payloads['min-byte'] = [ min(bytearray(str(row), 'utf8')) for row in payloads['payload']]
    return payloads

create_feature_min_byte_value(payloads)
display(payloads.head())

plot_feature_distribution(payloads['min-byte'])

def create_feature_max_byte_value(payloads):
    '''
    Feature
    Maximum byte value in payload
    '''

    payloads['max-byte'] = [ max(bytearray(str(row), 'utf8')) for row in...


I am working on the file "2_Data_analysis.ipynb".
The first task is:
Step3B - Feature engineering using bag of words techniques.
Additional to our custom feature space, we will create 6 more feature spaces using bag-of-words techniques
The following vectorizers below is another way of creating features for text input.
We will test the performance of these techniques independently from our custom features in Step 3A.
We will create vectorizers of these combinations:
- 1-grams CountVectorizer
- 2-grams CountVectorizer
- 3-grams CountVectorizer
- 1-grams TfidfVectorizer
- 2-grams TfidfVectorizer
- 3-grams TfidfVectorizer
The type of N-gram function determines how the actual "words" should be created from the payload string
Each vectorizer is used later in Step4 in Pipeline objects before training
See report for further explanation
1-Grams features
create a Countvectorizer and TF-IDFvectorizer that uses 1-grams.
1-grams equals one feature for each letter/symbol recorded
Can you write Python code for it?

def get1Grams(payload_obj):
    '''Divides a string into 1-grams

    Example: input - payload: "<script>"
             output- ["<","s","c","r","i","p","t",">"]
    '''
    payload = str(payload_obj)
    ngrams = []
    for i in range(0,len(payload)-1):
        ngrams.append(payload[i:i+1])
    return ngrams

tfidf_vectorizer_1grams = TfidfVectorizer(tokenizer=get1Grams)
count_vectorizer_1grams = CountVectorizer(min_df=1, tokenizer=get1Grams)


I am working on the file "2_Data_analysis.ipynb".
The first task is:
Step3C - Feature space visualization
After creating our different feature spaces to later train each classifier on,
we first examine them visually by projecting the feature spaces into two dimensions using Principle Component Analysis
Graphs are shown below displaying the data in 3 out of 7 of our feature spaces
Can you write Python code for it?

def visualize_feature_space_by_projection(X,Y,title='PCA'):
    '''Plot a two-dimensional projection of the dataset in the specified feature space

    input: X - data
           Y - labels
           title - title of plot
    '''
    pca = TruncatedSVD(n_components=2)
    X_r = pca.fit(X).transform(X)

    # Percentage of variance explained for each components
    print('explained variance ratio (first two components): %s'
          % str(pca.explained_variance_ratio_))

    plt.figure()
    colors = ['blue', 'darkorange']
    lw = 2

    #Plot malicious and non-malicious separately with different colors
    for color, i, y in zip(colors, [0, 1], Y):
        plt.scatter(X_r[Y == i, 0], X_r[Y == i, 1], color=color, alpha=.3, lw=lw,
                    label=i)

    plt.legend(loc='best', shadow=False, scatterpoints=1)
    plt.title(title)
    plt.show()


I am working on the file "2_Data_analysis.ipynb".
The first task is:
Step4 - Model selection and evaluation
First, we will automate hyperparameter tuning and out of sample testing using train_model below
Can you write Python code for it?

def train_model(clf, param_grid, X, Y):
    '''Trains and evaluates the model clf from input

    The function selects the best model of clf by optimizing for the validation data,
    then evaluates its performance using the out of sample test data.

    input - clf: the model to train
            param_grid: a dict of hyperparameters to use for optimization
            X: features
            Y: labels

    output - the best estimator (trained model)
             the confusion matrix from classifying the test data
    '''

    #First, partition into train and test data
    X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=42)

    n_iter = 5
    #If number of possible iterations are less than prefered number of iterations,
    #set it to the number of possible iterations
    #number of possible iterations are not less than prefered number of iterations if any argument is expon()
    #because expon() is continous (writing 100 instead, could be any large number)
    n_iter = min(n_iter,np.prod([
        100 if type(xs) == type(expon())
        else len(xs)
        for xs in param_grid.values()
    ]))

    #perform a grid search for the best parameters on the training data.
    #Cross validation is made to select the parameters, so the training data is actually split into
    #a new train data set and a validation data set, K number of times
    cv = ShuffleSplit(n=len(X_train), n_iter=5, test_size=0.2, random_state=0) #DEBUG: n_iter=10
    #cv = KFold(n=len(X), n_folds=10)
    random_grid_search = RandomizedSearchCV(
        clf,
        param_distributions=param_grid,
        cv=cv,
        scoring='f1',
        n_iter=n_iter, #DEBUG 1
        random_state=5,
        refit=True,
        verbose=10
    )

    '''Randomized search used instead. We have limited computing power
    grid_search = GridSearchCV(
        clf,
        param_grid=param_grid,
        cv=cv,
        scoring='f1', #accuracy/f1/f1_weighted all give same result?
       ...


I am working on the file "2_Data_analysis.ipynb".
The first task is:
Step6 - Website integration extract
This is the code needed when implementing the saved classifier in tfidf_2grams_randomforest.p on a server
Can you write Python code for it?

import pickle
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.ensemble import RandomForestClassifier

def get2Grams(payload_obj):
    '''Divides a string into 2-grams

    Example: input - payload: "<script>"
             output- ["<s","sc","cr","ri","ip","pt","t>"]
    '''
    payload = str(payload_obj)
    ngrams = []
    for i in range(0,len(payload)-2):
        ngrams.append(payload[i:i+2])
    return ngrams

classifier = pickle.load( open("data/tfidf_2grams_randomforest.p", "rb"))

def injection_test(inputs):
    variables = inputs.split('&')
    values = [ variable.split('=')[1] for variable in variables]
    print(values)
    return 'MALICIOUS' if classifier.predict(values).sum() > 0 else 'NOT_MALICIOUS'

#test injection_test
display(injection_test("val1=%3Cscript%3Ekiddie"))


I am working on the file "2_Data_analysis.ipynb".
The first task is:
Then, we will use the train_model function to train, optimize and retrieve out of sample testing results from a range of classifiers.
Classifiers tested using our custom feature space:
 - AdaBoost
 - SGD classifier
 - MultiLayerPerceptron classifier
 - Logistic Regression
 - Support Vector Machine
 - Random forest
 - Decision Tree
 - Multinomial Naive Bayes
Classifiers tested using bag-of-words feature spaces:
 - MultiLayerPerceptron classifier
 - Logistic Regression
 - Support Vector Machine
 - Random forest
 - Multinomial Naive Bayes
Some classifiers were unable to train using a bag-of-words feature space because they couldn't handle sparse graphs
All their best parameters with their performance is stored in a dataframe called classifier_results
Make dictionary of models with parameters to optimize using bag-of-words feature spaces
Can you write Python code for it?

def create_classifier_inputs_using_vectorizers(vectorizer, subscript):
    '''make pipelines of the specified vectorizer with the classifiers to train

    input - vectorizer: the vectorizer to add to the pipelines
            subscript:  subscript name for the dictionary key

    output - A dict of inputs to use for train_model(); a pipeline and a dict of params to optimize
    '''

    classifier_inputs = {}

    classifier_inputs[subscript + ' MLPClassifier'] = {
        'pipeline':Pipeline([('vect', vectorizer),('clf',MLPClassifier(
            activation='relu',
            solver='adam',
            early_stopping=False,
            verbose=True

        ))]),
        'dict_params': {
            'vect__min_df':[1,2,5,10,20,40],
            'clf__hidden_layer_sizes':[(500,250,125,62)],
            'clf__alpha':[0.0005,0.001,0.01,0.1,1],
            'clf__learning_rate':['constant','invscaling'],
            'clf__learning_rate_init':[0.001,0.01,0.1,1],
            'clf__momentum':[0,0.9],
        }
    }
    '''
    classifier_inputs[subscript + ' MultinomialNB'] = {
        'pipeline':Pipeline([('vect', vectorizer),('clf',MultinomialNB())]),
        'dict_params': {
            'vect__min_df':[1,2,5,10,20,40]
        }
    }
    classifier_inputs[subscript + ' RandomForest'] = {
        'pipeline':Pipeline([('vect', vectorizer),('clf',RandomForestClassifier(
            max_depth=None,min_samples_split=2, random_state=0))]),
        'dict_params': {
            'vect__min_df':[1,2,5,10,20,40],
            'clf__n_estimators':[10,20,40,60]
        }
    }
    classifier_inputs[subscript + ' Logistic'] = {
        'pipeline':Pipeline([('vect', vectorizer), ('clf',LogisticRegression())]),
        'dict_params': {
            'vect__min_df':[1,2,5,10,20,40],
            'clf__C':[0.001, 0.01, 0.1, 1, 10, 100, 1000]
        }
    }
    classifier_inputs[subscript + ' SVM'] = {
        'pipeline':Pipeline([('vect', vectorizer), ('clf',SVC(probability=True))]),
       ...


I am working on the file "2_Data_analysis.ipynb".
The first task is:
Three examples of learning curves from the trained classifiers.
All learning curves have upsloping cross-validation score at the end,
which means that adding more data would potentially increase the accuracy
Can you write Python code for it?

#plot learning curve for tfidf 1grams RandomForest
X = payloads['payload']
Y = payloads['is_malicious']
plot_learning_curve(classifier_results.iloc[0],X,Y)

#plot learning curve for count 3grams MultinomialNB
X = payloads['payload']
Y = payloads['is_malicious']
plot_learning_curve(classifier_results.iloc[6],X,Y)

#plot learning curve for custom svm
X = create_features(pd.DataFrame(payloads['payload'].copy()))
Y = payloads['is_malicious']
plot_learning_curve(classifier_results.iloc[5],X,Y)


I am working on the file "2_Data_analysis.ipynb".
The first task is:
Use our custom feature space,
and train every classifier in classifier_inputs_custom with
P.S! Don't try to run this, it will take many hours to complete
Instead skip to Step4B
Can you write Python code for it?

classifier_inputs_custom = {}

#Get classifiers and parameters to optimize
classifier_inputs_custom.update(create_classifier_inputs('custom'))

#Extract payloads and labels
Y = payloads['is_malicious']
X = create_features(pd.DataFrame(payloads['payload'].copy()))

#Select the best features
X_new = SelectKBest(score_func=chi2, k=4).fit_transform(X,Y)

#Call train_model for every classifier and save results to classifier_results
for classifier_name, inputs in classifier_inputs_custom.items():
    if classifier_name in classifier_results.index.values.tolist():
        print('Skipping ' + classifier_name + ', already trained')
    else:
        result_dict = train_model(inputs['pipeline'],inputs['dict_params'],X,Y)
        classifier_results.loc[classifier_name] = result_dict

display(classifier_results)

#pickle.dump( classifier_results, open( "data/trained_classifiers_custom_all_features.p", "wb" ) )

#Save classifiers in a pickle file to be able to re-use them without re-training
pickle.dump( classifier_results, open( "data/trained_classifiers.p", "wb" ) )


I am working on the file "2_Data_analysis.ipynb".
The first task is:
define a function that makes a feature vector from the payload using the custom features
Can you write Python code for it?

def create_features(payloads):
    features = create_feature_length(payloads)
    features = create_feature_non_printable_characters(features)
    features = create_feature_punctuation_characters(features)
    features = create_feature_max_byte_value(features)
    features = create_feature_min_byte_value(features)
    features = create_feature_mean_byte_value(features)
    features = create_feature_std_byte_value(features)
    features = create_feature_distinct_bytes(features)
    features = create_feature_sql_keywords(features)
    features = create_feature_javascript_keywords(features)
    del features['payload']

    return features


I am working on the file "2_IPython.ipynb".
The first task is:
A notebook contains a linear succession of cells and output areas. A cell contains Python code, in one or multiple lines. The output of the code is shown in the corresponding output area.

Now, we do a simple arithmetic operation.
Can you write Python code for it?


I am working on the file "2_IPython.ipynb".
The first task is:
IPython not only accepts Python code, but also shell commands. Those are defined by the operating system (Windows, Linux, Mac OS X, etc.). We first type ! in a cell before typing the shell command. Here, we get the list of notebooks in the current directory.
Can you write Python code for it?


I am working on the file "2_IPython.ipynb".
The first task is:
Let's call the function once to view the solutions. For this set of parameters, we see the trajectories swirling around two points, called attractors.
Can you write Python code for it?


I am working on the file "2_IPython.ipynb".
The first task is:
Now, we illustrate the latest interactive features in IPython 2.0+. This version brings graphical widgets in the notebook that can interact with Python objects. We will create a drop-down menu allowing us to display one among several videos.
Can you write Python code for it?

from collections import OrderedDict
from IPython.display import display, clear_output
from IPython.html.widgets import DropdownWidget

# We create a DropdownWidget, with a dictionary containing
# the keys (video name) and the values (Youtube identifier)
# of every menu item.
dw = DropdownWidget(values=OrderedDict([
                        ('SciPy 2012', 'iwVvqwLDsJo'),
                        ('PyCon 2012', '2G5YTlheCbw'),
                        ('SciPy 2013', 'j9YpkSX7NNM')]))
# We create a callback function that displays the requested
# Youtube video.
def on_value_change(name, val):
    clear_output()
    display(YouTubeVideo(val))
# Every time the user selects an item, the function
# `on_value_change` is called, and the `val` argument
# contains the value of the selected item.
dw.on_trait_change(on_value_change, 'value')
# We choose a default value.
dw.value = dw.values['SciPy 2013']
# Finally, we display the widget.
display(dw)


I am working on the file "2_IPython.ipynb".
The first task is:
Similarly here, we create a SVG graphics dynamically.
Can you write Python code for it?


I am working on the file "2_IPython.ipynb".
The first task is:
The interactive features of IPython 2.0 bring a whole new dimension in the notebook, and we can expect many developments in the months and years to come.

You'll find all the explanations, figures, references, and much more in the book (to be released later this summer).
IPython Cookbook, by Cyrille Rossant, Packt Publishing, 2014 (500 pages).

Interactive Widgets: Exploring the Lorenz System of Differential Equations
In this Notebook we explore the Lorenz system of differential equations:
$$
\begin{aligned}
\dot{x} & = \sigma(y-x) \
\dot{y} & = \rho x - y - xz \
\dot{z} & = -\beta z + xy
\end{aligned}
$$
This is one of the classic systems in non-linear differential equations. It exhibits a range of different behaviors as the parameters ($\sigma$, $\beta$, $\rho$) are varied.
Imports
First, we import the needed things from IPython, NumPy, Matplotlib and SciPy.
Can you write Python code for it?

%matplotlib inline

from IPython.html.widgets import interact, interactive
from IPython.display import clear_output, display, HTML

import numpy as np
from scipy import integrate

from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib.colors import cnames
from matplotlib import animation


I am working on the file "2_IPython.ipynb".
The first task is:
Using IPython's interactive function, we can explore how the trajectories behave as we change the various parameters.
Can you write Python code for it?


I am working on the file "2_IPython.ipynb".
The first task is:
We covered the basics of IPython and the notebook. Let's now turn to the rich display and interactive features of the notebook. Until now, we only created code cells, i.e. cells that contain... code. There are other types of cells, notably Markdown cells. Those contain rich text formatted with Markdown, a popular plain text formatting syntax. This format supports normal text, headers, bold, italics, hypertext links, images, mathematical equations in LaTeX, code, HTML elements, and other features, as shown below.

New paragraph
This is rich text with links, equations:
$$\hat{f}(\xi) = \int_{-\infty}^{+\infty} f(x)\, \mathrm{e}^{-i \xi x}$$
code with syntax highlighting:
python
print("Hello world!")
and images:

By combining code cells and Markdown cells, you can create a standalone interactive document that combines computations (code), text and graphics.

That was it for Markdown cells. IPython also comes with a sophisticated display system that lets you insert rich web elements in the notebook. Here, we show how to add HTML, SVG (Scalable Vector Graphics) and even Youtube videos in a notebook.

First, we need to import some classes.
Can you write Python code for it?

from IPython.display import HTML, SVG, YouTubeVideo


I am working on the file "2_IPython.ipynb".
The first task is:
We create an HTML table dynamically with Python, and we display it in the (HTML-based) notebook.
Can you write Python code for it?

HTML('''

''' +
''.join(['
' +
         ''.join(['{row},{col}	'.format(
            row=row, col=col
            ) for col in range(5)]) +
         '
' for row in range(5)]) +
'''

''')


I am working on the file "2_Indexing_and_Selecting_Data.ipynb".
The first task is:
Notice that in this case, the output is itself a dataframe.
Can you write Python code for it?

type(market_df[['Cust_id', 'Sales', 'Profit']])

# Similarly, if you select one column using double square brackets,
# you'll get a df, not Series

type(market_df[['Sales']])


I am working on the file "2_Indexing_and_Selecting_Data.ipynb".
The first task is:
Notice that, by default, pandas assigns integer labels to the rows, starting at 0.
Can you write Python code for it?

# Selecting the rows from indices 2 to 6
market_df[2:7]

# Selecting alternate rows starting from index = 5
market_df[5::2].head()


I am working on the file "2_Indexing_and_Selecting_Data.ipynb".
The first task is:
Now imagine you had a column with entries [2, 4, 7, 8 ...], and you set that as the index. What should df[2] return?
The second row, or the row with the index value = 2?
Taking an example from this dataset, say you decide to assign the Order_Quantity column as the index.
Can you write Python code for it?


I am working on the file "2_Indexing_and_Selecting_Data.ipynb".
The first task is:
Pandas throws an error because it is confused whether the [2] is an index or a label. Recall from the previous section that you can change the row indices.
Can you write Python code for it?

# Changing the row indices to Ord_id
market_df.set_index('Ord_id').head()


I am working on the file "2_Indexing_and_Selecting_Data.ipynb".
The first task is:
Selecting Columns
There are two simple ways to select a single column from a dataframe - df['column_name'] and df.column_name.
Can you write Python code for it?

# Using df['column']
sales = market_df['Sales']
sales.head()

# Using df.column
sales = market_df.Sales
sales.head()

# Notice that in both these cases, the resultant is a Series object
print(type(market_df['Sales']))
print(type(market_df.Sales))


I am working on the file "2_Indexing_and_Selecting_Data.ipynb".
The first task is:
Selecting Multiple Columns
You can select multiple columns by passing the list of column names inside the []: df[['column_1', 'column_2', 'column_n']].
For instance, to select only the columns Cust_id, Sales and Profit:
Can you write Python code for it?

# Select Cust_id, Sales and Profit:
market_df[['Cust_id', 'Sales', 'Profit']].head()


I am working on the file "2_Indexing_and_Selecting_Data.ipynb".
The first task is:
Selecting Subsets of Dataframes
Until now, you have seen selecting rows and columns using the following ways:
* Selecting rows: df[start:stop]
* Selecting columns: df['column'] or df.column or df[['col_x', 'col_y']]
    * df['column'] or df.column return a series
    * df[['col_x', 'col_y']] returns a dataframe
But pandas does not prefer this way of indexing dataframes, since it has some ambiguity. For instance, let's try and select the third row of the dataframe.
Can you write Python code for it?

# Trying to select the third row: Throws an error
market_df[2]


I am working on the file "2_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Booleans
Can you write Python code for it?


I am working on the file "2_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Classes
Can you write Python code for it?

class Car(object):

    engine = 'V4'    # class attribute

    def start(self):  # class method
        print "Starting the car with a", self.engine, "engine"

mycar = Car()

type(mycar)

mycar.engine

mycar.start()

mycar.engine = 'V6'

mycar.engine

mycar.start()


I am working on the file "2_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Containers

Tuples
Can you write Python code for it?


I am working on the file "2_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Floating point numbers
Can you write Python code for it?


I am working on the file "2_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Flow control

conditional (if, else, elif)
Can you write Python code for it?

a = 10
if a == 10:
    print "a is 10"

a = 10
if a > 10:
    print "a is larger than 10"
else:
    print "a is less than 10... or maybe equal too"

a = 4
if a > 10:
    print "a is larger than 10"
elif a < 10:
    print "a is less than 10"
else:
    print "a is equal to 10"


I am working on the file "2_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Functions
Can you write Python code for it?

def func():
    print "Hello world"

func()

def func2(name):
    print "Hello", name

func2("Jonathan")

def times2(x):
    return x * 2

y = times2(2)
print y

def times_something(x, y=2):
    print x*y

times_something(3)

times_something(3, 3)


I am working on the file "2_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Indexing
Can you write Python code for it?

l

print l[0]
print l[1]
print l[2]

print l[-1]   # last element
print l[-2]

l[0] = 'orange'
print l


I am working on the file "2_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Lists can store different type of variable in each element
Can you write Python code for it?

ll = [5, 22.9, 14.8+1j, 'hello', [1,2,3]]

ll

print ll[0]
print ll[1]
print ll[2]
print ll[3]
print ll[4]


I am working on the file "2_Intro_to_Python_and_SciPy.ipynb".
The first task is:
The Scientific Python ecosystem

NumPy
Can you write Python code for it?

import numpy as np

a = np.array([0, 1, 2, 3, 4, 5, 6, 7])

a

a.shape

a.ndim

a.dtype

a[0::2]

a[a>3]

a * 2 + 100

a.mean()


I am working on the file "2_Intro_to_Python_and_SciPy.ipynb".
The first task is:
The Scientific Python ecosystem

SciPy
Can you write Python code for it?


I am working on the file "2_Intro_to_Python_and_SciPy.ipynb".
The first task is:
The Scientific Python ecosystem

matplotlib
Can you write Python code for it?

%pylab inline

plot([1,2,3])

a = np.random.rand(30, 30)
imshow(a)
colorbar()


I am working on the file "2_Introduction_to_Python.ipynb".
The first task is:
Define Variables
Can you write Python code for it?

a_integer = 5

a_float = 1.41421356237

a_integer + a_float

a_number = a_integer + a_float

print(a_number)

a_string = 'How you doing, world?'

print(a_string)

a_integer + a_string

print(a_integer, a_string)

str(a_integer) + a_string

a_float = 3.1417

print(a_float)

who

# This is a comment

"""This is a multi-
line comment"""


I am working on the file "2_Introduction_to_Python.ipynb".
The first task is:
Exercise: Can you rework this code so that you can test if a number is prime (so only it and 1 divides it)
Can you write Python code for it?

a = 29
b = 1
c = 2
while b != 0:
    b = a%c
    if a == c:
        print(str(a) + ' is a prime number.')
    elif b == 0:
        print(str(a) + ' is divisable by ' + str(c))
    else:
        c += 1


I am working on the file "2_Logistic_Regression.ipynb".
The first task is:
It's time to test a data set with the learning tool we just built.
We've loaded the Iris data set from the Scikit-Learn Module. It has 4 features and 3 classes.
Can you write Python code for it?

from sklearn.datasets import load_iris
iris = load_iris()

X = iris['data']
y = iris['target']

print X.shape
print y.shape
print "Number of Classes: {}".format(len(np.unique(y)))


I am working on the file "2_Logistic_Regression.ipynb".
The first task is:
Now let's focus on multi-class classification
We can use the "one-versus-all" scheme to achive the goal. To be specific, for each class, we code it as the positive class (1), while the rest of classes are coded as the negative class (0). We then apply Binary Logistic Regression and predict the probabilities of being positive for each class against all others. The class with the highest probability of being positive will be chosen as the final prediction of the classification task.
The following class MultiClassLogisticRegression() implements this idea.
Can you write Python code for it?

class MultiClassLogisticRegression():

    def __init__(self, lamda):

        self._lamda = lamda

    # Define class variables
    _values = None
    _n_samples = None
    _n_classes = None
    _lr_list = []

    def fit(self, X, y):

        # fit the model
        self._values = np.unique(y)
        self._n_samples = X.shape[0]
        self._n_classes = len(self._values)
        y_copy = np.array(y)

        for value in self._values:
            idx = (y == value)
            y_copy[idx] = 1
            y_copy[~idx] = 0
            lr = BinaryLogisticRegression(lamda=self._lamda)
            self._lr_list.append(lr.fit(X, y_copy))

        return self

    def predict_prob(self, X):

        # predict the probabilities for each class
        y_prob_ini = np.empty((X.shape[0], self._n_classes))
        for k, lr in enumerate(self._lr_list):
            y_prob_ini[:, k] = lr.predict_prob(X)

        row_sums = y_prob_ini.sum(axis=1)
        y_prob = y_prob_ini / row_sums[:, np.newaxis]
        return y_prob

    def predict(self, X):

        # predict with one_versus_all scheme
        p = self.predict_prob(X)
        y_predict_idx = np.argmax(p, axis=1)
        y_predict = np.array([self._values[i] for i in y_predict_idx])
        return y_predict


I am working on the file "2_Logistic_Regression.ipynb".
The first task is:
This is a multi-class problem. We then use the defined class MultiClassLogisticRegression() to train the model and predict the results for test examples.
Can you write Python code for it?

lr = MultiClassLogisticRegression(lamda=1)
lr = lr.fit(X_train, y_train)
y_predict = lr.predict(X_test)

print "True Values:      {}".format(y_test)
print "Predicted Values: {}".format(y_predict)
print "Prediction Accuracy: {:.2%}".format(np.mean((y_predict == y_test).astype(float)))


I am working on the file "2_Logistic_Regression.ipynb".
The first task is:
We can split the original data for test purposes
Can you write Python code for it?

from sklearn.cross_validation import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=26)


I am working on the file "2_Mathematical_Groundwork/2_11_least_squares.ipynb".
The first task is:
2.11 Least-squares Minimization
In the field of radio interferometry, we often encounter problems which must be solved numerically. One such problem is least-squares minimization. Conceptually, it is very simple. Given a model and some data, we want to find the values of a set of parameters which minimize the difference between our model and our observations.
Firstly, we need to phrase the problem in simple mathematics. Let us start by defining the quantities of interest and the function we wish to minimize.
We will refer to our data vector as $\mathbf{d}$ and our model vector as $\mathbf{m}$. These vectors contain the measured values and those predicted by the model respectively. We wish to minimize the $L^2$ or Euclidean vector norm of their difference. Whilst you may not have called it that in the past, you have almost certainly encountered it,
$$\lVert\mathbf{r}\rVert = \lVert\mathbf{d}-\mathbf{m}\rVert = \sqrt{\sum\limits_{i=1}^N(d_i - m_i)^2}.$$
$\mathbf{r}$ is the residual vector and it is a measure of the difference between the values predicted by our model and the observed values.
It is important to note that in general $\mathbf{m}$ is a function of a number of parameters, such as $(x_1, x_2, x_3, \dots)$. These parameters form the parameter vector $\mathbf{x}$ which is what we ultimately want to determine.
There are many methods which solve problems of the given form. However, we will stick to explaining two of the most commonly used in radio interferometry. Specifically, these are Gauss-Newton and Levenberg–Marquardt; both of which are technically non-linear least squares solvers. They can be applied to linear problems too.
We will not present the derivations of the methods, although they are readily available. The Gauss-Newton update rule is given by the following:
$$\delta \mathbf{x} = {(\mathbb{J}^T\mathbb{J})}^{-1} \mathbb{J}^T \mathbf{r}.$$
This is far simpler than it may at first appear. $\delta \mathbf{x}$ is simply the update to the current best...
Can you write Python code for it?

def sinusoid(x, t):
    """
    Returns a vector containing the values of a sinusoid with parameters x evaluated at points t.

    INPUTS:
    t    Value of independent variable at the sampled points.
    x    Vector of parameters.
    """

    x1 = x[0]     #Amplitude
    x2 = x[1]     #Frequency
    x3 = x[2]     #Phase-shift

    return x1*np.sin(2*np.pi*x2*t + x3)

def sinusoid_jacobian(x, t):
    """
    Returns the Jacobian corresponding to the function defined in sinusoid.

    INPUTS:
    t    Value of independent variable at the sampled points.
    x    Vector of parameters.
    """

    x1 = x[0]     #Amplitude
    x2 = x[1]     #Frequency
    x3 = x[2]     #Phase-shift

    jacobian = np.empty([t.shape[0], x.shape[0]])
    jacobian[:,0] = np.sin(2*np.pi*x2*t + x3)
    jacobian[:,1] = 2*np.pi*t*x1*np.cos(2*np.pi*x2*t + x3)
    jacobian[:,2] = x1*np.cos(2*np.pi*x2*t + x3)

    return jacobian

def sinusoid_residual(x, t, d):
    """
    Returns a vector containing the residual values.

    INPUTS:
    d    Vector of measured values.
    t    Value of independent variable at the sampled points.
    x    Vector of parameters.
    """

    return d - sinusoid(x, t)


I am working on the file "2_Mathematical_Groundwork/2_11_least_squares.ipynb".
The first task is:
A final, important thing to note is that the Levenberg-Marquardt algorithm is already implemented in Python. It is used in scipy.optimise.leastsq. This is often useful for doing rapid numerical solution without the need for an analytic Jacobian. As a simple proof, we can call the built-in method to verify our results.
Can you write Python code for it?

x = np.array([8., 43.5, 1.05])

leastsq_x = leastsq(sinusoid_residual, x, args=(t, d))

print("scipy.optimize.leastsq: ", leastsq_x[0])
print("Our LM: ", solved_x)

plt.plot(t, d, label="Data")
plt.plot(t, sinusoid(leastsq_x[0], t), label="optimize.leastsq")
plt.xlabel("t")
plt.legend(loc='upper right')
plt.show()


x = np.array([8., 43.5, 1.05])

leastsq_x = leastsq(sinusoid_residual, x, args=(t, d))

print "scipy.optimize.leastsq: ", leastsq_x[0]
print "Our LM: ", solved_x

plt.plot(t, d, label="Data")
plt.plot(t, sinusoid(leastsq_x[0], t), label="optimize.leastsq")
plt.xlabel("t")
plt.legend(loc='upper right')
plt.show()


I am working on the file "2_Mathematical_Groundwork/2_11_least_squares.ipynb".
The first task is:
Import section specific modules:
Can you write Python code for it?

from scipy.optimize import leastsq
plt.rcParams['figure.figsize'] = (18, 6)

from IPython.display import HTML
HTML('../style/code_toggle.html')


I am working on the file "2_Mathematical_Groundwork/2_11_least_squares.ipynb".
The first task is:
In this case, the built-in method clearly fails. I have done this deliberately to illustrate a point - a given implementation of an algorithm might not be the best one for your application. In this case, the manner in which the tuning parameters are handled prevents the solution from converging correctly. This can be avoided by choosing a starting guess closer to the truth and once again highlights the importance of initial values in problems of this type.
Can you write Python code for it?

x = np.array([8., 35., 1.05])

leastsq_x = leastsq(sinusoid_residual, x, args=(t, d))

print("scipy.optimize.leastsq: ", leastsq_x[0])
print("Our LM: ", solved_x)

plt.plot(t, d, label="Data")
plt.plot(t, sinusoid(leastsq_x[0], t), label="optimize.leastsq")
plt.xlabel("t")
plt.legend(loc='upper right')
plt.show()


x = np.array([8., 35., 1.05])

leastsq_x = leastsq(sinusoid_residual, x, args=(t, d))

print "scipy.optimize.leastsq: ", leastsq_x[0]
print "Our LM: ", solved_x

plt.plot(t, d, label="Data")
plt.plot(t, sinusoid(leastsq_x[0], t), label="optimize.leastsq")
plt.xlabel("t")
plt.legend(loc='upper right')
plt.show()


I am working on the file "2_Mathematical_Groundwork/2_11_least_squares.ipynb".
The first task is:
The above is the main function of the Levenberg-Marquardt algorithm. The code may appear daunting at first, but all it does is implement the Levenberg-Marquardt update rule and some checks of convergence. We can now apply it to the problem with relative ease to obtain a numerical solution for our parameter vector.
Can you write Python code for it?

solved_x = levenberg_marquardt(d, t, x, sinusoid_residual, sinusoid_jacobian)
print solved_x


I am working on the file "2_Mathematical_Groundwork/2_11_least_squares.ipynb".
The first task is:
The fitted values are so close to the true values that it is almost impossible to differentiate between the red and green lines in the above plot. The true values have been omitted from the following plot to make it clearer that the numerical solution does an excellent job of arriving at the correct parameter values.
Can you write Python code for it?

plt.plot(t, d, label="Data")
plt.plot(t, sinusoid(solved_x, t), label="LM")
plt.xlabel("t")
plt.legend(loc='upper right')
plt.show()


I am working on the file "2_Mathematical_Groundwork/2_11_least_squares.ipynb".
The first task is:
The three functions defined above will be used frequently during the Levenberg-Marquardt solution procedure. The following few lines of code just set up the values we need to call the Levenberg-Marquardt solver.
Can you write Python code for it?

t = np.arange(-0.06, 0.06, 0.06/300)          #The points at which we will be taking our "measurements"
noise = 2*np.random.normal(size=(t.shape[0])) #A noise vector which we will use to manufacture "real" measurements.
true_x = np.array([10., 33.3, 0.52])          #The true values of our parameter vector.
x = np.array([8., 43.5, 1.05])                #Initial guess of parameter vector for our solver.

d = sinusoid(true_x, t) + noise #Our "observed" data, contructed from our true parameter values and the noise vector.
m = sinusoid(x, t)              #Our fitted function using the initial guess parameters.