I am working on the file "P2:_student_intervention/P2 Building a Student Intervention System.ipynb".
The first task is:
Logistic Regression
Description
Logistic Regression despite it's name is a stastical model that analyzes the relationship between multiple independent variables and a categorical dependent variable to estimates the probability of the output class by fitting data to a logistic curve. Although this is method is suitable for binary classification, multivariate classifications can be performed using a technique called one-vs-all where multiple models are trained and the class with the highest probability is chosen for output. Since, the model only outputs the probability of a class it's required that the we provide the threshold value that controls the output class.
Complexity
The complexity of the model is essentially governed by the optimizer used to solve the optimization of the cost-function. We shall consider the the generalized linear optimiser(Conjugate Gradient) here. ref, Logistic Regression for Data Mining and High Dimensional Classification
Space Complexity

The space complexity of logistic regression is O(1).

Time Complexity

Given m-samples in the training set, the time complexiy is roughly (less than) O(m).

General Applications
Logistic regression is ideally used for any kind of classification and is suitable as long as there is no overlapping of classes. This includes stock market analysis, market segmentation, Optical character recognition and digit recognition.
Strength

Logistic regression is intrinsically simple and hence a fast learner.
It has low variance when regularized and so is less prone to overfitting.
The classes need not be normally distributed and need not have equal variance in each class.
No linear relationship assumptions need to be made between the independent and dependent variables.

Weakness

It needs relatively high number of training sample to make reasonable assumptions.
It assumes a linear decision boundary and when this assumption is violated it tends to perform badly.

Why...
Can you write Python code for it?

display("LogisticRegression")
display(clfs["LogisticRegression"])
print "For the entire training set the LogisticRegression classifier\
 reports an F1 Score of {:.3f} on the testing set".format(clfs["LogisticRegression"]
                                                                 ["F1 Score on Testing Set"].loc[300])
plot_learning_curve(clfs["LogisticRegression"],"LogisticRegression")

"""
    Produce a table showing training time, prediction time,
    F1 score on training set and F1 score on test set, for each training set size.
    Note: You need to produce 3 such tables - one for each model.
"""
from IPython.display import display, HTML
for k,v in clfs.iteritems():
    display(k)
    display(v)


I am working on the file "P2:_student_intervention/P2 Building a Student Intervention System.ipynb".
The first task is:
Preparing the Data
In this section, we will prepare the data for modeling, training and testing.
Identify feature and target columns
It is often the case that the data you obtain contains non-numeric features. This can be a problem, as most machine learning algorithms expect numeric data to perform computations with.
Let's first separate our data into feature and target columns, and see if any features are non-numeric.<br/>
Note: For this dataset, the last column ('passed') is the target or label we are trying to predict.
Can you write Python code for it?

# Extract feature (X) and target (y) columns
feature_cols = list(student_data.columns[:-1])  # all columns but last are features
target_col = student_data.columns[-1]  # last column is the target/label
print "Feature column(s):-\n{}".format(feature_cols)
print
print "Target column: {}".format(target_col)

X_all = student_data[feature_cols]  # feature values for all students
y_all = student_data[target_col]  # corresponding targets/labels
print "\nFeature values:-"
print X_all.head()  # print the first 5 rows


I am working on the file "P2:_student_intervention/P2 Building a Student Intervention System.ipynb".
The first task is:
Preprocess feature columns
As you can see, there are several non-numeric columns that need to be converted! Many of them are simply yes/no, e.g. internet. These can be reasonably converted into 1/0 (binary) values.
Other columns, like Mjob and Fjob, have more than two values, and are known as categorical variables. The recommended way to handle such a column is to create as many columns as possible values (e.g. Fjob_teacher, Fjob_other, Fjob_services, etc.), and assign a 1 to one of them and 0 to all others.
These generated columns are sometimes called dummy variables, and we will use the pandas.get_dummies() function to perform this transformation.
Can you write Python code for it?

# Preprocess feature columns
def preprocess_features(X):
    """
    Function: Change the input categorical features into binary features
            using the pandas get_dummies function
    Params:
        X: A pandas DataFrame containing the categorical features.
    Returns:
        outX: A pandas DataFrame with dummy variables created
            in place of categorical features.
    """

    outX = pd.DataFrame(index=X.index)  # output dataframe, initially empty

    # Check each column
    for col, col_data in X.iteritems():
        # If data type is non-numeric, try to replace all yes/no values with 1/0
        if col_data.dtype == object:
            col_data = col_data.replace(['yes', 'no'], [1, 0])
        # Note: This should change the data type for yes/no columns to int

        # If still non-numeric, convert to one or more dummy variables
        if col_data.dtype == object:
            col_data = pd.get_dummies(col_data, prefix=col)  # e.g. 'school' => 'school_GP', 'school_MS'

        outX = outX.join(col_data)  # collect column(s) in output dataframe

    return outX

X_all = preprocess_features(X_all) #pre-process features
y_all = y_all.replace(['yes','no'],[1,0]) #pre-process target labels
print "Processed feature columns ({}):-\n{}".format(len(X_all.columns), list(X_all.columns))


I am working on the file "P2:_student_intervention/P2 Building a Student Intervention System.ipynb".
The first task is:
Split data into training and test sets
So far, we have converted all categorical features into numeric values. In this next step, we split the data (both features and corresponding labels) into training and test sets.
Can you write Python code for it?

from sklearn.cross_validation import train_test_split

# First, decide how many training vs test samples you want
num_all = student_data.shape[0]  # same as len(student_data)
num_train = 300  # about 75% of the data
num_test = num_all - num_train

# TODO: Then, select features (X) and corresponding labels (y) for the training and test sets
# Note: Shuffle the data or randomly select samples to avoid any bias due to ordering in the dataset
X_train,X_test,y_train,y_test = train_test_split(X_all,y_all,test_size=.24,random_state=42)
print "Training set: {} samples".format(X_train.shape[0])
print "Test set: {} samples".format(X_test.shape[0])
# Note: If you need a validation set, extract it from within training data


I am working on the file "P2:_student_intervention/P2 Building a Student Intervention System.ipynb".
The first task is:
Support Vector Machine
Description
Support Vector Machine (SVM) is a supervised machine learning algorithm that is used for both regression and classification problems. The classifier is best described as a large margin classifier. The intuition is to plot the features in an n-dimensional space and find a hyperplane that linearly separates the classes by optimizing the margin (distance between the hyperplane and the nearest points a.k.a support vectors). Since, it works only on linearly separable classes, non-linear hyperplanes are found using ingenious mathematical transformations known as "Kernels". This is often referred to as the "Kernel Trick"
Complexity
A number of factors affect the complexity of the support vector machine algorithm, these include the underlying metric space used to identify the margin, the kernel used and the type of SVM optimizer used.
However, the following is a ballpark reference for non-linear SVMs from Journal of Machine Learning Vol6.
Space Complexity

Given m-samples in the training set, the space complexity is roughly O(m2)

Time Complexity

Given m-samples in the training set, the time complexity is roughly O(m3)

General Applications
Since, SVMs are used where there are complex non-linear hypotheses to be learnt, they are applied in a variety of fields such as Computer Vision, Natural Language Processing, Neuroimaging, Bio-informatics.
Strength

Flexibility in space, using kernels.
Nonlinear functions can be learned with simplicity.
Since, it's a large margin classifier we can be certain of the decision boundary margin to be optimum.
Good out-of-sample generalization since it only uses the support vectors for classification.

Weakness

Can be prone to overfitting
For very large data sets they have relatively bad performance and can be slow.
The performance can be affected when the classes overlap at complicated levels and are noisy.

Why SVM ?
Since the data set is high...
Can you write Python code for it?

display("SVM")
display(clfs["SVC"])
print "For the entire training set the Support Vector Machine classifier\
 reports an F1 Score of {:.3f} on the testing set".format(clfs["SVC"]
                                                                 ["F1 Score on Testing Set"].loc[300])
plot_learning_curve(clfs["SVC"],"SVC")


I am working on the file "P2:_student_intervention/P2 Building a Student Intervention System.ipynb".
The first task is:
The final f1 score of the model on the test set is:
Can you write Python code for it?

print "F1 Score: {:.3f}".format(f1_score(y_test,y_pred))


I am working on the file "P2:_student_intervention/P2 Building a Student Intervention System.ipynb".
The first task is:
What is the theoretical O(n) time & space complexity in terms of input size?
What are the general applications of this model? What are its strengths and weaknesses?
Given what you know about the data so far, why did you choose this model to apply?
Naive Bayes
Description
The Naive Bayes classifier is a learning algorithm based on the Bayes's theorem. The fundamental idea is to use
Bayesian reasoning to identify the most likely of a set of hypotheses. It assumes an underlying
probabilistic distribution and updates the distribution using the probabilities of the outcome of training and
is a generative classifier. It's "Naive" in that it assumes conditional independence of features.
Complexity

The values below are referred from,
A Comparative Study of Semi-naive Bayes Methods in Classification Learning
Space Complexity

Given n-features and m-classes, the space complexity of a Naive Bayes classifier is roughly O(n*m).

Time Complexity

Given n-features, k-samples and m-classes, the time complexity of Naive Bayes classifier is roughly

for training - O(n*k + n*c)
for testing - O(n*k*c)

General Applications
Due to its computational simplicity(~Linear) and independence assumption Naive Bayes classifiers are widely used in a variety of applications that have relatively high dimensional features such as text classification, facial recognition, market segmentation and medical diagnosis.
Strength

Computationally tractable.
Fast learning speed.
Simplicity.
Independence from dimensionality.

Weakness

Poor performance when the independence assumption does not hold.
Has difficulty with zero-frequency values.
Very simple representation doesn't allow for rich hypotheses.

Why Naive Bayes ?
Given that the data at hand is high dimensional (~30 features) and has comparatively lesser number of training samples (~300 samples). A low variance/high bias classifier such a Naive Bayes should perform well in prediction as it's prone to avoid...
Can you write Python code for it?

display("MultinimialNB")
display(clfs["MultinomialNB"])
print "For the entire training set the MultinomialNB classifier\
 reports an F1 Score of {:.3f} on the testing set".format(clfs["MultinomialNB"]
                                                                 ["F1 Score on Testing Set"].loc[300])
plot_learning_curve(clfs["MultinomialNB"],"MultinomialNB")


I am working on the file "P2:_student_intervention/student_intervention.ipynb".
The first task is:
4. Training and Evaluating Models
Choose 3 supervised learning models that are available in scikit-learn, and appropriate for this problem. For each model:

What is the theoretical O(n) time & space complexity in terms of input size?
What are the general applications of this model? What are its strengths and weaknesses?
Given what you know about the data so far, why did you choose this model to apply?
Fit this model to the training data, try to predict labels (for both training and test sets), and measure the F1 score. Repeat this process with different training set sizes (100, 200, 300), keeping test set constant.

Produce a table showing training time, prediction time, F1 score on training set and F1 score on test set, for each training set size.
Note: You need to produce 3 such tables - one for each model.
Can you write Python code for it?

# Train a model
import time

def train_classifier(clf, X_train, y_train):
    print "Training {}...".format(clf.__class__.__name__)
    start = time.time()
    clf.fit(X_train, y_train)
    end = time.time()
    print "Done!\nTraining time (secs): {:.3f}".format(end - start)

# TODO: Choose a model, import it and instantiate an object
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression()

# Fit model to training data
train_classifier(clf, X_train, y_train)  # note: using entire training set here
print clf  # you can inspect the learned model by printing it

# Predict on training set and compute F1 score
from sklearn.metrics import f1_score

def predict_labels(clf, features, target):
    print "Predicting labels using {}...".format(clf.__class__.__name__)
    start = time.time()
    y_pred = clf.predict(features)
    end = time.time()
    print "Done!\nPrediction time (secs): {:.3f}".format(end - start)
    return f1_score(target.values, y_pred, pos_label='yes')

train_f1_score = predict_labels(clf, X_train, y_train)
print "F1 score for training set: {}".format(train_f1_score)

# Predict on test data
print "F1 score for test set: {}".format(predict_labels(clf, X_test, y_test))

# Train and predict using different training set sizes
def train_predict(clf, X_train, y_train, X_test, y_test):
    print "------------------------------------------"
    print "Training set size: {}".format(len(X_train))
    train_classifier(clf, X_train, y_train)
    print "F1 score for training set: {}".format(predict_labels(clf, X_train, y_train))
    print "F1 score for test set: {}".format(predict_labels(clf, X_test, y_test))

# TODO: Run the helper function above for desired subsets of training data
# Note: Keep the test set constant

for i in [100,200,300]:
    train_predict(clf,X_train.iloc[:i],y_train.iloc[:i],X_test,y_test)

# TODO: Train and predict using two other models
from sklearn.svm import SVC
clf2 = SVC()
for i in [100,200,300]:
   ...


I am working on the file "P2:_student_intervention/student_intervention.ipynb".
The first task is:
5. Choosing the Best Model

Based on the experiments you performed earlier, in 1-2 paragraphs explain to the board of supervisors what single model you chose as the best model. Which model is generally the most appropriate based on the available data, limited resources, cost, and performance?
In 1-2 paragraphs explain to the board of supervisors in layman's terms how the final model chosen is supposed to work (for example if you chose a Decision Tree or Support Vector Machine, how does it make a prediction).
Fine-tune the model. Use Gridsearch with at least one important parameter tuned and with at least 3 settings. Use the entire training set for this.
What is the model's final F1 score?
Can you write Python code for it?

from sklearn.grid_search import GridSearchCV
from sklearn.cross_validation import StratifiedShuffleSplit
from sklearn.metrics import f1_score
from sklearn.metrics import make_scorer
f1_scorer = make_scorer(f1_score, pos_label="yes")
params = {"C":np.linspace(0.001,10,200),"penalty":["l1","l2"],"solver":["liblinear"]}
clf = LogisticRegression(random_state=42)
cv = StratifiedShuffleSplit(y_all,n_iter=25,test_size=0.24)

gclf = GridSearchCV(clf,param_grid=params,scoring=f1_scorer,cv=cv,n_jobs=4)
gclf.fit(X_all,y_all)

print gclf.best_params_

print gclf.best_score_


I am working on the file "P2:_student_intervention/student_intervention.ipynb".
The first task is:
Now, can you find out the following facts about the dataset?
- Total number of students
- Number of students who passed
- Number of students who failed
- Graduation rate of the class (%)
- Number of features
Use the code block below to compute these values. Instructions/steps are marked using TODOs.
Can you write Python code for it?

# TODO: Compute desired values - replace each '?' with an appropriate expression/function call
n_students = len(student_data.index)
n_features = len(student_data.columns)-1
n_passed = len(student_data[student_data.passed=="yes"].index)
n_failed = n_students-n_passed
grad_rate = float(n_passed)/n_students
print "Total number of students: {}".format(n_students)
print "Number of students who passed: {}".format(n_passed)
print "Number of students who failed: {}".format(n_failed)
print "Number of features: {}".format(n_features)
print "Graduation rate of the class: {:.2f}%".format(grad_rate)


I am working on the file "P2:_student_intervention/student_intervention.ipynb".
The first task is:
Preprocess feature columns
As you can see, there are several non-numeric columns that need to be converted! Many of them are simply yes/no, e.g. internet. These can be reasonably converted into 1/0 (binary) values.
Other columns, like Mjob and Fjob, have more than two values, and are known as categorical variables. The recommended way to handle such a column is to create as many columns as possible values (e.g. Fjob_teacher, Fjob_other, Fjob_services, etc.), and assign a 1 to one of them and 0 to all others.
These generated columns are sometimes called dummy variables, and we will use the pandas.get_dummies() function to perform this transformation.
Can you write Python code for it?

# Preprocess feature columns
def preprocess_features(X):
    outX = pd.DataFrame(index=X.index)  # output dataframe, initially empty

    # Check each column
    for col, col_data in X.iteritems():
        # If data type is non-numeric, try to replace all yes/no values with 1/0
        if col_data.dtype == object:
            col_data = col_data.replace(['yes', 'no'], [1, 0])
        # Note: This should change the data type for yes/no columns to int

        # If still non-numeric, convert to one or more dummy variables
        if col_data.dtype == object:
            col_data = pd.get_dummies(col_data, prefix=col)  # e.g. 'school' => 'school_GP', 'school_MS'

        outX = outX.join(col_data)  # collect column(s) in output dataframe

    return outX

X_all = preprocess_features(X_all)
print "Processed feature columns ({}):-\n{}".format(len(X_all.columns), list(X_all.columns))


I am working on the file "P2:_student_intervention/student_intervention.ipynb".
The first task is:
Split data into training and test sets
So far, we have converted all categorical features into numeric values. In this next step, we split the data (both features and corresponding labels) into training and test sets.
Can you write Python code for it?


I am working on the file "P2P 1 - Datos de cigarrillos.ipynb".
The first task is:
Además, vamos a añadir el tipo de cigarro para que la tabla quede como la presentada en el curso.
Can you write Python code for it?

df["Clases"] = ['Rubio', 'Negro', 'Negro', 'Rubio', 'Rubio',
                'Negro', 'Rubio', 'Rubio', 'Negro', 'Rubio',
                'Rubio', 'Rubio', 'Rubio', 'Rubio', 'Rubio',
                'Rubio', 'Negro', 'Rubio', 'Negro', 'Rubio',
                'Negro', 'Rubio', 'Negro', 'Negro', 'Rubio']

df[["Clases", "Alquitrán", "Nicotina", "Peso", "Monóxido"]]


I am working on the file "P2P 1 - Datos de cigarrillos.ipynb".
The first task is:
Observamos una gran variabilidad de los contenidos de alquitrán y monóxido de carbono, mientras que las cantidades de nicotina son más estables y el peso de los cigarrillos prácticamente no cambia. Los resultados son similares a los obtenidos estudiando la media y su dispersión.

Parte 3: Box plots
Por último, utilizamos la biblioteca matplotlib para representar los diagramas de cajas. Por defecto son de tipo Tukey, es decir: los bigotes llegan hasta 1.5 veces el recorrido intercuartílico por encima del percentil 75 y por debajo del percentil 25.
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use("fivethirtyeight")

plt.figure(figsize=(8, 8))

plt.subplot(2, 2, 1)
df.boxplot("Monóxido", return_type='axes')

plt.subplot(2, 2, 2)
df.boxplot("Alquitrán", return_type='axes')
plt.ylim(0, 35)  # Para ver una medida discordante

plt.subplot(2, 2, 3)
df.boxplot("Nicotina", return_type='axes')

plt.subplot(2, 2, 4)
df.boxplot("Peso", return_type='axes')
plt.ylim(0, 1.20)  # Importante


I am working on the file "P2P 1 - Datos de cigarrillos.ipynb".
The first task is:
Recuperamos además la mediana y el recorrido intercuartílico:
Can you write Python code for it?

df.median()

iqr = df.quantile(.75) - df.quantile(.25)
iqr


I am working on the file "P2P 1 - Datos de cigarrillos.ipynb".
The first task is:
Tanto el monóxido como el peso presentan distribuciones bastante simétricas, mientras que el alquitrán tiene un claro sesgo positivo. Especial atención merece el peso en este caso, pues una correcta escala vertical es esencial para no percibir una variabilidad errónea. Tanto en los datos de nicotina como en los de alquitrán se aprecian sendos valores discordantes, que invitarían a no comprar esa marca de cigarrillos.
Can you write Python code for it?

df[df["Alquitrán"] > df["Alquitrán"].quantile(.75) + 1.5 * iqr["Alquitrán"]]


I am working on the file "P2P 1 - Datos de cigarrillos.ipynb".
The first task is:
Vamos a cargar los en Python con pandas. pandas es una biblioteca de Python para trabajar con tablas de datos (llamados DataFrames) de forma cómoda. En Pybonacci escribimos un tutorial de pandas desde lo más básico a usos un poco más intermedios.
Can you write Python code for it?

import pandas as pd

df = pd.read_csv("cigarettes.dat", delim_whitespace=True, header=None,
                 names=["Marca", "Alquitrán", "Nicotina", "Peso", "Monóxido"])
df.head()


I am working on the file "P2_Building_a_Student_Intervention_System/P2_Student_Intervention_System.ipynb".
The first task is:
3. Preparing the Data
In this section, we will prepare the data for modeling, training and testing.
Identify feature and target columns
It is often the case that the data you obtain contains non-numeric features. This can be a problem, as most machine learning algorithms expect numeric data to perform computations with.
Let's first separate our data into feature and target columns, and see if any features are non-numeric.<br/>
Note: For this dataset, the last column ('passed') is the target or label we are trying to predict.
Can you write Python code for it?

# %%capture
# Extract feature (X) and target (y) columns
feature_cols = list(student_data.columns[:-1])  # all columns but last are features
target_col = student_data.columns[-1]           # last column is the target/label
print "Feature column(s):-\n{}".format(feature_cols)
print "Target column: {}".format(target_col)

X_all = student_data[feature_cols]              # feature values for all students
y_all = student_data[target_col]                # corresponding targets/labels
print "\nFeature values:-"
print X_all.head()                              # print the first 5 rows


I am working on the file "P2_Building_a_Student_Intervention_System/P2_Student_Intervention_System.ipynb".
The first task is:
4. Training and Evaluating Models
Choose 3 supervised learning models that are available in scikit-learn, and appropriate for this problem. For each model:

What are the general applications of this model? What are its strengths and weaknesses?
Given what you know about the data so far, why did you choose this model to apply?
Fit this model to the training data, try to predict labels (for both training and test sets), and measure the F1 score. Repeat this process with different training set sizes (100, 200, 300), keeping test set constant.

Produce a table showing training time, prediction time, F1 score on training set and F1 score on test set, for each training set size.
Note: You need to produce 3 such tables - one for each model.
Can you write Python code for it?

# Train a model
import time
from sklearn.linear_model import SGDClassifier, LogisticRegression
from sklearn.ensemble import AdaBoostClassifier, RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.metrics import f1_score

def train_classifier(clf, X_train, y_train):
    print "\nTraining {}...".format(clf.__class__.__name__)
    start = time.time()
    clf.fit(X_train, y_train)
    end = time.time()
    duration = end - start
    print "Training time (secs): {:.4f}".format(duration)
    return duration

def predict_labels(clf, features, target):
    # print "Predicting labels using {}...".format(clf.__class__.__name__)
    start = time.time()
    y_pred = clf.predict(features)
    end = time.time()
    print "Prediction time (secs): {:.4f}".format(end - start)
    return f1_score(target.values, y_pred, pos_label='yes')

def train_predict(clf, X_train, y_train, X_test, y_test):
    print "----------"
    print "Training set size: {}".format(len(X_train))
    train_classifier(clf, X_train, y_train)
    print "Training set:"
    train_f1_score = predict_labels(clf, X_train, y_train)
    print "Testing set:"
    test_f1_score = predict_labels(clf, X_test, y_test)
    print "F1 score for training set: {}".format(train_f1_score)
    print "F1 score for test set: {}".format(test_f1_score)
    return train_f1_score, test_f1_score

# TODO: Choose a model, import it and instantiate an object
# TODO: Run the helper function above for desired subsets of training data

clfs = [DecisionTreeClassifier(random_state=42),
        KNeighborsClassifier(),
        LogisticRegression(random_state=42)]

for clf in clfs:
    print "============================================="

    # Fit model to training data
    train_classifier(clf, X_train, y_train)  # note: using entire training set here

    # Predict on training & testing set and compute F1 score
    train_f1_score...


I am working on the file "P2_Building_a_Student_Intervention_System/P2_Student_Intervention_System.ipynb".
The first task is:
5. Choosing the Best Model

Based on the experiments you performed earlier, in 1-2 paragraphs explain to the board of supervisors what single model you chose as the best model. Which model is generally the most appropriate based on the available data, limited resources, cost, and performance?
In 1-2 paragraphs explain to the board of supervisors in layman's terms how the final model chosen is supposed to work (for example if you chose a Decision Tree or Support Vector Machine, how does it make a prediction).
Fine-tune the model. Use Gridsearch with at least one important parameter tuned and with at least 3 settings. Use the entire training set for this.
What is the model's final F1 score?
Can you write Python code for it?

%%capture
# Takes around 6 mins to run on a 4 Ghz, quad-core machine

# TODO: Fine-tune your model and report the best F1 score
import time
import numpy as np
import pandas as pd
from sklearn.cross_validation import train_test_split
from sklearn.ensemble import AdaBoostClassifier, RandomForestClassifier
from sklearn.grid_search import GridSearchCV
from sklearn.linear_model import SGDClassifier, LogisticRegression
from sklearn.metrics import f1_score, precision_score, recall_score, accuracy_score
from sklearn.preprocessing import LabelEncoder
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier

# time the script
start = time.time()

# calc_scores (f1_score, accuracy_score, recall_score, precision_score)
def calc_scores(y, y_pred):
    return (f1_score       (y, y_pred),
            accuracy_score (y, y_pred),
            recall_score   (y, y_pred),
            precision_score(y, y_pred))

# import data
student_data = pd.read_csv("student-data.csv")

# extract feature (X) and target (y) columns
feature_cols = list(student_data.columns[:-1])
target_col   = student_data.columns[-1]
le_y         = LabelEncoder()

X_all = pd.get_dummies(student_data[feature_cols])
y_all = student_data[target_col]
y_all = le_y.fit_transform(y_all)

# initialize classifiers for evaluations of performance
clfs_set = [AdaBoostClassifier(),
            DecisionTreeClassifier(),
            KNeighborsClassifier(),
            LogisticRegression(),
            SVC(),
            SGDClassifier(),
            RandomForestClassifier()]

clfs_best    = []
train_scores = []
test_scores  = []

# building param_grids for GridSearchCV
ada_grid = {'algorithm': ['SAMME', 'SAMME.R'],
            'n_estimators': np.linspace(1, 6, num=5).astype(int),
            'learning_rate': (0.001, 0.01, 0.1, 1, 10)}

dt_grid = {'criterion': ['gini', 'entropy'],
           'max_features': ['auto', 'sqrt', 'log2'],
           'max_depth': np.linspace(1, 10, num=10),
           'min_samples_split': np.linspace(2, 10, 1),
          ...


I am working on the file "P2_Building_a_Student_Intervention_System/P2_Student_Intervention_System.ipynb".
The first task is:
Preprocess feature columns
As you can see, there are several non-numeric columns that need to be converted! Many of them are simply yes/no, e.g. internet. These can be reasonably converted into 1/0 (binary) values.
Other columns, like Mjob and Fjob, have more than two values, and are known as categorical variables. The recommended way to handle such a column is to create as many columns as possible values (e.g. Fjob_teacher, Fjob_other, Fjob_services, etc.), and assign a 1 to one of them and 0 to all others.
These generated columns are sometimes called dummy variables, and we will use the pandas.get_dummies() function to perform this transformation.
Can you write Python code for it?

# Preprocess feature columns
def preprocess_features(X):

    # output dataframe, initially empty
    outX = pd.DataFrame(index=X.index)

    # Check each column
    for col, col_data in X.iteritems():
        # If data type is non-numeric, try to replace all yes/no values with 1/0
        if col_data.dtype == object:
            col_data = col_data.replace(['yes', 'no'], [1, 0])
        # Note: This should change the data type for yes/no columns to int
        # If still non-numeric, convert to one or more dummy variables
        if col_data.dtype == object:
            col_data = pd.get_dummies(col_data, prefix=col)  # e.g. 'school' => 'school_GP', 'school_MS'

        outX = outX.join(col_data)  # collect column(s) in output dataframe

    return outX

X_all = preprocess_features(X_all)
# X_all = pd.get_dummies(X_all)

print "Processed feature columns ({}):-\n{}".format(len(X_all.columns), list(X_all.columns))


I am working on the file "P3 Data Munge/Data Wrangling Final Project.ipynb".
The first task is:
3. Other Ideas About the Datasets
Cleaning Up OpenStreetMap Data
Open Street Map data is not only incomplete but also filled with inconsistencies and lack of structure. Some ideas need to be discussed at a public forum to organize OpenStreetMap data into a more structured form to make this public open source collaboration more relevant compared with Google Maps, Apple Maps, and Yelp.
Improvement 1: Fill out the city, county, zip code for all the nodes
Can you write Python code for it?

bayarea.find({"type": "node"}).count()

bayarea.find({"type": "node", "address.city": {"$exists": 0}}).count()

bayarea.find({"type": "node", "address.county": {"$exists": 0}}).count()

bayarea.find({"type": "node", "address.postcode": {"$exists": 0}}).count()


I am working on the file "P3 Data Munge/Data Wrangling Final Project.ipynb".
The first task is:
Data Wrangling with OpenStreetMap <br/> Final Project
Author: William Truong
Map Area: San Francisco Bay Area, CA, United States
Sources:

Mapzen Metro Extract Link
Audit and Shaping into JSON IPython Notebook
Audit and Shaping into JSON IPython Notebook (html)

1. Problems Encountered in the Map
Several inconsistencies were discovered during the audit:

Concerning the "addr:street" key
Street names that don't share a common suffix ("1st Street vs. 1st St")
Inconsistent use of N vs North, S vs South, etc.
Address number included in the street names ("Adrian court, Suite A")
Intersections are used for some of the values (Pacheco Blvd @ Blum Rd N Of Sr 4)
Landmarks are used for some of the values (Tanforan Shopping Center).
Discernable amount of keys prefixed with tiger
Postal codes (95624 vs. 95624-6003.

"addr:street" conversion"
In the process of converting the "addr:street" keys into json under the "address"  name, street name suffixes were only fixed since the analysis that was performed did not concern street names in particular.
tiger data
There was a big number of keys prefixed with tiger. After some researching, it was found that tiger keys originate from Census data. OpenStreetMap initially used tiger data to import mass amounts of map information during the early years.  Future census data will no longer be imported and tiger data is being slowly converted to fit the recommended OpenStreetMap schema. Until it is properly converted, some of these key values need to be converted to the "address" dictionary.
Two keys were converted:
"tiger":"zip_left" -> "address":"postcode"
"tiger":"county" -> "address":"county"
These two keys were converted only if there wasn't a value in place for those key values located under the address dictionary.
2. Data Overview
File sizes
san-francisco-bay_california.osm .............. 1.9G
san-francisco-bay_california.osm.json ......... 2.2G
Number of documents
Can you write Python code for it?


I am working on the file "P3 Data Munge/Data Wrangling Final Project.ipynb".
The first task is:
For the city, county, and postcode there is only < 1% coverage for all the nodes. During the shaping of the data to JSON, these values can be programmatically filled out if I had a geographical database that can fill out the city, county, and postcode from latitudinal and longitudinal coordinates. Once these values are completed, then more inferences can be made for city-city, county-county, or post code - post code comparisons.
It also would not hurt to recommend to OpenStreetMap to automatically fill these fields for a user when a node or way is inputted or editted.
Improvement 2: Making Key Values More Relevant
At this stage, I believe OpenStreetMap data for the bay area is too sparse and fragmented to be able to make solid inferences from it. If we take a look at the database for the types of values for the leisure keys:
Can you write Python code for it?


I am working on the file "P3 Data Munge/Data Wrangling Final Project.ipynb".
The first task is:
Number of nodes
Can you write Python code for it?


I am working on the file "P3 Data Munge/Data Wrangling Final Project.ipynb".
The first task is:
Number of ways
Can you write Python code for it?


I am working on the file "P3 Data Munge/Data Wrangling Final Project.ipynb".
The first task is:
Top 10 types of amenities
Can you write Python code for it?

pipeline = [{"$match": {"amenity": {"$ne": None}}},
                       {"$group": {"_id": "$amenity",
                                   "count": {"$sum": 1}}},
                       {"$sort": {"count": -1}},
                       {"$limit": 10}]
result = bayarea.aggregate(pipeline)
pprint(result)


I am working on the file "P3 Data Munge/Data Wrangling Final Project.ipynb".
The first task is:
Top 10 types of leisurely activities
Can you write Python code for it?

pipeline = [{"$match": {"leisure": {"$exists": 1}}},
            {"$group": {"_id": "$leisure", "count": {"$sum": 1}}},
                       {"$sort": {"count": -1}},
                       {"$limit": 10}]
result = bayarea.aggregate(pipeline)
pprint(result)


I am working on the file "P3 Data Munge/Data Wrangling Final Project.ipynb".
The first task is:
Top cities with the most ways and nodes
Can you write Python code for it?

pipeline = [{"$match": {"address.city": {"$exists": 1}}},
            {"$group": {"_id": "$address.city", "count": {"$sum": 1}}},
            {"$sort": {"count": -1}},
            {"$limit": 10}]
result = bayarea.aggregate(pipeline)
pprint(result)


I am working on the file "P3 wrangle_data/.ipynb_checkpoints/DataWrangling_ganga-checkpoint.ipynb".
The first task is:

3. Additional Data Exploration using MongoDB
I am going to use the pipeline function to retrive data from the database
Can you write Python code for it?

def pipeline(city):
    p= [{"$match":{"created.user":{"$exists":1},
                                          "city":city}},
                 {"$group": {"_id": {"City":"$city",
                                     "User":"$created.user"},
                            "contribution": {"$sum": 1}}},
                 {"$project": {'_id':0,
                               "City":"$_id.City",
                               "User_Name":"$_id.User",
                               "Total_contribution":"$contribution"}},
                 {"$sort": {"Total_contribution": -1}},
                 {"$limit" : 5 }]
    return p
result1 =mongo_db["cities"].aggregate(pipeline('bunkyo'))
for each in result1:
    print(each)
print("\n")
result2 =mongo_db["cities"].aggregate(pipeline('hoodi'))
for each in result2:
    print(each)


I am working on the file "P3 wrangle_data/.ipynb_checkpoints/DataWrangling_ganga-checkpoint.ipynb".
The first task is:

Merging Both cities
These two maps are selected since ,right now i am living at Hoodi,Bengaluru. And one day i want do my masters in japan in robotics,so i had selected locality of University of tokyo, Bunkyo.I really wanted to explore differences between the regions.
I need to add a tag named "city" so i can differentiate them from the database.

2. Data Overview
This section contains basic statistics about the dataset and the MongoDB queries used to gather them.
File Sizes
Can you write Python code for it?

bangalore.osm -40MB
bangalore.osm.json-51MB
tokyo1.osm- 82MB
tokyo1.osm.json-102.351MB


I am working on the file "P3 wrangle_data/.ipynb_checkpoints/DataWrangling_ganga-checkpoint.ipynb".
The first task is:
As compared to hoodi ,bunkyo have few atms,And parking can be commonly found in bunkyo locality
<hr>

popular places of worship
Can you write Python code for it?

    p = [{"$match":{"Additional Information.amenity":{"$exists":1},
                                  "Additional Information.amenity":"place_of_worship",
                                  "city":city}},
                       {"$group":{"_id": {"City":"$city",
                                          "Religion":"$Additional Information.religion"},
                                  "count":{"$sum":1}}},
                       {"$project":{"_id":0,
                                    "City":"$_id.City",
                                    "Religion":"$_id.Religion",
                                    "Count":"$count"}},
                       {"$sort":{"Count":-1}},
                       {"$limit":6}]


I am working on the file "P3 wrangle_data/.ipynb_checkpoints/DataWrangling_ganga-checkpoint.ipynb".
The first task is:
The top contributors for hoodi are no where near since bunkyo being a more compact region than hoodi ,there are more places to contribute.
<hr>

To get the top Amenities in Hoodi and Bunkyo
I will be showing the pipeline that will go in the above mentioned "Pipleline" function
Can you write Python code for it?

pipeline=[{"$match":{"Additional Information.amenity":{"$exists":1},
                                          "city":city}},
                 {"$group": {"_id": {"City":"$city",
                                    "Amenity":"$Additional Information.amenity"},
                            "count": {"$sum": 1}}},
                 {"$project": {'_id':0,
                               "City":"$_id.City",
                               "Amenity":"$_id.Amenity",
                               "Count":"$count"}},
                 {"$sort": {"Count": -1}},
                 {"$limit" : 10 }]


I am working on the file "P3 wrangle_data/.ipynb_checkpoints/DataWrangling_ganga-checkpoint.ipynb".
The first task is:
There are quite a few ATM in Bunkyo as compared to hoodi
<hr>

Martial arts or Dojo Center near locality
Can you write Python code for it?

##  Martial arts or Dojo Center near locality
import re

pat = re.compile(r'dojo', re.I)
d=mongo_db.cities.aggregate([{"$match":{ "$or": [ { "Additional Information.name": {'$regex': pat}}
                                                   ,{"Additional Information.amenity": {'$regex': pat}}]}}
                            ,{"$group":{"_id":{"City":"$city"
                             , "Sport":"$Additional Information.name"}}}])
for each in d:
    print(each)

bunkyo:
{u'_id': {u'City': u'bunkyo', u'Sport': u'Aikikai Hombu Dojo'}}
{u'_id': {u'City': u'bunkyo', u'Sport': u'Kodokan Dojo'}}

hoodi:
{u'_id': {u'City': u'hoodi', u'Sport': u"M S Gurukkal's Kalari Academy"}}


I am working on the file "P3 wrangle_data/.ipynb_checkpoints/DataWrangling_ganga-checkpoint.ipynb".
The first task is:
Total Number of contributor.
Can you write Python code for it?

print "Constributors:", len(mongo_db.cities.distinct("created.user"))

Contributors: 858


I am working on the file "P3 wrangle_data/.ipynb_checkpoints/DataWrangling_ganga-checkpoint.ipynb".
The first task is:
{u'Count': 582, u'City': u'bunkyo'}
{u'Food': u'japanese', u'City': u'bunkyo', u'Count': 192}
{u'Food': u'chinese', u'City': u'bunkyo', u'Count': 126}
{u'Food': u'italian', u'City': u'bunkyo', u'Count': 69}
{u'Food': u'indian', u'City': u'bunkyo', u'Count': 63}
{u'Food': u'sushi', u'City': u'bunkyo', u'Count': 63}
{u'Count': 213, u'City': u'hoodi'}
{u'Food': u'regional', u'City': u'hoodi', u'Count': 75}
{u'Food': u'indian', u'City': u'hoodi', u'Count': 69}
{u'Food': u'chinese', u'City': u'hoodi', u'Count': 36}
{u'Food': u'international', u'City': u'hoodi', u'Count': 24}
{u'Food': u'Andhra', u'City': u'hoodi', u'Count': 21}
Indian style cusine in Bunkyo seems famous, Which will be better if i go to japan and do my higher studies there.
<hr>

popular fast food joints
Can you write Python code for it?

 p = [{"$match":{"Additional Information.amenity":{"$exists":1},
                                 "Additional Information.amenity":"fast_food",
                                 "city":city}},
                      {"$group":{"_id":{"City":"$city",
                                        "Food":"$Additional Information.cuisine"},
                                 "count":{"$sum":1}}},
                      {"$project":{"_id":0,
                                  "City":"$_id.City",
                                  "Food":"$_id.Food",
                                  "Count":"$count"}},
                      {"$sort":{"Count":-1}},
                      {"$limit":6}]


I am working on the file "P3 wrangle_data/DataWrangling_ganga.ipynb".
The first task is:

3. Additional Data Exploration using MongoDB
I am going to use the pipeline function to retrive data from the database
Can you write Python code for it?


I am working on the file "P3 wrangle_data/DataWrangling_ganga.ipynb".
The first task is:
As compared to hoodi ,bunkyo have few atms,And parking can be commonly found in bunkyo locality
<hr>

popular places of worship
Can you write Python code for it?


I am working on the file "P3 wrangle_data/DataWrangling_ganga.ipynb".
The first task is:
As expected japan is popular with buddism,
but india being a secular country it will be having most of the reglious places of worship,where hinduism being majority
<hr>

popular restaurants
Can you write Python code for it?

p = [{"$match":{"Additional Information.amenity":{"$exists":1},
                                 "Additional Information.amenity":"restaurant",
                                 "city":city}},
                      {"$group":{"_id":{"City":"$city",
                                        "Food":"$Additional Information.cuisine"},
                                 "count":{"$sum":1}}},
                      {"$project":{"_id":0,
                                  "City":"$_id.City",
                                  "Food":"$_id.Food",
                                  "Count":"$count"}},
                      {"$sort":{"Count":-1}},
                      {"$limit":6}]


I am working on the file "P3 wrangle_data/DataWrangling_ganga.ipynb".
The first task is:
Bunkyo: 1268292
Hoodi: 667842

Number of node nodes.
Can you write Python code for it?

print "Bunkyo:",mongo_db.cities.find({"type":"node",
                                    'city':'bunkyo'}).count()
print "Hoodi:",mongo_db.cities.find({"type":"node",
                                    'city':'hoodi'}).count()

Bunkyo: 1051170
Hoodi: 548862


I am working on the file "P3 wrangle_data/DataWrangling_ganga.ipynb".
The first task is:
Burger seems very popular among japanese in fast foods,i was expecting ramen to be more popular
, but in hoodi pizza is really common,being a metropolitan city.
<hr>

ATM's near locality
Can you write Python code for it?

    p = [{"$match":{"Additional Information.amenity":{"$exists":1},
                                 "Additional Information.amenity":"atm",
                                 "city":city}},
                      {"$group":{"_id":{"City":"$city",
                                        "Name":"$Additional Information.name:en"},
                                 "count":{"$sum":1}}},
                      {"$project":{"_id":0,
                                  "City":"$_id.City",
                                  "Name":"$_id.Name",
                                  "Count":"$count"}},
                      {"$sort":{"Count":-1}},
                      {"$limit":4}]


I am working on the file "P3 wrangle_data/DataWrangling_ganga.ipynb".
The first task is:
I wanted to learn martial arts ,
In japan is known for its akido and other ninjistsu martial arts , where i can find some in bunkyo
Where as in hoodi,india Kalaripayattu Martial Arts are one of the ancient arts that ever existed.
<hr>

most popular shops.
Can you write Python code for it?

 p = [{"$match":{"Additional Information.shop":{"$exists":1},
                                          "city":city}},
                       {"$group":{"_id":{"City":"$city",
                                  "Shop":"$Additional Information.shop"},
                           "count":{"$sum":1}}},
                       {"$project": {'_id':0,
                                     "City":"$_id.City",
                                     "Shop":"$_id.Shop",
                                     "Count":"$count"}},
                       {"$sort":{"Count":-1}},
                       {"$limit":10}]

{u'Shop': u'convenience', u'City': u'bunkyo', u'Count': 1035}
{u'Shop': u'clothes', u'City': u'bunkyo', u'Count': 282}
{u'Shop': u'books', u'City': u'bunkyo', u'Count': 225}
{u'Shop': u'mobile_phone', u'City': u'bunkyo', u'Count': 186}
{u'Shop': u'confectionery', u'City': u'bunkyo', u'Count': 156}
{u'Shop': u'supermarket', u'City': u'bunkyo', u'Count': 150}
{u'Shop': u'computer', u'City': u'bunkyo', u'Count': 126}
{u'Shop': u'hairdresser', u'City': u'bunkyo', u'Count': 90}
{u'Shop': u'electronics', u'City': u'bunkyo', u'Count': 90}
{u'Shop': u'anime', u'City': u'bunkyo', u'Count': 90}

{u'Shop': u'clothes', u'City': u'hoodi', u'Count': 342}
{u'Shop': u'supermarket', u'City': u'hoodi', u'Count': 129}
{u'Shop': u'bakery', u'City': u'hoodi', u'Count': 120}
{u'Shop': u'shoes', u'City': u'hoodi', u'Count': 72}
{u'Shop': u'furniture', u'City': u'hoodi', u'Count': 72}
{u'Shop': u'sports', u'City': u'hoodi', u'Count': 66}
{u'Shop': u'electronics', u'City': u'hoodi', u'Count': 60}
{u'Shop': u'beauty', u'City': u'hoodi', u'Count': 54}
{u'Shop': u'car', u'City': u'hoodi', u'Count': 36}
{u'Shop': u'convenience', u'City': u'hoodi', u'Count': 36}

The general stores are quite common in both the places


I am working on the file "P3 wrangle_data/DataWrangling_ganga.ipynb".
The first task is:
Number of documents
Can you write Python code for it?


I am working on the file "P3 wrangle_data/DataWrangling_ganga.ipynb".
The first task is:
Number of way nodes.
Can you write Python code for it?

print "Bunkyo:",mongo_db.cities.find({'type':'way',
                                  'city':'bunkyo'}).count()
print "Hoodi:",mongo_db.cities.find({'type':'way',
                                  'city':'hoodi'}).count()

Bunkyo: 217122
Hoodi: 118980


I am working on the file "P3 wrangle_data/DataWrangling_ganga.ipynb".
The first task is:
The top contributors for hoodi are no where near since bunkyo being a more compact region than hoodi ,there are more places to contribute.
<hr>

To get the top Amenities in Hoodi and Bunkyo
I will be showing the pipeline that will go in the above mentioned "Pipleline" function
Can you write Python code for it?


I am working on the file "P3 wrangle_data/DataWrangling_ganga.ipynb".
The first task is:
most popular supermarkets
Can you write Python code for it?

    p = [{"$match":{"Additional Information.shop":{"$exists":1},
                           "city":city,
                           "Additional Information.shop":"supermarket"}},
                       {"$group":{"_id":{"City":"$city",
                                  "Supermarket":"$Additional Information.name"},
                           "count":{"$sum":1}}},
                       {"$project": {'_id':0,
                                     "City":"$_id.City",
                                     "Supermarket":"$_id.Supermarket",
                                     "Count":"$count"}},
                       {"$sort":{"Count":-1}},
                       {"$limit":5}]

{u'Count': 120, u'City': u'bunkyo'}
{u'Count': 9, u'City': u'bunkyo', u'Supermarket': u'Maruetsu'}
{u'Count': 3, u'City': u'bunkyo', u'Supermarket': u"Y's Mart"}
{u'Count': 3, u'City': u'bunkyo', u'Supermarket': u'SainE'}
{u'Count': 3, u'City': u'bunkyo', u'Supermarket': u'DAIMARU Peacock'}

{u'Count': 9, u'City': u'hoodi', u'Supermarket': u'Reliance Fresh'}
{u'Count': 9, u'City': u'hoodi'}
{u'Count': 6, u'City': u'hoodi', u'Supermarket': u"Nilgiri's"}
{u'Count': 3, u'City': u'hoodi', u'Supermarket': u'Royal Mart Supermarket'}
{u'Count': 3, u'City': u'hoodi', u'Supermarket': u'Safal'}


I am working on the file "P3 wrangle_data/DataWrangling_ganga.ipynb".
The first task is:
{u'Count': 582, u'City': u'bunkyo'}
{u'Food': u'japanese', u'City': u'bunkyo', u'Count': 192}
{u'Food': u'chinese', u'City': u'bunkyo', u'Count': 126}
{u'Food': u'italian', u'City': u'bunkyo', u'Count': 69}
{u'Food': u'indian', u'City': u'bunkyo', u'Count': 63}
{u'Food': u'sushi', u'City': u'bunkyo', u'Count': 63}
{u'Count': 213, u'City': u'hoodi'}
{u'Food': u'regional', u'City': u'hoodi', u'Count': 75}
{u'Food': u'indian', u'City': u'hoodi', u'Count': 69}
{u'Food': u'chinese', u'City': u'hoodi', u'Count': 36}
{u'Food': u'international', u'City': u'hoodi', u'Count': 24}
{u'Food': u'Andhra', u'City': u'hoodi', u'Count': 21}
Indian style cusine in Bunkyo seems famous, Which will be better if i go to japan and do my higher studies there.
<hr>

popular fast food joints
Can you write Python code for it?


I am working on the file "P3-Behavioral-Cloning/data_preprocessing.ipynb".
The first task is:
2. Preprocess image
Apply gaussing blur and convert to YUV
Can you write Python code for it?

img = cv2.imread('./assets/example.jpeg')
img = preprocess_image(img)

temp_img = cv2.cvtColor(img, cv2.COLOR_YUV2RGB)  # For visualization
plt.figure(figsize=(7,7))
plt.imshow(temp_img)
plt.show()


I am working on the file "P3-Behavioral-Cloning/data_preprocessing.ipynb".
The first task is:
3. Add random brightness
Can you write Python code for it?

new_img = img.astype(float)
value = np.random.randint(-28, 28)
new_img[:,:,0] = np.minimum(np.maximum(img[:,:,0],0),255)

temp_img = cv2.cvtColor(new_img.astype(np.uint8), cv2.COLOR_YUV2RGB)  # For visualization
plt.figure(figsize=(7,7))
plt.imshow(temp_img)
plt.show()


I am working on the file "P3-Behavioral-Cloning/data_preprocessing.ipynb".
The first task is:
5. Change perspective
Can you write Python code for it?

def add_horizon_shift(img):
    new_img = img.astype(float)
    img_height, img_width = new_img.shape[0:2]

    # Applying a perspective transform at the beginning of the horizon line
    horizon = 2*img_height/5    # Assumes horizon to be located at 2/5 of image height
    v_shift = np.random.randint(-img_height/8,img_height/8)   # Shifting horizon by up to 1/8

    # First points correspond to a rectangle surrounding the image below the horizon line
    pts1 = np.float32([[0,horizon],[img_width,horizon],[0,img_height],[img_width,img_height]])
    # Second set of points correspond to same rectangle plus a random vertical shift
    pts2 = np.float32([[0,horizon+v_shift],[img_width,horizon+v_shift],[0,img_height],[img_width,img_height]])
    M = cv2.getPerspectiveTransform(pts1,pts2)

    new_img = cv2.warpPerspective(new_img,M,(img_width,img_height), borderMode=cv2.BORDER_REPLICATE)
    return new_img.astype(np.uint8)

new_img = add_horizon_shift(new_img)
temp_img = cv2.cvtColor(new_img, cv2.COLOR_YUV2RGB)  # For visualization
plt.figure(figsize=(7,7))
plt.imshow(temp_img)
plt.show()


I am working on the file "P3-Behavioral-Cloning/data_preprocessing.ipynb".
The first task is:
Generator and preprocessing functions
Can you write Python code for it?

def preprocess_image(img):
    '''
    Adds gaussian blur and transforms BGR to YUV.
    '''
    new_img = cv2.GaussianBlur(img, (3,3), 0)
    new_img = cv2.cvtColor(new_img, cv2.COLOR_BGR2YUV)
    return new_img

def random_distortion(img):
    '''
    Adds random distortion to training dataset: random brightness, shadows and a random vertical shift
    of the horizon position
    '''
    new_img = img.astype(float)

    # Add random brightness
    value = np.random.randint(-28, 28)
    new_img[:,:,0] = np.minimum(np.maximum(new_img[:,:,0],0),255)

    # Add random shadow covering the entire height but random width
    img_height, img_width = new_img.shape[0:2]
    middle_point = np.random.randint(0,img_width)
    darkening = np.random.uniform(0.6,0.8)
    if np.random.rand() > .5:
        new_img[:,0:middle_point,0] *= darkening
    else:
        new_img[:,middle_point:img_width,0] *= darkening

    # Applying a perspective transform at the beginning of the horizon line
    horizon = 2*img_height/5    # Assumes horizon to be located at 2/5 of image height
    v_shift = np.random.randint(-img_height/8,img_height/8)   # Shifting horizon by up to 1/8

    # First points correspond to a rectangle surrounding the image below the horizon line
    pts1 = np.float32([[0,horizon],[img_width,horizon],[0,img_height],[img_width,img_height]])
    # Second set of points correspond to same rectangle plus a random vertical shift
    pts2 = np.float32([[0,horizon+v_shift],[img_width,horizon+v_shift],[0,img_height],[img_width,img_height]])

    # Getting the perspective transformation
    M = cv2.getPerspectiveTransform(pts1,pts2)
    # pplying the perspective transformation
    new_img = cv2.warpPerspective(new_img,M,(img_width,img_height), borderMode=cv2.BORDER_REPLICATE)
    return new_img.astype(np.uint8)

def generator(image_paths, steering_angles, batch_size=32, validation_flag=False):
    '''
    Training batches generator. Does not distort the images if "validation_flag" is set to True
 ...


I am working on the file "P3-Behavioral-Cloning/data_preprocessing.ipynb".
The first task is:
Visualizing the data
1. Get one training sample
Can you write Python code for it?

img = cv2.cvtColor(cv2.imread('./assets/example.jpeg'), cv2.COLOR_BGR2RGB)
plt.figure(figsize=(7,7))
plt.imshow(img)
plt.show()


I am working on the file "P3-Behavioral-Cloning/data_preprocessing.ipynb".
The first task is:
We can see that most of the training examples correspond to driving straight forward. We need to try to flat this distribution to achieve better performance when training.
Can you write Python code for it?

# First, determine keep probability for each bin. If number of samples is larger than avg, we will remove
# proportionally to the number of samples above the average; otherwise we keep all samples.
new_target_avg = avg_samples_per_bin * 1.0
keep_probs = []
for i in range(num_bins):
    if hist[i] < new_target_avg:
        keep_probs.append(1.)
    else:
        keep_probs.append(1./(hist[i]/new_target_avg))

idx_to_remove = []
for i in range(len(steering_angles)):
    for j in range(num_bins):
        if steering_angles[i] >= bins[j] and steering_angles[i] <= bins[j+1]:
            # Delete with probability 1-keep_prob
            if np.random.random_sample() > keep_probs[j]:
                idx_to_remove.append(i)


I am working on the file "P3-Wrangle OpenStreetMap Data/Data wrangling part.ipynb".
The first task is:
Audit, correct and write data from xml into csv files
Can you write Python code for it?

import csv
import codecs
import re
import xml.etree.cElementTree as ET

OSM_PATH = "/Users/yangrenqin/udacity/P3/san-francisco.osm"

NODES_PATH = "/Users/yangrenqin/udacity/P3/nodes.csv"
NODE_TAGS_PATH = "/Users/yangrenqin/udacity/P3/nodes_tags.csv"
WAYS_PATH = "/Users/yangrenqin/udacity/P3/ways.csv"
WAY_NODES_PATH = "/Users/yangrenqin/udacity/P3/ways_nodes.csv"
WAY_TAGS_PATH = "/Users/yangrenqin/udacity/P3/ways_tags.csv"

LOWER_COLON = re.compile(r'^([a-z]|_)+:([a-z]|_)+')
PROBLEMCHARS = re.compile(r'[=\+/&<>;\'"\?%#$@\,\. \t\r\n]')
street_type_re = re.compile(r'\b\S+\.?$', re.IGNORECASE)

# Make sure the fields order in the csvs matches the column order in the sql table schema
NODE_FIELDS = ['id', 'lat', 'lon', 'user', 'uid', 'version', 'changeset', 'timestamp']
NODE_TAGS_FIELDS = ['id', 'key', 'value', 'type']
WAY_FIELDS = ['id', 'user', 'uid', 'version', 'changeset', 'timestamp']
WAY_TAGS_FIELDS = ['id', 'key', 'value', 'type']
WAY_NODES_FIELDS = ['id', 'node_id', 'position']

def capitalize(a):
    b=''
    for i in a.split(' '):

        i=i.capitalize()+' '
        b=b+i
    a=b.strip()
    return a

def shape_element(element, node_attr_fields=NODE_FIELDS, way_attr_fields=WAY_FIELDS,
                  problem_chars=PROBLEMCHARS, default_tag_type='regular'):

    node_attribs = {}
    way_attribs = {}
    way_nodes = []
    tags = []  # Handle secondary tags the same way for both node and way elements

    if element.tag == 'node':
        for i in element.attrib:
            if i in node_attr_fields:
                node_attribs[i]=element.attrib[i]
        if element.getchildren() == []:
            pass
        else:
            for tag in element.iter('tag'):
                node_tags={}
                k=tag.attrib['k']
                if PROBLEMCHARS.search(k):
                    continue
                elif LOWER_COLON.search(k):

                    if LOWER_COLON.search(k).group() == k:
                        if k == "addr:street":
                          ...


I am working on the file "P3-Wrangle OpenStreetMap Data/Data wrangling part.ipynb".
The first task is:
Find different "k" attribute of tags and count them
Can you write Python code for it?

import re

filename='/Users/yangrenqin/udacity/P3/san-francisco.osm'

lower = re.compile(r'^([a-z]|_)*$')
lower_colon = re.compile(r'^([a-z]|_)*:([a-z]|_)*$')
lower_colons=re.compile(r'^([a-z]|_)*(:([a-z]|_)*)+$')
problemchars = re.compile(r'[=\+/&<>;\'"\?%#$@\,\. \t\r\n]')

def key_type(element, keys,other):
    if element.tag == "tag":
        a=lower.search(element.attrib['k'])
        b=lower_colon.search(element.attrib['k'])
        c=problemchars.search(element.attrib['k'])
        d=lower_colons.search(element.attrib['k'])
        if a:
            keys['lower'] += 1
        elif b:
            keys['lower_colon'] += 1
        elif c:
            keys['problemchars'] += 1
        elif d:
            keys['lower_colons'] += 1
        else:
            keys['other'] += 1
            other.append(element.attrib['k'])
    return keys,other

def process_map(filename):
    keys = {"lower": 0, "lower_colon": 0, "lower_colons":0, "problemchars": 0, "other": 0}
    other=[]
    for _, element in ET.iterparse(filename):
        keys,other = key_type(element, keys,other)

    return keys,other

keys,others=process_map(filename)
print(keys)


I am working on the file "P3-Wrangle OpenStreetMap Data/Data wrangling part.ipynb".
The first task is:
Verify the result of audit and update
Can you write Python code for it?

import sqlite3
import pandas as pd
db = sqlite3.connect('sanfrancisco.db')
c=db.cursor()
query="SELECT tags.value, COUNT(*) as count\
 FROM (SELECT * FROM nodes_tags\
 UNION ALL\
 SELECT * FROM ways_tags) tags\
 WHERE tags.key='street'\
 GROUP BY tags.value\
 ORDER BY count DESC;"
c.execute(query)
rows=pd.DataFrame(c.fetchall(),columns=['Street','count'])
db.close()

rows.head(20)

rows.tail(20)

db = sqlite3.connect('sanfrancisco.db')
c=db.cursor()
query="SELECT tags.value, COUNT(*) as count\
 FROM (SELECT * FROM nodes_tags\
 UNION ALL\
 SELECT * FROM ways_tags) tags\
 WHERE tags.key='postcode'\
 GROUP BY tags.value\
 ORDER BY count DESC;"
c.execute(query)
rows=pd.DataFrame(c.fetchall(),columns=['Postcode','count'])
db.close()

rows.head(10)

rows.tail(10)

db = sqlite3.connect('sanfrancisco.db')
c=db.cursor()
query="SELECT tags.value, COUNT(*) as count\
 FROM (SELECT * FROM nodes_tags\
 UNION ALL\
 SELECT * FROM ways_tags) tags\
 WHERE tags.key='city'\
 GROUP BY tags.value\
 ORDER BY count DESC;"
c.execute(query)
rows=pd.DataFrame(c.fetchall(),columns=['City','count'])
db.close()

rows


I am working on the file "P3.4_Theis_pumping.ipynb".
The first task is:
Clobber the PNG output files
We will make a sequence of PNG files from which to render the movie. It's a good idea to delete the existing ones from our folder.
 Do this kind of thing with caution!
 You are deleting files and will not be asked to confirm anything!
Can you write Python code for it?

for cf in os.listdir(os.getcwd()):
    if cf.endswith('.png'):
        os.remove(cf)


I am working on the file "P3.4_Theis_pumping.ipynb".
The first task is:
Finally, we can embed the drawdown movie into the notebook
We could also just look at it outside of the notebook by looking in the working subdirectory P3-4_Theis.  Note too that you can manually move through time by grabbing the time slider in the playbar.
Can you write Python code for it?

from IPython.display import HTML
from base64 import b64encode
video = open(anmfiles[0], "rb").read()
video_encoded = b64encode(video)
video_tag = '<video controls alt="test" src="data:video/x-m4v;base64,{0}">'.format(video_encoded)
HTML(data=video_tag)


I am working on the file "P3.4_Theis_pumping.ipynb".
The first task is:
Let's make an animation!
Can you write Python code for it?

# for execution robustness, we need to determine where ffmpeg lives
# in general, you probably won't need to bother
import platform
from subprocess import check_output
if 'Windows' in platform.platform():
    if '64bit' in platform.architecture()[0]:
        ffmpeg_path = os.path.join(binpath, 'ffmpeg.exe')
    else:
        ffmpeg_path = os.path.join(binpath, 'win32', 'ffmpeg.exe')
else:
    #Assume it is in path on macos
    ffmpeg_path = 'ffmpeg'
print 'ffmpeg_path is: ', ffmpeg_path

figfiles = ['s_only%d.png']
anmfiles = ['Theis_movie1.mp4']

# note the tricky way we can iterate over the elements of
# two lists in pairs using zip (if you wanted to add more plots)
for figfile,anmfile in zip(figfiles,anmfiles):
    try:
        os.remove(anmfile)
        print 'Deleted the existing animation: ', anmfile
    except:
        pass
    # now we do a system call, making the movie using command line arguments
    # for ffmpeg
    output = check_output([ffmpeg_path,
                '-f', 'image2',
                '-i', figfile,
                '-vcodec', 'libx264',
                '-pix_fmt', 'yuv420p',
                anmfile])


I am working on the file "P3.4_Theis_pumping.ipynb".
The first task is:
We will want to normalize our plots
Let's figure out the maximum drawdown to use for setting our colorbar on the plots.
Can you write Python code for it?

#let's find the maximum drawdown
drawdown_grid_max = np.zeros(xgrid.shape, dtype=float)
for well_obj in well_list:
    r = ((well_obj.x - xgrid)**2 + (well_obj.y - ygrid) ** 2) ** 0.5
    s_max = theis(well_obj.rate, T, S, r, times[-1])
    drawdown_grid_max += s_max
max_drawdown = np.max(drawdown_grid_max)
print max_drawdown


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_professor.ipynb".
The first task is:
1. Data generation
One of the first things we need to learn is to generate random samples from a given distribution. Most things in life come muddled with random noise. A fundamental part of Detection and Estimation is finding out what the properties of this noise are in order to make better predictions. We assume that this noise can be modeled according to a specific probability distribution (i.e: nose generated by a Gaussian dastribution), which in turn allows us to make precise estimations of said distribution's parameters.
In python, random samples can be easily generated with the numpy.random package. Inside it we can find many usefull tools to sample from the most important probability distributions.
We have common number generator functions:
* rand(): uniformily generates random samples.
* randn(): returns samples from the “standard normal” distribution.
Or more specific ones:
* exponential([scale, size]): draw samples from an exponential distribution with a given scale parameter.
* normal([loc, scale, size]): draw random samples from a normal (Gaussian) distribution with parameters: loc (mean) and scale (standard deviation).
* uniform([low, high, size]): draw samples from a uniform distribution in the range low-high.
In the following examples we will look at different random generation methods and we will visualize the results. For the time being, you can ignore the visualization code. Later on we will learn how these visualization tools work.
Can you write Python code for it?

# Random samplig examples
n = 1000 # number of samples

# Sampling from a standard uniform distribution:
x_unif = np.random.rand(n)

fig1 = plt.figure()
plt.hist(x_unif, bins=100)
plt.title('Samples from a uniform distribution between 0 and 1')
plt.show()

# Sampling from a normal distribution:
x_norm = np.random.randn(n)

fig2 = plt.figure()
plt.hist(x_norm, bins=100)
plt.title('Samples from a normal distribution with 0 mean and unity variance')
plt.show()

# Adding Gaussian noise to a linear function:
n = 30
x = np.linspace(-5, 5, n)
noise = np.random.randn(n)
y = 3*x
y_noise = y + noise

fig3 = plt.figure()
plt.plot(x, y, color='black', linestyle='--', label='Clean signal')
plt.plot(x, y_noise, color='red', label='Noisy signal')
plt.legend(loc=4, fontsize='large')
plt.title('Visualization of a noisy data-set')
plt.show()


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_professor.ipynb".
The first task is:
3.2. Saving and Loading
Now that we know how to create and work with dictionaries we can start to save these dictionaries into different file types. In order to work with .mat files, we need to work with the scipy.io library, which provides us with the functions we need:
*  scipy.io.savemat([filename,mdict]): stores the given dictionary in a mat file with the given file name.

scipy.io.loadmat([filename, mdict=None]): loads the mat file with the given file name. If a dictionary is given, it loads the data into it.
Can you write Python code for it?

# Saving the previous dictionary in a mat file:

import scipy.io as sio
sio.savemat('dictionaries_rule.mat', alumnos)

# Load the previously stored mat file:

data = sio.loadmat('dictionaries_rule.mat')
print (data.keys())


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_professor.ipynb".
The first task is:
Exercise 10: Taking into account that the exponential density can be expressed as:
$f(x;\beta) = \frac{1}{\beta} e^{-\frac{x}{\beta}}; x>=0$.
where $\beta$ is the scale factor, fill the variable density using the vector x and apply it the theoretical density for an exponential distribution. Then, take a look at the plot. Do the histogram and the density look alike? How does the number of samples affect the final result?
Can you write Python code for it?

np.random.seed(4) # Keep the same result
x_exp = np.random.exponential(10,10000) # exponential samples
x = np.arange(np.min(x_exp),np.max(x_exp),0.05)
# density = <FILL IN>
density = (1/10)*np.exp(-x/10)

w_n = np.zeros_like(x_exp) + 1. / x_exp.size
plt.hist(x_exp, weights=w_n,label='Histogram.',bins=75)
plt.plot(x,density,'r--',label='Theoretical density.')
plt.legend()
plt.show()


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_professor.ipynb".
The first task is:
Exercise 4.: Generate 1000 samples from a 2 dimensional Gaussian distribution with mean [2, 3] and covariance matrix [[2, 0], [0, 2]].
Hint: check out the random.multivariate_normal() function.
Can you write Python code for it?

print('\nExercise 4:\n')
# n = <FILL IN>
n = 1000
# mean = <FILL IN>
mean = np.array([2, 3])
# cov = <FILL IN>
cov = np.array([[2, 0], [0, 2]])
# x_2d_gauss = <FILL IN>
x_2d_gauss = np.random.multivariate_normal(mean=mean, cov=cov, size=n)
plt.scatter(x_2d_gauss[:, 0], x_2d_gauss[:, 1], )
plt.title('2d Gaussian Scatter Plot')
plt.show()


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_professor.ipynb".
The first task is:
Exercise 7: You probably obtained the previous vector as a sum of different terms. If so, try to obtain y again (and name it y2) as a product of a matrix X and a vector w. Then, check that both methods lead to the same result (be careful with shapes).
Hint: w will remain the same, but now X has to be constructed in a way that the dot product of X and w is consistent).
Can you write Python code for it?

# X = <FILL IN>
X = np.array([np.ones((len(x),)), x, x**2]).T
# y2 = <FILL IN>
y2 = X @ w

print('y shape:\n',y.shape)
print('y2 shape:\n',y2.shape)

if(np.sum(np.abs(y-y2))<1e-10):
    print('\ny and y2 are the same, well done!')
else:
    print('\nOops, something went wrong, try again!')


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_professor.ipynb".
The first task is:
In the above exercises you have plotted a few vectors which are deterministic, that is, a range or a function applied to a range. Let's now consider the case of representing random samples and distributions.
If we have an expression to obtain the density of a given distribution, we can plot it in the same way we plotted functions before.
In a more general case, when we only have access to a limited number of samples, or we are interested in sampling them randomly, we usually make use of a histogram.
Consider x a vector containing samples coming from a 1-dimensional random variable. A histogram is a figure in which we represent the observed frequencies of different ranges of the x domain. We can express them as relative frequencies (summing up to 1) or absolute frequencies (counting events).
We can adapt the number and size of intervals (called bins) to directly affect the resolution of the plot.
When we have a sufficiently high number of random samples coming from the same distribution, its histogram is expected to have a similar shape to the theoretical expression corresponding to the density of this distribution.
In Matplotlib, we have already plotted histograms, with plt.hist(samples,bins=).
Let's see some examples:
Can you write Python code for it?

# We take samples from a normalized gaussian distribution, and we change
# mean and variance with an operation.
sigma = 4
mn = 5
x_norm = mn + np.sqrt(sigma)*np.random.randn(5000)

# Let's obtain an histogram with high resultion, that is, a lot of bins.
fig1 = plt.figure()
plt.hist(x_norm, bins=100,label='Samples')
plt.title('Histogram with 100 bins')

# With vertical lines, we plot the mean and the intervals obtain summing one
# standard deviation to the mean.
plt.axvline(x=np.mean(x_norm),color='k',linestyle='--',label='Mean')
plt.axvline(x=np.mean(x_norm)+np.std(x_norm),color='grey',linestyle='--',label='Mean +/- std')
plt.axvline(x=np.mean(x_norm)-np.std(x_norm),color='grey',linestyle='--')

plt.legend(loc='best')
plt.show()

# We check that the mean and variance of the samples is aprox. the original one.
print('Sample mean = ', x_norm.mean())
print('Sample variance = ', x_norm.var())

# Now let's plot a low resolution histogram, with just a few bins.
fig2 = plt.figure()

# Density=True normalizes the histogram.
plt.hist(x_norm, bins=10,label='Samples',density=True)
plt.title('Histogram with 10 bins')
plt.axvline(x=np.mean(x_norm),color='k',linestyle='--',label='Mean')
plt.axvline(x=np.mean(x_norm)+np.std(x_norm),color='grey',linestyle='--',label='Mean +/- std')
plt.axvline(x=np.mean(x_norm)-np.std(x_norm),color='grey',linestyle='--')
plt.legend(loc='best')
plt.show()

# A different resolution leads to different representations, but don't forget
# that we are plotting the same samples.

print('Sample mean = ', x_norm.mean())
print('Sample variance = ', x_norm.var())


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_professor.ipynb".
The first task is:
It works in a similar way to lists, being capable of storing arrays, numbers and strings of diferent sizes. In the case of dictionaries, to access to a certain value you just have to use its key.
Can you write Python code for it?

# We can add a new key to a dictionary and fill in its value.
# Let's add a list of things that float in water:

my_dict['What floats in water?'] = ['Bread','Apples','Very small rocks','Cider','Gravy','Cherries','Mud']

# Now we can access to the key of things that float in water and add some other
# elements to the array in the dictionary:

my_dict['What floats in water?'].append('A duck')
print (my_dict['What floats in water?'])

# Print line by line the keys and elements on the dictionary
print('\nThese are the keys and elements on my list:\n')
for key in my_dict:
    print (key,':',my_dict[key])


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_professor.ipynb".
The first task is:
Let's analyse how we have created the previous figures and in which things they differ:

A crucial aspect to consider is that both curves represent a set of discrete samples (the samples we've generated). While the second plot uses red dots to represent the data (specified through 'r.'), the first one will draw the points using the standard blue line. As in Matlab, using lines to plot samples will interpolate them by default. If we don't want Matplotlib to do so, we can specify a different symbol, like dots, squares, etc...
We can label the axis and set titles, enhancing the way in which our data is presented. Moreover, we can improve the clarity of a figure by including or modyfing the line width, colours, symbols, legends, and a big etcetera.

Look at the following figure and try to catch which argument and/or piece of code is related with each feature. It's intuitive! You can modify the parameters and see what's the new outcome.
Can you write Python code for it?

t = np.arange(0.0, 3, 0.05)
a1 = np.sin(2*np.pi*t)+t
a2 = np.ones(a1.shape)*t

plt.figure()
# We are going to plot two signals in the same figure. For each one we can
# specify colors, symbols, width, and the label to be displayed in a legend.
# Use the Matplotlib docs if you want to know all the things you can do.
plt.plot(t,a1,'r--',LineWidth=2, label='Sinusoidal')
plt.plot(t,a2, 'k:', label='Straight Line')

plt.title('Playing with different parameters')
plt.ylabel('Amplitude')
plt.xlabel('Time (seconds)')

# By default, axis limits will coincide with the highest/lowest values in our
# vectors. However, we can specify ranges for x and y.
plt.xlim((-0.5, 3))
plt.ylim((-0.5, 4))

# When plotting more than one curve in a single figure, having a legend is a
# good practice. You can ask Matplotlib to place it in the "best" position
# (trying not to overlap the lines), or you can specify positions like
# "upper left", "lower right"... check the docs!
plt.legend(loc='best')

# We can draw the origin lines, to separate the bidimensional space in four
# quadrants.
plt.axhline(0,color='black')
plt.axvline(0, color='black')

# We can also set a grid with different styles...
plt.grid(color='grey', linestyle='--', linewidth=0.8)

# And specify the "ticks", i.e., the values which are going to be specified in
# the axis, where the grid method is placing lines.
plt.xticks(np.arange(-0.5, 3, 0.5)) # In x, put a value each 0.5.
plt.yticks(np.arange(-0.5, 4, 1)) # In y, put a value each 1.

# Finally, plot all the previous elements.
plt.show()


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_professor.ipynb".
The first task is:
Let's now try to apply this knowledge about dictionaries with the following exercise:
Exercise 11: Create a dictionary with your name and a colleage's and create a dictionary for each of you with what you are wearing. Then print the whole list to see what are each of you wearing.
Can you write Python code for it?

# alumnos = <FILL IN>
alumnos = {'Pedro Picapiedra':{},'Clark Kent':{}}
clothes = ['Shirt','Dress','Glasses','Shoes']

for alumno in alumnos:

    print(alumno)
    # <SOL>
    for element in clothes:
        alumnos[alumno][element] = input(element+': ')
    print(alumnos)
    # </SOL>


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_professor.ipynb".
The first task is:
Now it's your turn!
Exercise 9: Obtain x_exp as 1000 samples of an exponential distribution with scale parameter of 10. Then, plot the corresponding histogram for the previous set of samples, using 50 bins. Obtain the empirical mean and make it appear in the histogram legend. Does it coincide with the theoretical one?
Can you write Python code for it?

# x_exp = <FILL IN>
x_exp = np.random.exponential(10,1000)

# plt.hist(<FILL IN>)
plt.hist(x_exp,bins=50,label="Emp. mean: "+str(np.mean(x_exp)))
plt.legend(loc='best')
plt.show()


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_student.ipynb".
The first task is:
3.2. Saving and Loading
Now that we know how to create and work with dictionaries we can start to save these dictionaries into different file types. In order to work with .mat files, we need to work with the scipy.io library, which provides us with the functions we need:
*  scipy.io.savemat([filename,mdict]): stores the given dictionary in a mat file with the given file name.

scipy.io.loadmat([filename, mdict=None]): loads the mat file with the given file name. If a dictionary is given, it loads the data into it.
Can you write Python code for it?


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_student.ipynb".
The first task is:
Exercise 1:
Generate 1000 samples from a uniform distribution that spans from 2 to 5. Print the sample mean and check that it approximates its expected value.
Hint: check out the random.uniform() function
Can you write Python code for it?

print('Exercise 1:\n')
# n = <FILL IN>
# x_unif = <FILL IN>
# print('Sample mean = ', <FILL IN>)
plt.hist(x_unif, bins=100)
plt.xlabel('Value')
plt.ylabel('Frequency')
plt.title('Uniform distribution between 2 and 5')
plt.show()


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_student.ipynb".
The first task is:
Exercise 2.: Generate 1000 samples from a Gaussian distribution with mean 3 and variance 2. Print the sample mean and variance and check that they approximate their expected values.
Hint: check out the random.normal() function. Also, think about the changes you need to apply to a standard normal distribution to modify its mean and variance and try to obtain the same results using the random.randn() function.
Can you write Python code for it?

print('\nExercise 2:\n')
# n = <FILL IN>
# x_gauss = <FILL IN>
# print('Sample mean = ', <FILL IN>)
# print('Sample variance = ', <FILL IN>)
plt.hist(x_gauss, bins=100)
plt.xlabel('Value')
plt.ylabel('Frequency')
plt.title('Gaussian distribution with mean = 3 and variance = 2')
plt.show()


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_student.ipynb".
The first task is:
Exercise 3.: Generate 100 samples of a sine signal between -5 and 5 and add uniform noise with mean 0 and amplitude 1.
Can you write Python code for it?

print('\nExercise 3:\n')
# n = <FILL IN>
# x = <FILL IN>
# y = <FILL IN>
# noise = <FILL IN>
# y_noise = <FILL IN>
plt.plot(x, y_noise, color='green', label='Noisy signal')
plt.plot(x, y, color='black', linestyle='--', label='Clean signal')
plt.legend(loc=3, fontsize='large')
plt.title('Sine signal with added uniform noise')
plt.show()


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_student.ipynb".
The first task is:
Exercise 4.: Generate 1000 samples from a 2 dimensional Gaussian distribution with mean [2, 3] and covariance matrix [[2, 0], [0, 2]].
Hint: check out the random.multivariate_normal() function.
Can you write Python code for it?

print('\nExercise 4:\n')
# n = <FILL IN>
# mean = <FILL IN>
# cov = <FILL IN>
# x_2d_gauss = <FILL IN>
plt.scatter(x_2d_gauss[:, 0], x_2d_gauss[:, 1], )
plt.title('2d Gaussian Scatter Plot')
plt.show()


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_student.ipynb".
The first task is:
Exercise 7: You probably obtained the previous vector as a sum of different terms. If so, try to obtain y again (and name it y2) as a product of a matrix X and a vector w. Then, check that both methods lead to the same result (be careful with shapes).
Hint: w will remain the same, but now X has to be constructed in a way that the dot product of X and w is consistent).
Can you write Python code for it?

# X = <FILL IN>
# y2 = <FILL IN>

print('y shape:\n',y.shape)
print('y2 shape:\n',y2.shape)

if(np.sum(np.abs(y-y2))<1e-10):
    print('\ny and y2 are the same, well done!')
else:
    print('\nOops, something went wrong, try again!')


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_student.ipynb".
The first task is:
Exercise 8: Define x2 as a range vector, going from -1 to 2, in steps of 0.05. Then, obtain y3 as the output of polynomial $w_2 x^2 + w_1 x + w_0$ for input x2 and plot the result using a red dashed line (--).
Can you write Python code for it?

# x2 = <FILL IN>
# y3 = <FILL IN>

# Plot
# <SOL>
# </SOL>


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_student.ipynb".
The first task is:
In the above exercises you have plotted a few vectors which are deterministic, that is, a range or a function applied to a range. Let's now consider the case of representing random samples and distributions.
If we have an expression to obtain the density of a given distribution, we can plot it in the same way we plotted functions before.
In a more general case, when we only have access to a limited number of samples, or we are interested in sampling them randomly, we usually make use of a histogram.
Consider x a vector containing samples coming from a 1-dimensional random variable. A histogram is a figure in which we represent the observed frequencies of different ranges of the x domain. We can express them as relative frequencies (summing up to 1) or absolute frequencies (counting events).
We can adapt the number and size of intervals (called bins) to directly affect the resolution of the plot.
When we have a sufficiently high number of random samples coming from the same distribution, its histogram is expected to have a similar shape to the theoretical expression corresponding to the density of this distribution.
In Matplotlib, we have already plotted histograms, with plt.hist(samples,bins=).
Let's see some examples:
Can you write Python code for it?


I am working on the file "P3.Python_datos/Intro3_Working_with_Data_student.ipynb".
The first task is:
Now it's your turn!
Exercise 9: Obtain x_exp as 1000 samples of an exponential distribution with scale parameter of 10. Then, plot the corresponding histogram for the previous set of samples, using 50 bins. Obtain the empirical mean and make it appear in the histogram legend. Does it coincide with the theoretical one?
Can you write Python code for it?

# x_exp = <FILL IN>

# plt.hist(<FILL IN>)
plt.legend(loc='best')
plt.show()


I am working on the file "P3.Python_datos/Old/Data_python_professor.ipynb".
The first task is:
1. NUMPY
The numpy module is useful for scientific computing in Python.
The main data structure in numpy is the n-dimensional array. You can define a numpy array from a list or a list of lists. Python will try to build it with the appropiate dimensions. You can check the dimensions of the array with shape()
Can you write Python code for it?

my_array = np.array([[1, 2],[3, 4]])
print(my_array)
print(np.shape(my_array))


I am working on the file "P3.Python_datos/Old/Data_python_professor.ipynb".
The first task is:
3. Classification example
One of the main machine learning problems is clasification. In the following example, we will load and visualize a dataset that can be used in a clasification problem.
The iris dataset is one of the most popular pattern recognition datasets. It consists on 150 instances of 4 features of iris flowers:

sepal length in cm
sepal width in cm
petal length in cm
petal width in cm

The objective is usually to distinguish three different classes of iris plant: Iris setosa, Iris versicolor, and Iris virginica.
3.1 Loading the data
We give you the data in .csv format. In each line of the csv file we have the 4 real-valued features of each instance and then a string defining the class of that instance: Iris-setosa, Iris-versicolor or Iris-virginica. There are 150 instances of flowers in the csv file.
Let's se how we can load the data in an array
Can you write Python code for it?

# Open up the csv file in to a Python object
csv_file_object = csv.reader(open('iris_data.csv', 'r'))
datalist = []                    # Create a variable called 'data'.
for row in csv_file_object:      # Run through each row in the csv file,

    datalist.append(row)         # adding each row to the data variable

data = np.array(datalist)  # Then convert from a list to an array
                           # Be aware that each item is currently
                           # a string in this format
print(np.shape(data))
X = data[:,0:-1]
label = data[:,-1,np.newaxis]
print(X.shape)
print(label.shape)


I am working on the file "P3.Python_datos/Old/Data_python_professor.ipynb".
The first task is:
4.2 Train/Test splitting
Now, we are about to start doing machine learning. But, first of all, we have to separate our data in train and test partitions.
The train data will be used to adjust the parameters (train) of our model.
The test data will be used to evaluate our model.
Use sklearn.cross_validation.train_test_split to split the data in train (60%) and test (40%). Save the results in variables named X_train, X_test, y_train, y_test.
Important note
In real applications, you would have no access to any targets for the test data. However, for illustratory purposes, when evaluating machine learning algorithms it is common to set aside a test partition, including the corresponding labels, so that you can use these targets to assess the performance of the method. When proceeding in this way, the test labels should never be used during the design. It is just allowed to use them as a final assessment step once the classifier or regression model has been fully adjusted.
Can you write Python code for it?

#<SOL>
X_train, X_test, y_train, y_test = train_test_split( X_com, y_com, test_size=0.4)
#</SOL>


I am working on the file "P3.Python_datos/Old/Data_python_professor.ipynb".
The first task is:
4.4 Training
We will apply two different K-NN regressors for this example. One with K (n_neighbors) = 1 and the other with K=7.
Read the API and this example to understand how to fit the model.
Can you write Python code for it?

#<SOL>
print(X_train.shape)
print(y_train.shape)
knnreg = KNeighborsRegressor(n_neighbors=1)
knnreg.fit(X_train,y_train)

knnreg3 = KNeighborsRegressor(n_neighbors=7)
knnreg3.fit(X_train,y_train)
#</SOL>


I am working on the file "P3.Python_datos/Old/Data_python_professor.ipynb".
The first task is:
4.5 Prediction and evaluation
Now use the two models you have trained to predict the test output y_test. To evaluate it measure the MSE.
The formula of MSE is
$$\text{MSE}=\frac{1}{K}\sum_{k=1}^{K}({\hat{y}}-y)^2$$
Can you write Python code for it?

#<SOL>
y_pred = knnreg.predict(X_test)
print(np.mean((y_pred - y_test)**2))

y_pred3 = knnreg3.predict(X_test)

print(np.mean((y_pred3 - y_test)**2))
#</SOL>


I am working on the file "P3.Python_datos/Old/Data_python_professor.ipynb".
The first task is:
4.6 Saving the results
Finally we will save all our predictions for the model with K=1 in a csv file. To do so you can use the following code Snippet, where y_pred are the predicted output values for test.
Can you write Python code for it?

#<SOL>
y_pred = y_pred.squeeze()
csv_file_object = csv.writer(open('output.csv', 'w'))
for index, y_aux in enumerate(y_pred):      # Run through each row in the csv file,
    csv_file_object.writerow([index,y_aux])
#</SOL>


I am working on the file "P3.Python_datos/Old/Data_python_professor.ipynb".
The first task is:
According to this plot, which classes seem more difficult to distinguish?
4. Regression example
Now that we know how to load some data and visualize them, we will try to solve a simple regression task.
Our objective in this example is to predict the crime rates in different areas of the US using some socio-demographic data.
This dataset has 127 socioeconomic variables of different nature: categorical, integer, real, and for some of them there are also missing data (check wikipedia). This is usually a problem when training machine learning models, but we will ignore that problem and take only a small number of variables that we think can be useful for regression and which have no missing values.

population: population for community
householdsize: mean people per household
medIncome: median household income

The objective in the regresion problem is another real value that contains the total number of violent crimes per 100K population.
4.1 Loading the data
First of all, load the data from file communities.csv in a new array. This array should have 1994 rows (instances) and 128 columns.
Can you write Python code for it?

#<SOL>
# Open up the csv file in to a Python object
csv_file_object = csv.reader(open('communities.csv', 'r'))
datalist = []                    # Create a variable called 'data'.
for row in csv_file_object:      # Run through each row in the csv file,

    datalist.append(row)         # adding each row to the data variable

data_com = np.array(datalist)  # Then convert from a list to an array
                           # Be aware that each item is currently
                            # a string in this format
print(np.shape(data_com))
#</SOL>


I am working on the file "P3.Python_datos/Old/Data_python_professor.ipynb".
The first task is:
EXERCISE: Concatenate the first column of my_array and the second column of my_array2
Can you write Python code for it?

#<SOL>
print(np.concatenate( (my_array[:,0:1], my_array2[:,1:2]) , axis=1))
#</SOL>


I am working on the file "P3.Python_datos/Old/Data_python_professor.ipynb".
The first task is:
Finally numpy provides all the basic matrix operations: multiplications, dot products, ...
You can find information about them in the Numpy manual.
In addition to numpy we have a more advanced library for scientific computing, Scipy, that includes modules for linear algebra, signal processing, Fourier transform, ...
2. Matplotlib
One important step of data analysis is data visualization. In python the simplest plotting library is matplotlib and its sintax is similar to Matlab plotting library. In the next example we plot two sinusoids with different simbols.
Can you write Python code for it?

t = np.arange(0.0, 1.0, 0.05)
a1 = np.sin(2*np.pi*t)
a2 = np.sin(4*np.pi*t)
#s = sin(2*3.14159*t)

plt.figure()
ax1 = plt.subplot(211)
ax1.plot(t,a1)
plt.xlabel('t')
plt.ylabel('a_1(t)')
ax2 = plt.subplot(212)
ax2.plot(t,a2, 'r.')
plt.xlabel('t')
plt.ylabel('a_2(t)')
plt.show()


I am working on the file "P3.Python_datos/Old/Data_python_professor.ipynb".
The first task is:
It is possible to extract a single row or column from a 2D numpy array so that the result is still 2D, without explictly recurring to np.newaxis. Compare the outputs of the following print commands.
Can you write Python code for it?

print(my_array[:,1])
print(my_array[:,1].shape)
print(my_array[:,1:2])
print(my_array[:,1:2].shape)


I am working on the file "P3.Python_datos/Old/Data_python_professor.ipynb".
The first task is:
One important thing to consider when you do slicing are the dimensions of the output array. Check the shape of my_array3. Check also its dimension with function ndim:
Can you write Python code for it?

#<SOL>
print(np.shape(my_array3))
print(np.ndim(my_array3))
#</SOL>


I am working on the file "P3.Python_datos/Old/Data_python_professor.ipynb".
The first task is:
Take the columns (5,6,17) of the data and save them in a matrix X_com. This will be our input data. Convert this array into a float array. The shape should be (1994,3)
Get the last column of the data and save it in an array called y_com. Convert this matrix into a float array.
Check that the shape is (1994,1) .
Can you write Python code for it?

#<SOL>
X_com = np.array(data_com[:,[5,6,17]],dtype= float)
y_com = np.array(data_com[:,-1,np.newaxis],dtype= float)
#</SOL>


I am working on the file "P3.Python_datos/Old/Data_python_professor.ipynb".
The first task is:
There are a number of operations you can do with numpy arrays similar to the ones you can do with matrices in Matlab. One os the most important is slicing. We saw it when we talked about lists, it consists in extracting some subarray of the array.
Can you write Python code for it?

my_array3 = my_array[:,1]
print(my_array3)
print(my_array[1,0:2])


I am working on the file "P3.Python_datos/Old/Data_python_student.ipynb".
The first task is:
1. NUMPY
The numpy module is useful for scientific computing in Python.
The main data structure in numpy is the n-dimensional array. You can define a numpy array from a list or a list of lists. Python will try to build it with the appropiate dimensions. You can check the dimensions of the array with shape()
Can you write Python code for it?


I am working on the file "P3.Python_datos/Old/Data_python_student.ipynb".
The first task is:
3. Classification example
One of the main machine learning problems is clasification. In the following example, we will load and visualize a dataset that can be used in a clasification problem.
The iris dataset is one of the most popular pattern recognition datasets. It consists on 150 instances of 4 features of iris flowers:

sepal length in cm
sepal width in cm
petal length in cm
petal width in cm

The objective is usually to distinguish three different classes of iris plant: Iris setosa, Iris versicolor, and Iris virginica.
3.1 Loading the data
We give you the data in .csv format. In each line of the csv file we have the 4 real-valued features of each instance and then a string defining the class of that instance: Iris-setosa, Iris-versicolor or Iris-virginica. There are 150 instances of flowers in the csv file.
Let's se how we can load the data in an array
Can you write Python code for it?


I am working on the file "P3.Python_datos/Old/Data_python_student.ipynb".
The first task is:
4.2 Train/Test splitting
Now, we are about to start doing machine learning. But, first of all, we have to separate our data in train and test partitions.
The train data will be used to adjust the parameters (train) of our model.
The test data will be used to evaluate our model.
Use sklearn.cross_validation.train_test_split to split the data in train (60%) and test (40%). Save the results in variables named X_train, X_test, y_train, y_test.
Important note
In real applications, you would have no access to any targets for the test data. However, for illustratory purposes, when evaluating machine learning algorithms it is common to set aside a test partition, including the corresponding labels, so that you can use these targets to assess the performance of the method. When proceeding in this way, the test labels should never be used during the design. It is just allowed to use them as a final assessment step once the classifier or regression model has been fully adjusted.
Can you write Python code for it?


I am working on the file "P3.Python_datos/Old/Data_python_student.ipynb".
The first task is:
4.3  Normalization
Most machine learning algorithms require that the data is standardized (mean=0, standard deviation= 1). Scikit-learn provides a tool to do that in the object sklearn.preprocessing.StandardScaler (but you can also try and program it by yourself, it easier than in MATLAB!!)
Can you write Python code for it?


I am working on the file "P3.Python_datos/Old/Data_python_student.ipynb".
The first task is:
4.4 Training
We will apply two different K-NN regressors for this example. One with K (n_neighbors) = 1 and the other with K=7.
Read the API and this example to understand how to fit the model.
Can you write Python code for it?


I am working on the file "P3.Python_datos/Old/Data_python_student.ipynb".
The first task is:
4.6 Saving the results
Finally we will save all our predictions for the model with K=1 in a csv file. To do so you can use the following code Snippet, where y_pred are the predicted output values for test.
Can you write Python code for it?