I am working on the file "Day1/Lab_1_Pandas.ipynb".
The first task is:
Ejercicio 4
A pesar de que en el ejercicio anterior hemos visto un resumen de las características de los datos, conviene siempre hacer representaciones gráficas, que nos ofrecen, en general un mejor entendimiento de los datos. Para ello vamos a realizar en este ejercicio los histogramas de las variables: AGE, BMI, BP, S3, y Y
Pregunta: A la vista de las distribuciones, discuta la normalidad de cada una de ellas (media, skwennes, kurtosis)
Can you write Python code for it?

#hist
diabetes['AGE'].plot.hist(x = 'Age',alpha=0.5)

#hist 2
diabetes['BMI'].plot.hist(x = 'Age',alpha=0.5)

#hist 3
diabetes['BP'].plot.hist(x = 'Age',alpha=0.5)

#hist 4
diabetes['S3'].plot.hist(x = 'Age',alpha=0.5)

#hist 5
diabetes['Y'].plot.hist(x = 'Age',alpha=0.5)


I am working on the file "Day1/Lab_1_Pandas.ipynb".
The first task is:
Ejercicio 5
Una vez que hemos analizado cada una de las variables por separado, el siguiente paso en un análisis exploratorio sería el entender las relaciones entre cada una de las variables y la variable respuesta. En un paso posterior analizaremos las relaciones entre cada una de las variables, pero dado que nuestro objetivo es modelar la variable respuesta con las variables explicativas, vamos en primer lugar a analizar estas relaciones.
Para ello vamos a representar un scatter plot con la variable respuesta como variable dependiente y cada una de las variables explicativas como variable independiente. En el caso de la variabel sex, dado que esta es una variable cualitativa binaria, vamos a utilizar un boxplot.
Haga las 10 representaciones y responda a las siguientes preguntas.
Pregunta: ¿Cómo definiría la relación existente entre la variables respuesta, progresión de la diabetes, con respecto a la edad del sujeto?
Pregunta: ¿Existe alguna variación significativa de la progresión de la enfermedad que se pueda atribuir sólo al sexo?
Pregunta ¿Cómo es la relación entre BMI y la progresión de la enfermedad? Si nuestro objetivo es analizar mediante regresión lineal la relación, se le ocurre algún tipo de transformación de la variable BMI para poder explotar la relación lineal.
Pregunta ¿Cómo define la relación entre BP y la progresión de la enfermedad?
Pregunta Una de las variables S presenta una relación opuesta a la de las otras variables, indique cuál de ellas es y explique en que consiste la relación con la variable respuesta.
Pregunta Una de las variables S presenta un pratón diferente que las demás, indique de qué variable estamos hablando y en qué consiste ese patrón diferente.
Can you write Python code for it?

import numpy as np
#representa la variable respueta vs cada una de las variables explicativas
#Y vs AGE, scatter plot
diabetes.plot(kind = "scatter", x="AGE", y="Y")
#Y vs SEX, scatter plot
diabetes.boxplot(by = "SEX", column="Y")
#Y vs BMI, scatter plot
diabetes.plot(kind = "scatter", x="BMI", y="Y")

#Y vs BP, scatter plot
diabetes.plot(kind = "scatter", x="BP", y="Y")

#Y vs S1, scatter plot
diabetes.plot(kind = "scatter", x="S1", y="Y")

#Y vs S2, scatter plot

diabetes.plot(kind = "scatter", x="S2", y="Y")

#Y vs S3, scatter plot
diabetes.plot(kind = "scatter", x="S3", y="Y")

#Y vs S4, scatter plot
diabetes.plot(kind = "scatter", x="S4", y="Y")

#Y vs S5, scatter plot
diabetes.plot(kind = "scatter", x="S5", y="Y")

#Y vs S6, scatter plot
diabetes.plot(kind = "scatter", x="S6", y="Y")


I am working on the file "Day1/Presentation_05_model_evaluation.ipynb".
The first task is:
Can you find an even better value for K?
Can you write Python code for it?

# try K=1 through K=25 and record testing accuracy
k_range = range(1, 26)
scores = [] # calculate accuracies for each value of K!

#Now we plot:

import matplotlib.pyplot as plt
# allow plots to appear within the notebook
%matplotlib inline

plt.plot(k_range, scores)
plt.xlabel('Value of K for KNN')
plt.ylabel('Testing Accuracy')

# try K=1 through K=25 and record testing accuracy
k_range = range(1, 26)
scores = []
for k in k_range:
    knn = KNeighborsClassifier(n_neighbors=k)
    knn.fit(X_train, y_train)
    y_pred = knn.predict(X_test)
    scores.append(metrics.accuracy_score(y_test, y_pred))


I am working on the file "Day1/Presentation_05_model_evaluation.ipynb".
The first task is:
KNN (K=1)
Can you write Python code for it?

knn = KNeighborsClassifier(n_neighbors=1)
knn.fit(X, y)
y_pred = knn.predict(X)
print metrics.accuracy_score(y, y_pred)


I am working on the file "Day1/Presentation_05_model_evaluation.ipynb".
The first task is:
KNN (K=5)
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X, y)
y_pred = knn.predict(X)
print metrics.accuracy_score(y, y_pred)


I am working on the file "Day1/Presentation_05_model_evaluation.ipynb".
The first task is:
Logistic regression
Can you write Python code for it?

# import the class
from sklearn.linear_model import LogisticRegression

# instantiate the model (using the default parameters)
logreg = LogisticRegression()

# fit the model with data
logreg.fit(X, y)

# predict the response values for the observations in X
y_pred = logreg.predict(X)
print(y_pred)
print("{0} predictions".format(len(y_pred)))


I am working on the file "Day1/Presentation_05_model_evaluation.ipynb".
The first task is:
Making predictions on out-of-sample data
Can you write Python code for it?

# instantiate the model with the best known parameters
knn = KNeighborsClassifier(n_neighbors=11)

# train the model with X and y (not X_train and y_train)
knn.fit(X, y)

# make a prediction for an out-of-sample observation
print(knn.predict([[3, 5, 4, 2]]))


I am working on the file "Day1/Presentation_05_model_evaluation.ipynb".
The first task is:
Model Evaluation Metrics for Regression
Mean Absolute Error (MAE) is the mean of the absolute value of the errors:
$$\frac 1n\sum_{i=1}^n|y_i-\hat{y}_i|$$
Mean Squared Error (MSE) is the mean of the squared errors:
$$\frac 1n\sum_{i=1}^n(y_i-\hat{y}_i)^2$$
Root Mean Squared Error (RMSE) is the square root of the mean of the squared errors:
$$\sqrt{\frac 1n\sum_{i=1}^n(y_i-\hat{y}_i)^2}$$
Read More
http://scikit-learn.org/stable/modules/model_evaluation.html#scoring-parameter
Generating an Optimal KNN classifier
Look back at 04_model_training and see how high an accuracy you can achieve for different values of n_neighbors.  Try to understand why different values do better than others in terms of the pictures we saw in 04_model_training.
You can change feature1 and feature2 in the cell below to visualize different projections of the data.
Can you write Python code for it?

feature1 = 1 # feature on x axis
feature2 = 3 # feature on y axis

data = X
f1vals = X[:,feature1]
f2vals = X[:,feature2]

import numpy as np
targets = dict(zip(range(3), iris.target_names))
features = dict(zip(range(4), iris.feature_names))
%matplotlib inline
import matplotlib.pyplot as plt
colors = ['g', 'r', 'b']
fig = plt.figure(figsize=(8,8))
ax = plt.subplot()
for species in targets.keys():
    f1 = f1vals[np.where(y==species)]
    f2 = f2vals[np.where(y==species)]
    ax.scatter(f1, f2, c=colors[species], label=targets[species], s=40)
    ax.set(xlabel=features[feature1], ylabel=features[feature2])
    ax.legend()


I am working on the file "Day1/Presentation_05_model_evaluation.ipynb".
The first task is:
Repeat for KNN with K=5:
Can you write Python code for it?

knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
y_pred = knn.predict(X_test)
print metrics.accuracy_score(y_test, y_pred)


I am working on the file "Day1/Presentation_05_model_evaluation.ipynb".
The first task is:
What did this accomplish?

Model can be trained and tested on different data
Response values are known for the testing set, and thus predictions can be evaluated
Testing accuracy is a better estimate than training accuracy of out-of-sample performance
Can you write Python code for it?

# STEP 1: split X and y into training and testing sets
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=4)

# print the shapes of the new X objects
print X_train.shape
print X_test.shape

# print the shapes of the new y objects
print y_train.shape
print y_test.shape

# STEP 2: train the model on the training set
logreg = LogisticRegression()
logreg.fit(X_train, y_train)

# STEP 3: make predictions on the testing set
y_pred = logreg.predict(X_test)

# compare actual response values (y_test) with predicted response values (y_pred)
print metrics.accuracy_score(y_test, y_pred)


I am working on the file "Day1/Presentation_05_model_evaluation.ipynb".
The first task is:
jupyter notebook 05_model_evaluation_tts.ipynb

Training Accuracy
Prediction accuracy on the same set of data you trained your model with.
Problems with training and testing on the same data

Goal is to estimate likely performance of a model on out-of-sample data
But, maximizing training accuracy rewards overly complex models that won't necessarily generalize
Unnecessarily complex models overfit the training data

Image Credit: Overfitting by Chabacano. Licensed under GFDL via Wikimedia Commons.
How Can We Avoid Overfitting?
Evaluation procedure #2: Train/test split

Split the dataset into two pieces: a training set and a testing set.
Train the model on the training set.
Test the model on the testing set, and evaluate how well we did.
Can you write Python code for it?

# print the shapes of X and y
print X.shape
print y.shape

# STEP 1: split X and y into training and testing sets
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=4)


I am working on the file "Day2/Decision_trees_forests.ipynb".
The first task is:
By averaging over 100 randomly perturbed models, we end up with an overall model which is a much better fit to our data!
(Note: above we randomized the model through sub-sampling... Random Forests use more sophisticated means of randomization, which you can read about in, e.g. the scikit-learn documentation)
As you can see, the non-parametric random forest model is flexible enough to fit the multi-period data, without us even specifying a multi-period model!
Example: Random Forest for Classifying Digits
We previously saw the hand-written digits data. Let's use that here to test the efficacy of the SVM and Random Forest classifiers.
Can you write Python code for it?

from sklearn.datasets import load_digits
digits = load_digits()
digits.keys()

X = digits.data
y = digits.target
print(X.shape)
print(y.shape)


I am working on the file "Day2/Decision_trees_forests.ipynb".
The first task is:
Notice that at each increase in depth, every node is split in two except those nodes which contain only a single class.
The result is a very fast non-parametric classification, and can be extremely useful in practice.
Question: Do you see any problems with this?
Decision Trees and over-fitting
One issue with decision trees is that it is very easy to create trees which over-fit the data. That is, they are flexible enough that they can learn the structure of the noise in the data rather than the signal! For example, take a look at two trees built on two subsets of this dataset:
Can you write Python code for it?

from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier()

plt.figure()
visualize_tree(clf, X[:200], y[:200], boundaries=False)
plt.figure()
visualize_tree(clf, X[-200:], y[-200:], boundaries=False)


I am working on the file "Day2/Decision_trees_forests.ipynb".
The first task is:
See how the details of the model change as a function of the sample, while the larger characteristics remain the same!
The random forest classifier will do something similar to this, but use a combined version of all these trees to arrive at a final answer:
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(n_estimators=100, random_state=0)
visualize_tree(clf, X, y, boundaries=False);


I am working on the file "Day2/Decision_trees_forests.ipynb".
The first task is:
The details of the classifications are completely different! That is an indication of over-fitting: when you predict the value for a new point, the result is more reflective of the noise in the model rather than the signal.
Random Forests
One possible way to address over-fitting is to use an Ensemble Method: this is a meta-estimator which essentially averages the results of many individual estimators which over-fit the data. Somewhat surprisingly, the resulting estimates are much more robust and accurate than the individual estimates which make them up!
One of the most common ensemble methods is the Random Forest, in which the ensemble is made up of many decision trees which are in some way perturbed.
The basic idea behind a RF is simple :
1- Grow a number "K" of different Decision trees, randomised over both the subset of training examples and the subset of attributes. The DTs should be splitted all the way to the pure state with no punning.
2- Use all of them to make predictions on the test set, and follow the majority vote.
There are volumes of theory and precedent about how to randomize these trees, but as an example, let's imagine an ensemble of estimators fit on subsets of the data. We can get an idea of what these might look like as follows:
Can you write Python code for it?

def fit_randomized_tree(random_state=0):
    X, y = make_blobs(n_samples=300, centers=4,
                      random_state=0, cluster_std=2.0)
    clf = DecisionTreeClassifier(max_depth=15)

    rng = np.random.RandomState(random_state)
    i = np.arange(len(y))
    rng.shuffle(i)
    visualize_tree(clf, X[i[:250]], y[i[:250]], boundaries=False,
                   xlim=(X[:, 0].min(), X[:, 0].max()),
                   ylim=(X[:, 1].min(), X[:, 1].max()))

from IPython.html.widgets import interact
interact(fit_randomized_tree, random_state=[0, 100]);


I am working on the file "Day2/Decision_trees_forests.ipynb".
The first task is:
We can check the accuracy of these classifiers:
Can you write Python code for it?

metrics.accuracy_score(ypred1, ytest)

metrics.accuracy_score(ypred2, ytest)


I am working on the file "Day2/Decision_trees_forests.ipynb".
The first task is:
We can quickly classify the digits using a decision tree as follows:
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
from sklearn import metrics

Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, random_state=0)

# Random Forest
RF1=RandomForestClassifier(n_estimators=100, criterion='gini', max_depth=None, min_samples_split=1,
                              min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features='auto',
                              max_leaf_nodes=None, bootstrap=True, oob_score=False, n_jobs=-1, random_state=None,
                              verbose=0, warm_start=False, class_weight=None)
RF1.fit(Xtrain, ytrain)
ypred1 = RF1.predict(Xtest)

# Decision Tree
DT1 = DecisionTreeClassifier(max_depth=None)
DT1.fit(Xtrain, ytrain)
ypred2 = DT1.predict(Xtest)


I am working on the file "Day2/Decision_trees_forests.ipynb".
The first task is:
We have some convenience functions in the repository that help
Can you write Python code for it?

from fig_code import visualize_tree, plot_tree_interactive


I am working on the file "Day2/SVC-basic.ipynb".
The first task is:
Let's examine how SVC tries to classify the points.
Before that, we want to first introduce a few attributes of the SVC class.

hyperplane: the plan that used to seperate the classes.
coef_:
intercept_:

support vectors: the training examples that are closest to the hyperplane are called support vectors.
decision functions: Distance of the samples X to the separating hyperplane.
Can you write Python code for it?

# get the separating hyper plane.
w=clf.coef_[0]
a=-w[0]/w[1]
xx=np.linspace(-5,5)
yy = a*xx-(clf.intercept_[0])/w[1]

# plot the paralles to the separating hyperplane that pass through the supporting vectors.
b=clf.support_vectors_[0]
yy_down = a*xx+(b[1]-a*b[0])
b=clf.support_vectors_[-1]
yy_up = a*xx + (b[1]-a*b[0])

#to compare with logistic regression
logistic = LogisticRegression()
logistic.fit(X,Y)
yy_log= -logistic.coef_[0][0]/logistic.coef_[0][1]*xx+logistic.intercept_[0]/logistic.coef_[0][1]

# let's look at the relation of these planes to our cluster of points.
plt.plot(xx,yy,'k-')
plt.plot(xx,yy_down,'k--')
plt.plot(xx,yy_up,'k--')
plt.plot(xx,yy_log,'r-')
plt.scatter(clf.support_vectors_[:,0],clf.support_vectors_[:,1],s=80,facecolors='none')
plt.scatter(X[:,0],X[:,1],c=Y,cmap=plt.cm.Paired)
plt.axis('tight')
plt.show()


I am working on the file "Day2/SVC-basic.ipynb".
The first task is:
We initial the SVC, and fit our data as simple as the following.
Can you write Python code for it?


I am working on the file "Day2/SVC-basic.ipynb".
The first task is:
We start with the simple example of linear SVC, here we use the 'linear' kernal of SVC.
We'll introduce the concept of a kernal later on.
We start with creating a data set that is a collection of two groups of points seperated by a plane.
Can you write Python code for it?

X = np.r_[np.random.randn(20,2) - [2,2],np.random.randn(20,2)+[2,2]]
Y = [0]*20+[1]*20
plt.scatter(X[:,0],X[:,1],c=Y,cmap=plt.cm.Paired)
plt.axis('tight')
plt.show()


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Approach 2:  Assign missing identifier
Can you write Python code for it?

df.loc[df['Cabin'].isnull(), 'Cabin'] = 'U0'
df.head()


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Approach 2: Factorizing (Make single multi-class feature)
Can you write Python code for it?

df['EmbarkedNum'] = pd.factorize(df['Embarked'])[0]
df.head(6)


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Approach 4: Fit a regression model to predict missing values
Can you write Python code for it?

for col in df.columns:
    print("NaNs in column {0} = {1}".format(col, df.loc[df[col].isnull()].shape[0]))

df.loc[df['Age'].isnull(), 'Age'] = df['Age'].mean()


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Boolean Masks
Can you write Python code for it?

mask = df['Embarked'] == 'C'
mask.head()


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Combining criteria / columns
Can you write Python code for it?

test = df.loc[(df['Age'] > 30.) & (df['Fare'] < 50.), 'Age':'Fare']
test.head()


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Continuous Features: Feature Scaling
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler().fit(df[['Age', 'Fare']]) # .fit(df) to scale all numerical columns
print("Means = {0}".format(scaler.mean_))
print("Stdevs = {0}".format(scaler.scale_))
df[['Age', 'Fare']] = scaler.transform(df[['Age', 'Fare']])
df.head()


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Data at a glance
Can you write Python code for it?


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Feature Engineering
Parsing Alphanumeric Features
Can you write Python code for it?


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Filtering
Can you write Python code for it?


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Filtering Dataframes
Can you write Python code for it?

df_filter = df[mask] # df_filter = df[df['Embarked'] == 'C']
df_filter.head()


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Getting a pipeline ready for sklearn
Combining Data Frames
Can you write Python code for it?

input_df = pd.read_csv('data/train.csv')
submit_df = pd.read_csv('data/test.csv')

print(input_df.shape)
print(submit_df.shape)

submit_df.head()

input_df.head()


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Indexing
Can you write Python code for it?

submit_df = pd.read_csv('data/test.csv', index_col=0)
submit_df.head()

input_df = pd.read_csv('data/train.csv', index_col=0)
submit_df = pd.read_csv('data/test.csv', index_col=0)
df = pd.concat([input_df, submit_df])
df.tail()

print(df.shape[1], "columns")
print(df.shape[0], "rows")
print(df.columns.values)


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Magic Pandas
Selecting Columns
Can you write Python code for it?

df['Fare'].head()

df[['Fare', 'Sex']].head()


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Plotting
Can you write Python code for it?

%matplotlib inline
import seaborn
fig = df['Pclass'].hist()

fig = df.hist(figsize=(15,5))


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Putting it all together
Can you write Python code for it?

def process_data(df):
    df['Female'] = pd.factorize(df['Sex'])[0]
    df.loc[df['Age'].isnull(), 'Age'] = df['Age'].mean()
    df.loc[df['Fare'].isnull(), 'Fare'] = df['Fare'].median()
    df.loc[df['Cabin'].isnull(), 'Cabin'] = 'U0'
    df.loc[df['Embarked'].isnull(), 'Embarked'] = df['Embarked'].dropna().mode()[0]

    dummies_df = pd.get_dummies(df['Embarked'])
    def addEmbarked(name):
        return 'Embarked_' + name
    dummies_df = dummies_df.rename(columns=addEmbarked)
    df = pd.concat([df, dummies_df], axis=1)
    df['EmbarkedNum'] = pd.factorize(df['Embarked'])[0]

    return df

df = process_data(df)
df.tail()

features = ['Age', 'Fare', 'Parch', 'Pclass', 'SibSp', 'Female', 'EmbarkedNum']
df_test = df.loc[df['Survived'].isnull(), features]
df_train = df.loc[df['Survived'].notnull(), features+['Survived']]
df_train.head()


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Regular Expressions
Can you write Python code for it?

import re
def getDeck(cabin):
    match = re.search("([A-Z])", cabin)
    return match.group(1) if match is not None else None
def getCabinNum(cabin):
    match = re.search("([0-9]+)", cabin)
    return match.group(1) if match is not None else None
print(getDeck('C237'))
print(getCabinNum('C237'))


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Requirements for working with data in scikit-learn

Features should not have missing values
Features and response should be numeric
Features and response should be NumPy arrays
Features and response are separate objects
Features and response should have specific shapes

Requirements for working with data in scikit-learn

Features should not have missing values
Features and response should be numeric
Features and response should be NumPy arrays
Features and response are separate objects
Features and response should have specific shapes

How to Find Valid Values
Can you write Python code for it?


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Requirements for working with data in scikit-learn

Features should not have missing values
Features and response should be numeric
Features and response should be NumPy arrays
Features and response are separate objects
Features and response should have specific shapes
Can you write Python code for it?

X_train = df_train[features].values
y_train = df_train['Survived'].values
print(X_train[0:5])
print(y_train[0:5])
print("X has {0} rows".format(X_train.shape[0]))
print("y has {0} rows".format(y_train.shape[0]))


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Right Way:
Can you write Python code for it?


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Selecting specific columns
df.loc[criterion, columns]
Can you write Python code for it?

test = df.loc[df['Age'] > 30., ['Age', 'Fare', 'Sex']]
test.head()


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
What are the classes?
Can you write Python code for it?

pd.factorize(df['Sex'])

df['Female'] = pd.factorize(df['Sex'])[0]
df.head()


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
What to do with the name?
Number of names
Can you write Python code for it?

testname = df.loc[1, 'Name']
print(testname)

re.split(' ', testname)

def numNames(name):
    return len(re.split(' ', name))

df['NumNames'] = df['Name'].map(numNames)
df.head()


I am working on the file "Day2/TitanicPandas.ipynb".
The first task is:
Wrong way:
Can you write Python code for it?

import numpy as np
df[df['Cabin']!=np.nan]


I am working on the file "Day2/Titanic_RF_Intermediate.ipynb".
The first task is:
Compare to Logistic Regression
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression
from sklearn import cross_validation
predictors = ["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare", "Embarked"]

alg    = LogisticRegression(random_state=1)
scores = cross_validation.cross_val_score(
    alg,
    train_data[predictors],
    train_data["Survived"],
    cv=3
)

print(scores.mean())


I am working on the file "Day2/Titanic_RF_Simple.ipynb".
The first task is:
Convert strings to integer classifiers.
Can you write Python code for it?

# female = 0, Male = 1
# Add a new column to the data file labeled "gender" that maps "Sex" into integer values.
train_df['Gender'] = train_df['Sex'].map( {'female': 0, 'male': 1} ).astype(int)


I am working on the file "Day2/Titanic_RF_Simple.ipynb".
The first task is:
Data is now ready to go. So lets fit the training data , then predict the test!
Can you write Python code for it?

# Convert back to a numpy array
train_data = train_df.values
test_data = test_df.values

print 'Training...'
# Initialize the training model, all parameters are default except n_estimators.
# This is the simplest random forest you can fit.
forest = RandomForestClassifier(n_estimators=1000, max_depth=None, min_samples_split=2,
                              min_samples_leaf=1, bootstrap=True, oob_score=False, n_jobs=-1)
# Fit the training data (X, Y)
forest = forest.fit( train_data[0::,1::], train_data[0::,0] )
print 'Predicting...'
# Predict outcomes from the test data
output = forest.predict(test_data).astype(int)

# Write your predictions into file
predictions_file = open("myfirstforest.csv", "wb")
open_file_object = csv.writer(predictions_file)
open_file_object.writerow(["PassengerId","Survived"])
open_file_object.writerows(zip(ids, output))
predictions_file.close()
print 'Done.'


I am working on the file "Day2/Titanic_RF_Simple.ipynb".
The first task is:
Redo the same for the test data
Can you write Python code for it?

test_df = pd.read_csv('data/test.csv', header=0)        # Load the test file into a dataframe

# I need to convert all strings to integer classifiers:
# female = 0, Male = 1
test_df['Gender'] = test_df['Sex'].map( {'female': 0, 'male': 1} ).astype(int)

# Embarked from 'C', 'Q', 'S'
# All missing Embarked -> just make them embark from most common place
if len(test_df.Embarked[ test_df.Embarked.isnull() ]) > 0:
    test_df.Embarked[ test_df.Embarked.isnull() ] = test_df.Embarked.dropna().mode().values
# Again convert all Embarked strings to int
test_df.Embarked = test_df.Embarked.map( lambda x: Ports_dict[x]).astype(int)

# All the ages with no data -> make the median of all Ages
median_age = test_df['Age'].dropna().median()
if len(test_df.Age[ test_df.Age.isnull() ]) > 0:
    test_df.loc[ (test_df.Age.isnull()), 'Age'] = median_age

# All the missing Fares -> assume median of their respective class
# Only for test data because training data is complete
if len(test_df.Fare[ test_df.Fare.isnull() ]) > 0:
    median_fare = np.zeros(3)
    for f in range(0,3):                                              # loop 0 to 2
        median_fare[f] = test_df[ test_df.Pclass == f+1 ]['Fare'].dropna().median()
    for f in range(0,3):                                              # loop 0 to 2
        test_df.loc[ (test_df.Fare.isnull()) & (test_df.Pclass == f+1 ), 'Fare'] = median_fare[f]


I am working on the file "Day2/Titanic_RF_advanced.ipynb".
The first task is:
Cleaning the data & preparing a submission file
Similar to what we did in the simplest version, but written differently.
Can you write Python code for it?

print('Cleaning the dataset...')
def harmonize_data(titanic):
    # Filling the blank data
    titanic["Age"] = titanic["Age"].fillna(titanic["Age"].mean())
    titanic["Fare"] = titanic["Fare"].fillna(titanic["Fare"].mean())
    titanic["Embarked"] = titanic["Embarked"].fillna("S")
    # Assigning binary form to data for calculation purpose
    titanic.loc[titanic["Sex"] == "male", "Sex"] = 1
    titanic.loc[titanic["Sex"] == "female", "Sex"] = 0
    titanic.loc[titanic["Embarked"] == "S", "Embarked"] = 0
    titanic.loc[titanic["Embarked"] == "C", "Embarked"] = 1
    titanic.loc[titanic["Embarked"] == "Q", "Embarked"] = 2
    return titanic

print('Defining submission file...')
def create_submission(rfc, train, test, predictors, filename):
    rfc.fit(train[predictors], train["Survived"])
    predictions = rfc.predict(test[predictors])
    submission = pd.DataFrame({
        "PassengerId": test["PassengerId"],
        "Survived": predictions
    })
    submission.to_csv(filename, index=False)

print('Defining the clean dataset...')
train_data = harmonize_data(train)
test_data  = harmonize_data(test)


I am working on the file "Day2/Titanic_RF_advanced.ipynb".
The first task is:
Creating new features
Can you write Python code for it?

print('Performing feature enginnering...')
train_data["PSA"] = train_data["Pclass"]*train_data["Sex"]*train_data["Age"]
train_data["SP"] = train_data["SibSp"]+train_data["Parch"]
test_data["PSA"] = test_data["Pclass"]*test_data["Sex"]*test_data["Age"]
test_data["SP"] = test_data["SibSp"]+test_data["Parch"]

print('Defining predictors...')
predictors = ["Pclass", "Sex", "Age", "PSA", "Fare", "Embarked", "SP"]


I am working on the file "Day2/Titanic_RF_advanced.ipynb".
The first task is:
Final random forest and submission
Can you write Python code for it?

# Finally, we just use the optimal parameters found to create the final RF and fit our data

print('Applying method...')
rfc = RandomForestClassifier(n_estimators=best_n, max_depth=best_m)
print('Creating submission...')
create_submission(rfc, train_data, test_data, predictors, "rfcsurvivors.csv")
print('Submitted.')


I am working on the file "Day2/Titanic_RF_advanced.ipynb".
The first task is:
Finding best values for n_estimators and max_depth
Can you write Python code for it?

print('Finding best n_estimators for RandomForestClassifier...')
max_score = 0
best_n = 0

# For each value of n_estimators:
# 1- Split/CV your data (KFold)
# 2- Fit RF
# 3- Calculate score for n_estimators by averaging up the individual scores of the 10 folds
# 4- Once best_n found, do the same for max_depth

for n in range(1,100):
    rfc_scr = 0.
    rfc = RandomForestClassifier(n_estimators=n)
    for train, test in KFold(len(train_data), n_folds=10, shuffle=True):
        rfc.fit(train_data[predictors].T[train].T, train_data["Survived"].T[train].T)
        rfc_scr += rfc.score(train_data[predictors].T[test].T, train_data["Survived"].T[test].T)/10
    if rfc_scr > max_score:
        max_score = rfc_scr
        best_n = n
print(best_n, max_score)

print('Finding best max_depth for RandomForestClassifier...')
max_score = 0
best_m = 0
for m in range(1,100):
    rfc_scr = 0.
    rfc = RandomForestClassifier(max_depth=m)
    for train, test in KFold(len(train_data), n_folds=10, shuffle=True):
        rfc.fit(train_data[predictors].T[train].T, train_data["Survived"].T[train].T)
        rfc_scr += rfc.score(train_data[predictors].T[test].T, train_data["Survived"].T[test].T)/10
    if rfc_scr > max_score:
        max_score = rfc_scr
        best_m = m
print(best_n, best_m, max_score)


I am working on the file "Day2/Transit Classification - Before we start .ipynb".
The first task is:
Classification problem with Unbalanced classes
We notice above that in our dataset, the amount of data in the two classes are extremely unbalanced.
13 Trues and 738 Falses.
Accuracy of the classification can no longer be a good criterial for our algorithm.
If predict False for every object, you can easily achieve 98.2% accuracy.
Instead, we want to look at the confusion metrix.
Can you write Python code for it?

# we create 40 separable points
rng = np.random.RandomState(0)
n_samples_1 = 1000
n_samples_2 = 100
X = np.r_[1.5 * rng.randn(n_samples_1, 2),
          0.5 * rng.randn(n_samples_2, 2) + [2, 2]]
y = [0] * (n_samples_1) + [1] * (n_samples_2)
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Paired)
plt.show()

# Split the data into a training set and a test set
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

# Run classifier, using a model that is too regularized (C too low) to see
# the impact on the results
#classifier = svm.SVC(kernel='linear', C=1)
classifier = svm.SVC(kernel='linear', C=1,class_weight={1: 10})
y_pred = classifier.fit(X_train, y_train).predict(X_test)

def plot_confusion_matrix(cm, title='Confusion matrix', cmap=plt.cm.Blues):
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(2)
    plt.xticks(tick_marks, ["0","1"], rotation=45)
    plt.yticks(tick_marks, ["0","1"])
    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')

# Compute confusion matrix
cm = confusion_matrix(y_test, y_pred)
np.set_printoptions(precision=2)
print('Confusion matrix, without normalization')
print(cm)
plt.figure()
plot_confusion_matrix(cm)

# Normalize the confusion matrix by row (i.e by the number of samples
# in each class)
cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
print('Normalized confusion matrix')
print(cm_normalized)
plt.figure()
plot_confusion_matrix(cm_normalized, title='Normalized confusion matrix')

plt.show()


I am working on the file "Day2/Transit Classification - Before we start .ipynb".
The first task is:
Precision, Recall and F measures:
Read More: http://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html#example-model-selection-plot-precision-recall-py
Can you write Python code for it?

from sklearn.metrics import precision_recall_curve, precision_score,recall_score,f1_score
classifier = svm.SVC(kernel='linear', C=1)
y_score = classifier.fit(X_train, y_train).decision_function(X_test)

y_pred = classifier.fit(X_train, y_train).predict(X_test)
print precision_score(y_test,y_pred), recall_score(y_test,y_pred), f1_score(y_test,y_pred)
precision, recall, _ = precision_recall_curve(y_test,y_score)
plt.plot(recall,precision)
classifier = svm.SVC(kernel='linear', C=1,class_weight={1: 10})

y_score2 = classifier.fit(X_train, y_train).decision_function(X_test)

y_pred2 = classifier.fit(X_train, y_train).predict(X_test)
print precision_score(y_test,y_pred2), recall_score(y_test,y_pred2),f1_score(y_test,y_pred2)

precision2, recall2, _ = precision_recall_curve(y_test,y_score2)
plt.plot(recall2,precision2)
plt.ylabel("Precision")
plt.xlabel("Recall")
plt.show()


I am working on the file "Day2/titanic_svm.ipynb".
The first task is:
Create a submission file with the Gender-Age model
First we want to read in the test data set and add in the gender features as what we did with the training data set.
Can you write Python code for it?

test_data = pd.read_csv('data/test.csv')
#print test_data.head()
#add in Sex_male features
test_data['Sex_male']=test_data.Sex.map({'female':0,'male':1})


I am working on the file "Day2/titanic_svm.ipynb".
The first task is:
Gender-Age model
We will introduce a few concepts while including the Age feature.
missing values

identify missing values
Can you write Python code for it?

#use the isnull function to check if there is any missing value in the Age column.
pd.isnull(data['Age']).any()


I am working on the file "Day2/titanic_svm.ipynb".
The first task is:
Here is how we examine how the selection works.
Can you write Python code for it?

xmin,xmax=X['Pclass'].min()-0.5,X['Pclass'].max()+0.5
ymin,ymax=X['Sex_male'].min()-0.5,X['Sex_male'].max()+0.5
print xmin,xmax,ymin,ymax
xx, yy = np.meshgrid(np.linspace(xmin, xmax, 200), np.linspace(ymin, ymax, 200))

model.fit(X,y)
Z = model.decision_function(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)

fig=plt.figure(figsize=(20,10))
ax=fig.add_subplot(111)
ax.pcolormesh(xx, yy, -Z, cmap=plt.cm.RdBu,alpha=0.5)
ax.scatter(X['Pclass']+np.random.randn(len(X['Pclass']))*0.1, X['Sex_male']+np.random.randn(len(X['Pclass']))*0.05, c=y,s=40, cmap=plt.cm.RdBu_r)
ax.set_xlabel("Pclass")
ax.set_ylabel("Sex_male")
ax.set_xlim([0.5,3.5])
ax.set_ylim([-0.5,1.5])
plt.show()


I am working on the file "Day2/titanic_svm.ipynb".
The first task is:
How many missing values are there?
Can you write Python code for it?


I am working on the file "Day2/titanic_svm.ipynb".
The first task is:
Let's examine the selection function of the model.
Can you write Python code for it?

xmin,xmax=X['Age'].min()-0.5,X['Age'].max()+0.5
ymin,ymax=X['Sex_male'].min()-0.5,X['Sex_male'].max()+0.5
print xmin,xmax,ymin,ymax
xx, yy = np.meshgrid(np.linspace(xmin, xmax, 200), np.linspace(ymin, ymax, 200))

model.fit(X,y)
Z = model.decision_function(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)

fig=plt.figure(figsize=(20,10))
ax=fig.add_subplot(111)
ax.pcolormesh(xx, yy, -Z, cmap=plt.cm.RdBu,alpha=0.5)
ax.scatter(X['Age'], X['Sex_male']+np.random.randn(len(X['Age']))*0.05, c=y,s=40, cmap=plt.cm.RdBu_r)
ax.set_xlabel("Normalized Age")
ax.set_ylabel("Sex_male")
ax.set_ylim([-0.5,1.5])
ax.set_xlim([-3,4.5])
plt.show()


I am working on the file "Day2/titanic_svm.ipynb".
The first task is:
Note here we give the missing values the mean age of the training data.
What's the pros and cons of doing this?
We want to get the features from the test data, and scale our age feature the same way as what we did in the training data.
Can you write Python code for it?

#generate our new feature
X_test=test_data[feature_cols]
X_test['Age']=(X_test['Age']-data['Age'].median())/data['Age'].std()


I am working on the file "Day2/titanic_svm.ipynb".
The first task is:
SVM does not allow features with missing values, what do we do?
One idea would be to fill in them with a number we think is reasonable.
Let's try to use the average age first.
Can you write Python code for it?


I am working on the file "Day2/titanic_svm.ipynb".
The first task is:
We notice again that some of the age value is missing in the test data, and want to fill in the same way as what we did with the training data.
Can you write Python code for it?

#use the isnull function to check if there is any missing value in the Age column.
pd.isnull(test_data['Age']).any()

print len(test_data['Age'][pd.isnull(test_data['Age'])])

test_data['Age'][pd.isnull(test_data['Age'])]=data['Age'].mean()


I am working on the file "Day2/titanic_svm.ipynb".
The first task is:
can you think of better ways to do this?
Can you write Python code for it?

#generate our new feature
feature_cols=['Age','Sex_male']
X=data[feature_cols]
X.head()

#use the default SVM rbf model
scores=cross_val_score(model,X,y,cv=10,scoring='accuracy')
print scores, np.mean(scores),np.std(scores)


I am working on the file "Day2/titanic_svm.ipynb".
The first task is:
feature rescaling
Can you write Python code for it?

X['Age']=(X['Age']-X['Age'].median())/X['Age'].std()
#X = StandardScaler().fit_transform(X)

scores=cross_val_score(model,X,y,cv=10,scoring='accuracy')
print scores, np.mean(scores),np.std(scores)


I am working on the file "Day2/titanic_svm_advanced.ipynb".
The first task is:
Are we nessary perform better than the simpler model?
The real test is to submit the file to kaggle and let their hold out set decide.
I did improve my result by ~0.03 with the newly added in name features.
Can you write Python code for it?

model.fit(X,y)
holdout_data = pd.read_csv('data/test_processed.csv')

# rescale age and fare as we did for training data.  This is critical
# Note that we can (and should) use the same scaler object we fit above to the training data
holdout_data[['Age', 'Fare']] = scaler.transform(holdout_data[['Age', 'Fare']])
holdout_data["PSA"] = holdout_data["Pclass"]*holdout_data["Female"]*holdout_data["Age"]
holdout_data["SP"] = holdout_data["SibSp"]+holdout_data["Parch"]

#use our new features.
#feature_cols=['Pclass','Age','SibSp','Parch','Fare','Female','Title_Dr','Title_Lady','Title_Master','Title_Miss','Title_Mr','Title_Mrs','Title_Rev','Title_Sir']
X_holdout=holdout_data[feature_cols]
X_holdout.head()

y_holdout=model.predict(X_holdout)
samplesubmit = pd.read_csv("data/titanic_submit_example.csv")
samplesubmit["Survived"]=np.int32(y_holdout)
samplesubmit.to_csv("data/titanic_submit_fancytitle.csv",index=False)


I am working on the file "Day2/titanic_svm_advanced.ipynb".
The first task is:
Below is a simple example of algorithm tuning with the rbf kernel of SVM.
Can you write Python code for it?

#tune the gamma parameter with our training set
scores_mean=[]
scores_std=[]
model=SVC()
model.C=1
gammas=np.logspace(-3,1,50)
for gamma in gammas:
    model.gamma=gamma
    scores=cross_val_score(model,X_train,y_train,cv=10,scoring='accuracy')
    scores_mean.append(np.mean(scores))
    scores_std.append(np.std(scores))
plt.semilogx(gammas,scores_mean,'.')
plt.show()


I am working on the file "Day2/titanic_svm_advanced.ipynb".
The first task is:
Of course, in real life you should perform parameter grid search in both C and gamma. Let's try out our new GridSearchCV tools learned in the morning.
Can you write Python code for it?

from sklearn.cross_validation import StratifiedShuffleSplit
from sklearn.grid_search import GridSearchCV
C_range = np.logspace(-3, 3, 10)
gamma_range = np.logspace(-3, 3, 10)
param_grid = dict(gamma=gamma_range, C=C_range)
cv = StratifiedShuffleSplit(y_train, n_iter=5, test_size=0.3, random_state=42)
grid = GridSearchCV(SVC(kernel='rbf'), scoring="accuracy", param_grid=param_grid, cv=cv)
grid.fit(X_train, y_train)
print("The best parameters are %s with a score of %0.4f" % (grid.best_params_, grid.best_score_))

model.gamma=grid.best_params_['gamma']
model.C=grid.best_params_['C']
model.fit(X_train,y_train)
y_predta=model.predict(X_train)
y_pred=model.predict(X_test)
train_score=metrics.accuracy_score(y_train,y_predta)
test_score=metrics.accuracy_score(y_test,y_pred)

scores=cross_val_score(model,X,y,cv=10,scoring='accuracy')
cvscore=np.mean(scores)
cvscore_std=np.std(scores)
print(train_score,test_score,cvscore,cvscore_std)


I am working on the file "Day2/titanic_svm_advanced.ipynb".
The first task is:
Select the features from data, and convert to numpy arrays:
Can you write Python code for it?


I am working on the file "Day2/titanic_svm_advanced.ipynb".
The first task is:
We want to perform algorithm tuning with CV now, to avoid information leak, let's create a hold out set
Can you write Python code for it?

#create a holdout set
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2,random_state=42)


I am working on the file "Day3/Intro-to-regression-ML.ipynb".
The first task is:
Gradient boosters
Gradient Boosting:
– Use a simple regression model to start
– Subsequent models predict the error residual of the previous predictions
– Overall prediction given by a weighted sum of the collection
Can you write Python code for it?

import pandas as pd
from sklearn.ensemble import GradientBoostingRegressor  #GBM algorithm
import numpy as np

from matplotlib import pyplot as plt
%matplotlib inline

bikes = pd.read_csv('data/bikes.csv')
regressor = GradientBoostingRegressor(max_depth=2,n_estimators=1000, learning_rate=0.05)
regressor.fit(bikes[['temperature', 'humidity']], bikes['count'])

nx = 30
ny = 30
# creating a grid of points
x_temperature = np.linspace(-5, 40, nx)
y_humidity = np.linspace(20, 80, ny)
xx, yy = np.meshgrid(x_temperature, y_humidity)
# evaluating the regresson on all the points
z_bikes = regressor.predict(np.array([xx.flatten(), yy.flatten()]).T)
zz = np.reshape(z_bikes, (nx, ny))

fig = plt.figure(figsize=(8, 8))
# plotting the predictions
plt.pcolormesh(x_temperature, y_humidity, zz, cmap=plt.cm.YlOrRd)
plt.colorbar(label='bikes predicted') # add a colorbar on the right
# plotting also the observations
plt.scatter(bikes['temperature'], bikes['humidity'], s=bikes['count']/25.0, c='g')
# setting the limit for each axis
plt.xlim(np.min(x_temperature), np.max(x_temperature))
plt.ylim(np.min(y_humidity), np.max(y_humidity))
plt.xlabel('temperature')
plt.ylabel('humidity')
plt.show()

plt.show()

from sklearn.metrics import mean_absolute_error
mean_absolute_error(bikes['count'], regressor.predict(bikes[['temperature','humidity']]))

from sklearn.cross_validation import cross_val_score
scores = -cross_val_score(regressor, bikes[['temperature', 'humidity']],
                          bikes['count'], scoring='mean_absolute_error', cv=10)

scores.mean()


I am working on the file "Day3/Intro-to-regression-ML.ipynb".
The first task is:
SVM regression
Can you write Python code for it?

import pandas as pd
from sklearn.svm import SVR
import numpy as np

from matplotlib import pyplot as plt
%matplotlib inline

bikes = pd.read_csv('data/bikes.csv')
regressor = SVR()
regressor.fit(bikes[['temperature', 'humidity']], bikes['count'])

nx = 30
ny = 30
# creating a grid of points
x_temperature = np.linspace(-5, 40, nx)
y_humidity = np.linspace(20, 80, ny)
xx, yy = np.meshgrid(x_temperature, y_humidity)
# evaluating the regresson on all the points
z_bikes = regressor.predict(np.array([xx.flatten(), yy.flatten()]).T)
zz = np.reshape(z_bikes, (nx, ny))

fig = plt.figure(figsize=(8, 8))
# plotting the predictions
plt.pcolormesh(x_temperature, y_humidity, zz, cmap=plt.cm.YlOrRd)
plt.colorbar(label='bikes predicted') # add a colorbar on the right
# plotting also the observations
plt.scatter(bikes['temperature'], bikes['humidity'], s=bikes['count']/25.0, c='g')
# setting the limit for each axis
plt.xlim(np.min(x_temperature), np.max(x_temperature))
plt.ylim(np.min(y_humidity), np.max(y_humidity))
plt.xlabel('temperature')
plt.ylabel('humidity')
plt.show()

plt.show()

from sklearn.metrics import mean_absolute_error
mean_absolute_error(bikes['count'], regressor.predict(bikes[['temperature','humidity']]))

from sklearn.cross_validation import cross_val_score
scores = -cross_val_score(regressor, bikes[['temperature', 'humidity']],
                          bikes['count'], scoring='mean_absolute_error', cv=10)

scores.mean()


I am working on the file "Day3/Lab_3_regress.ipynb".
The first task is:
Ejercicio 2. Separar nuestros datos en test y  training
Como siempre que estemos trabajando con modelos de machine learning. El primer paso, después de haber realizado el análisis explroratorio es separar en nuestros conjunto de test y training. Cabe recordar en este punto que esta separación es obligatoria para poder reportar una estimación de la performance de nuestro modelo lo más realista posible. En el caso de disponer de un menor número de datos, se puede optar por evaluar las prestaciones de los modelos mediante un esquema de cross-validation, pero en general este último esquema produce resultados un poco más optimistas. Este tipo de particiones se pueden realizar sencillamente con sklearn Cross-validation: evaluation estimator performance
Lo primero será obtener nuestra matrix de características $X$ y el vector respuesta $y$
Can you write Python code for it?

from sklearn.model_selection import train_test_split

X = data[list(data)[:-1]]
Y = data[list(data)[-1]]

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=0)

print X_train.shape
print X_test.shape


I am working on the file "Day3/Lab_3_regress.ipynb".
The first task is:
Ejercicio 3
En este ejercicio vamos entrenar nuestro primer modelo de regresión. En este caso, igual que para clasificación, os recomiendo por comenzar con el modelo más sencillo posible: Regresión Lineal. El modelo lineal permite aumentar nuestro conocimiento sobre la naturaleza del problema
Can you write Python code for it?

from sklearn import linear_model
from sklearn.metrics import accuracy_score

lin_reg = linear_model.LinearRegression()

lin_reg.fit(X_train, y_train)

coefs = [lin_reg.intercept_]
coefs.extend(list(lin_reg.coef_))

labels = ["bias"]
labels.extend(list(data)[:-1])

for n,c in zip(labels, coefs):
    print n,c(round(c, 3))
    print "------------------"


I am working on the file "Day3/Lab_3_regress.ipynb".
The first task is:
Ejercicio 5
En este punto, evolucionamos nuestro modelo y podemos utilizar modelos más complicados. Por ejemplo modelos no lineales: Vamos a compara con SVM y Random Forest
Igual que hicimos en el lab sobre clasificación, vamos a tener que utilizar una estrategia de búsqueda de los hiperparámetros de los modelos. Para ello vamos a utilizar GridSearch y cross-validation
Can you write Python code for it?

# SVM
from sklearn.svm import SVR
from sklearn.model_selection import GridSearchCV
#Se hace GridSearch para modelos que tengan hiperparametros

svm = SVR()
busqueda = {'C':np.logspace(1, 4, 10), 'epsilon': np.logspace(-4, 1, 10)}
gridSearch = GridSearchCV(svm, param_grid = busqueda, cv = 10, n_jobs = -1)
gridSearch.fit(X_train, y_train)

y_pred_svr = gridSearch.predict(X_test)
print "Usando SVM: ", r2_score(y_test, y_pred_svr)

#RandomForest
from sklearn.ensemble import RandomForestRegressor

rand_forest = RandomForestRegressor()
busquedaRF = {'n_estimators':[20, 50, 500]}
gridSearch = GridSearchCV(rand_forest, param_grid = busquedaRF, cv = 10, n_jobs = -1)
gridSearch.fit(X_train, y_train)

y_pred_rf = gridSearch.predict(X_test)

print "Usando Random Forest: ", r2_score(y_test, y_pred_rf)


I am working on the file "Day3/TransitClassification_Ensemble_part1.ipynb".
The first task is:
Frequency Spectra Features
SVM
Can you write Python code for it?

SVM_data=pd.read_csv("data/BL_CR_BLS_SVM_features.csv",index_col=0)
SVM_data.head()

X=SVM_data.drop(['Y','IDs'],axis=1)
Y=SVM_data['Y']
IDs=SVM_data['IDs']
X=StandardScaler().fit_transform(X)
trainX, testX, trainY, testY,trainIDs,testIDs= train_test_split(X, Y,IDs,test_size = 0.2)

model=SVC(gamma=0.05,C=50)
name="SVC"
data=[trainX,trainY,testX,testY,X,Y]
fit(model,name,data)

check_FPs(model,testX,testY,testIDs)


I am working on the file "Day3/TransitClassification_Ensemble_part1.ipynb".
The first task is:
GBC
Can you write Python code for it?

GBC_data=pd.read_csv("data/BL_CR_BLS_GBC_features.csv",index_col=0)
X=GBC_data.drop(['Y','IDs'],axis=1)
X=StandardScaler().fit_transform(X)
Y=GBC_data['Y']
IDs=GBC_data['IDs']
trainX, testX, trainY, testY,trainIDs,testIDs= train_test_split(X, Y,IDs,test_size = 0.2)
#GBC_data.head()

model=GradientBoostingClassifier(n_estimators=1000)
name="GBC"
data=[trainX,trainY,testX,testY,X,Y]
fit(model,name,data)

check_FPs(model,testX,testY,testIDs)


I am working on the file "Day3/TransitClassification_Ensemble_part1.ipynb".
The first task is:
Let's introduce the diagnostics we use for each algorithms:

confusion matrix
f1 score
ROC curve
FP/FN histogram
Can you write Python code for it?

def plot_confusion_matrix(cm, title='Confusion matrix', cmap=plt.cm.Blues):
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(2)
    plt.xticks(tick_marks, ['false positives', 'transits'], rotation=45)
    plt.yticks(tick_marks, ['false positives', 'transits'])
    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')

def fit(model,name,data):
    trainX,trainY,testX,testY,X,Y=data
    model.fit(trainX, trainY)
    predY = model.predict(testX)
    score = metrics.f1_score(predY, testY)
    cvscore= cross_val_score(model, X, Y, cv = 5, scoring = 'f1')
    print "#####################################"
    print "Result using",model
    print "f1 score from train test split %f" % score
    print "f1 score from CV5 %f" % np.mean(cvscore)
    cm = metrics.confusion_matrix(testY, predY)
    plot_confusion_matrix(cm)

    print cm
    make_ROC_curve(testY,predY,name)
    return

def check_FPs(model,testX,testY,testIDs):
    predY = model.predict(testX)
    index=predY==testY
    bins=np.linspace(1000,7000,13)
    print bins
    plt.hist(testIDs[index],bins=bins)
    plt.hist(testIDs[~index],bins=bins)
    return

def make_ROC_curve(testY, predY, name):
    fig2 = plt.figure()
    ax= fig2.add_subplot(1,1,1)
    fpr, tpr, _ = roc_curve(testY, predY)
    ax.plot(fpr, tpr, label = name)
    ax.set_title(('ROC Curve for %s') % name)
    ax.set_ylabel('True Positive Rate')
    ax.set_xlabel('False Positive Rate')


I am working on the file "Day3/TransitClassification_Ensemble_part1.ipynb".
The first task is:
Light Curve Features
Can you write Python code for it?


I am working on the file "Day3/TransitClassification_Ensemble_part1.ipynb".
The first task is:
RFC
Can you write Python code for it?


I am working on the file "Day3/TransitClassification_Ensemble_part1.ipynb".
The first task is:
SVM
Can you write Python code for it?

X=lc_data.drop('Y',axis=1)
Y=lc_data['Y']
#IDs=lc_darta['ID']
X=StandardScaler().fit_transform(X)
trainX, testX, trainY, testY= train_test_split(X, Y,test_size = 0.2)

model=SVC(gamma=0.005,C=100)
name="SVC"
data=[trainX,trainY,testX,testY,X,Y]
fit(model,name,data)


I am working on the file "Day3/restaurants_intermediate.ipynb".
The first task is:
Grid search the parameters space and fit a Gradient boosting
Can you write Python code for it?

# Define the parameters grid to search, notice the learning_rate parameter
param_grid2 = {'n_estimators':[100,1000],
'max_depth': [1,2,4],
'learning_rate': [0.1,0.01],
'min_samples_leaf': [1, 3, 5],
'max_features': [1.0, 0.3, 0.1]}
est2 = GradientBoostingRegressor()
gs_cv2 = GridSearchCV(est2, param_grid2,n_jobs=-1, cv=10).fit(X_train, Y_train)

# print best fit parameters
gs_cv2.best_params_

# Creating a GBR with the best fit parameters (entered manually)
gbr=GradientBoostingRegressor(max_depth= 4, max_features= 0.1, min_samples_leaf= 1, n_estimators= 100,learning_rate=0.01)
# Fit the training data
gbr=gbr.fit(X_train,Y_train )
# Predict the testing data
output = gbr.predict(testData)

# Write into submission file
predictions_file = open("interGB.csv", "w")
open_file_object = csv.writer(predictions_file)
open_file_object.writerow(["Id","Prediction"])
open_file_object.writerows(zip(ids, output))
predictions_file.close()


I am working on the file "Day3/restaurants_intermediate.ipynb".
The first task is:
Grid search the parameters space and fit a Random Forest
Can you write Python code for it?

# Define the parameters grid to search
param_grid = {'n_estimators':[100,1000],
'max_depth': [1,2,4],
'min_samples_leaf': [1, 3, 5],
'max_features': [1.0, 0.3, 0.1]}
est = RandomForestRegressor()
gs_cv = GridSearchCV(est, param_grid,n_jobs=-1, cv=10).fit(X_train, Y_train)

# print best fit parameters
gs_cv.best_params_

# Creating a RFR with the best fit parameters (entered manually)
forest=RandomForestRegressor(max_depth= 4, max_features= 0.1, min_samples_leaf= 3, n_estimators= 100)
# Fit the training data
forest=forest.fit(X_train,Y_train )
# Predict the testing data
output = forest.predict(testData)

# Write into submission file
predictions_file = open("interRF.csv", "w")
open_file_object = csv.writer(predictions_file)
open_file_object.writerow(["Id","Prediction"])
open_file_object.writerows(zip(ids, output))
predictions_file.close()


I am working on the file "Day3/resturants_simple.ipynb".
The first task is:
Convert 'Open date' , 'City', 'City Group, and 'Type' to numerical values
Can you write Python code for it?

# Replace 'Open Date' by a feature representing the age of the resturant in years
# Replace 'Type', 'City' and 'City Group' by integer indicators
trainData['Open Date'] = (datetime.now() - trainData['Open Date']).astype('timedelta64[D]') / 365
trainData['Type'] = LabelEncoder().fit_transform(trainData['Type'])
trainData['City Group'] = LabelEncoder().fit_transform(trainData['City Group'])
trainData['City'] = LabelEncoder().fit_transform(trainData['City'])
# Separate the Y array
Y_train = trainData['revenue']
# Drop the Id and Y variable to create the finale X array to be fitted
X_train = trainData.drop(['Id','revenue'], axis=1)

# Same for Test data
testData['Open Date'] = (datetime.now() - testData['Open Date']).astype('timedelta64[D]') / 365
testData['Type'] = LabelEncoder().fit_transform(testData['Type'])
testData['City Group'] = LabelEncoder().fit_transform(testData['City Group'])
testData['City'] = LabelEncoder().fit_transform(testData['City'])
ids = testData['Id'].values
testData = testData.drop(['Id'], axis=1)

# Note: You need to restart the Kernel after any modifications to this cell


I am working on the file "Day4/How HDBSCAN Works.ipynb".
The first task is:
Build the cluster hierarchy
Given the minimal spanning tree, the next step is to convert that into the hierarchy of connected components. This is most easily done in the reverse order: sort the edges of the tree by distance (in increasing order) and then iterate through, creating a new merged cluster for each edge. The only difficult part here is to identify the two clusters each edge will join together, but this is easy enough via a union-find data structure. We can view the result as a dendrogram as we see below:
Can you write Python code for it?


I am working on the file "Day4/How HDBSCAN Works.ipynb".
The first task is:
Now that we have the clusters it is a simple enough matter to turn that into cluster labelling as per the sklearn API. Any point not in a selected cluster is simply a noise point (and assigned the label -1). We can do a little more though: for each cluster we have the $\lambda_p$ for each point p in that cluster; If we simply normalize those values (so they range from zero to one) then we have a measure of the strength of cluster membership for each point in the cluster. The hdbscan library returns this as a probabilities_ attribute of the clusterer object. Thus, with labels and membership strengths in hand we can make the standard plot, choosing a color for points based on cluster label, and desaturating that color according the strength of membership (and make unclustered points pure gray).
Can you write Python code for it?

palette = sns.color_palette()
cluster_colors = [sns.desaturate(palette[col], sat)
                  if col >= 0 else (0.5, 0.5, 0.5) for col, sat in
                  zip(clusterer.labels_, clusterer.probabilities_)]
plt.scatter(test_data.T[0], test_data.T[1], c=cluster_colors, **plot_kwds)


I am working on the file "Day4/How HDBSCAN Works.ipynb".
The first task is:
Now, the best way to explain HDBSCAN is actually just use it and then go through the steps that occurred along the way teasing out what is happening at each step. So let's load up the hdbscan library and get to work.
Can you write Python code for it?

import hdbscan

clusterer = hdbscan.HDBSCAN(min_cluster_size=5, gen_min_span_tree=True)
clusterer.fit(test_data)


I am working on the file "Day4/How HDBSCAN Works.ipynb".
The first task is:
So now that we have clustered the data -- what actually happened? We can break it out into a series of steps

Transform the space according to the density/sparsity.
Build the minimum spanning tree of the distance weighted graph.
Construct a cluster hierarchy of connected components.
Condense the cluster hierarchy based on minimum cluster size.
Extract the stable clusters from the condensed tree.

Transform the space
To find clusters we want to find the islands of higher density amid a sea of sparser noise -- and the assumption of noise is important: real data is messy and has outliers, corrupt data, and noise. The core of the clustering algorithm is single linkage clustering, and it can be quite sensitive to noise: a single noise data point in the wrong place can act as a bridge between islands, gluing them together. Obviously we want our algorithm to be robust against noise so we need to find a way to help 'lower the sea level' before running a single linkage algorithm.
How can we characterize 'sea' and 'land' without doing a clustering? As long as we can get an estimate of density we can consider lower density points as the 'sea'. The goal here is not to perfectly distinguish 'sea' from 'land' -- this is an initial step in clustering, not the ouput -- just to make our clustering core a little more robust to noise. So given an identification of 'sea' we want to lower the sea level. For practical purposes that means making 'sea' points more distant from each other and from the 'land'.
That's just the intuition however. How does it work in practice? We need a very inexpensive estimate of density, and the simplest is the distance to the kth nearest neighbor. If we have the distance matrix for our data (which we will need imminently anyway) we can simply read that off; alternatively if our metric is supported (and dimension is low) this is the sort of query that kd-trees are good for. Let's formalise this and (following the DBSCAN, LOF, and HDBSCAN literature) call it the core distance defined for parameter k...
Can you write Python code for it?

clusterer.minimum_spanning_tree_.plot(edge_cmap='viridis',
                                      edge_alpha=0.6,
                                      node_size=80,
                                      edge_linewidth=2)


I am working on the file "Day4/How HDBSCAN Works.ipynb".
The first task is:
The next thing we'll need is some data. To make for an illustrative example we'll need the data size to be fairly small so we can see what is going on. It will also be useful to have several clusters, preferably of different kinds. Fortunately sklearn has facilities for generating sample clustering data so I'll make use of that and make a dataset of one hundred data points.
Can you write Python code for it?

moons, _ = data.make_moons(n_samples=50, noise=0.05)
blobs, _ = data.make_blobs(n_samples=50, centers=[(-0.75,2.25), (1.0, 2.0)], cluster_std=0.25)
test_data = np.vstack([moons, blobs])
plt.scatter(test_data.T[0], test_data.T[1], color='b', **plot_kwds)


I am working on the file "Day4/How HDBSCAN Works.ipynb".
The first task is:
This brings us to the point where robust single linkage stops. We want more though; a cluster hierarchy is good, but we really want a set of flat clusters. We could do that by drawing a a horizontal line through the above diagram and selecting the clusters that it cuts through. This is in practice what DBSCAN effectively does (declaring any singleton clusters at the cut level as noise). The question is, how do we know where to draw that line? DBSCAN simply leaves that as a (very unintuitive) parameter. Worse, we really want to deal with variable density clusters and any choice of cut line is a choice of mutual reachability distance to cut at, and hence a single fixed density level. Ideally we want to be able to cut the tree at different places to select our clusters. This is where the next steps of HDBSCAN begin and create the difference from robust single linkage.
Condense the cluster tree
The first step in cluster extraction is condensing down the large and complicated cluster hierarchy into a smaller tree with a little more data attached to each node. As you can see in the hierarchy above it is often the case that a cluster split is one or two points splitting off from a cluster; and that is the key point -- rather than seeing it as a cluster splitting into two new clusters we want to view it as a single persistent cluster that is 'losing points'. To make this concrete we need a notion of minimum cluster size which we take as a parameter to HDBSCAN. Once we have a value for minimum cluster size we can now walk through the hierarchy and at each split ask if one of the new clusters created by the split has fewer points than the minimum cluster size. If it is the case that we have fewer points than the minimum cluster size we declare it to be 'points falling out of a cluster' and have the larger cluster retain the cluster identity of the parent, marking down which points 'fell out of the cluster' and at what distance value that happened. If on the other hand the split is into two clusters each at least as...
Can you write Python code for it?


I am working on the file "Day4/Lab_4.ipynb".
The first task is:
Ejercicio 1
Ahora sí. Selección de características, usar L1 y SelectFromModel
Can you write Python code for it?

from sklearn.linear_model import LassoCV
from sklearn.feature_selection import SelectFromModel
#Hay que elegir un hiperparametro que será lambda

print X_train.shape
lscv = LassoCV() #Hará cross validation por dentro
lscv.fit(X_train, y_train) #Por defecto hay 100 lambdas, ejeY = MSE, ejeX = lambda. Se elige una lambda que me de el minimo
# punto de la curva.

models = SelectFromModel(lscv, prefit = True) #Extrae las caracteristicas mejores para el modelo que le pase
#En este caso es LassoCV

X_new = models.transform(X_train)

y_pred_lscv = lscv.predict(X_test)
print X_new.shape

print lscv.coef_

coefs = [lscv.intercept_]
coefs.extend(list(lscv.coef_))

labels = ['bias']
labels.extend(x_var_names)

"----"
for n,c in zip(labels,coefs):
    print
    print n,str(round(c,3))
    print "---------------"


I am working on the file "Day4/Lab_4.ipynb".
The first task is:
Ejercicio 2
Calcular el valor de r2_score para regresión lineal y para el modelo de LassoCV: ¿Cuál es mejor?
Can you write Python code for it?

from sklearn.metrics import r2_score

print "Regresion lineal -> ", r2_score(y_test, y_pred_rl)
print "LassoCV -> ", r2_score(y_test, y_pred_lscv)

li = linear_model.LinearRegression()
li.fit(X_new, y_train)

X_new_test = models.transform(X_test)

y_pred_newli = li.predict(X_new_test)

print "Regresion lineal new -> ", r2_score(y_test, y_pred_newli)


I am working on the file "Day4/Transit/MachineLearningWorkShop-TESSSimulatedData.ipynb".
The first task is:
Columns Ids, Catalog_Period, Depth, Catalog_Epoch records the information we have regarding the injected transits. Anything with period smaller than 0 is not a transit.
SNR is the signal to noise calculated for the transits using the catalog value.
SNR=\sqrt{Ntransit}*Depth/200mmag
There are three type of Y values included in this feature file:
-CatalogY marks True for all the transit planets with SNR larger than 8.5 and BLS_SignaltoPinkNoise_1_0 larger than 7.
-ManuleY marks True all the transits identified by eye.
-CombinedY marks True if either CatalogY and ManuleY is True.
The signals identified by manule effort but not by Catalog is due to blending. The signal missed by manule effort is either because of low signal to noise or due to cuts in Ntransit and Q value (standard practice before manuel inspection).
Let's drop the irrelevent columns before training:
Can you write Python code for it?

X=df.drop(['Ids','CatalogY','ManuleY','CombinedY','Catalog_Period','Depth','Catalog_Epoch','SNR'],axis=1)
#print X.isnull().any()

Y=df['CombinedY']

trainX, testX, trainY, testY= train_test_split(X, Y,test_size = 0.2)
data=[trainX,trainY,testX,testY,X,Y]
print X.shape, Y[Y==1].shape


I am working on the file "Day4/Transit/MachineLearningWorkShop-TESSSimulatedData.ipynb".
The first task is:
Feature Selection
Can you write Python code for it?

featurelist=X.columns
rfc= RandomForestClassifier(n_estimators=1000)
rfc.fit(trainX, trainY)

importances = rfc.feature_importances_
std = np.std([tree.feature_importances_ for tree in rfc.estimators_],
             axis=0)
indices = np.argsort(importances)[::-1]

# Print the feature ranking
print("Feature ranking:")
threshold=0.02
droplist=[]
for f in range(X.shape[1]):
    if importances[indices[f]]<threshold:
        droplist.append(featurelist[indices[f]])
    print("%d. feature %d (%s %f)" % (f + 1, indices[f], featurelist[indices[f]],importances[indices[f]]))

X_selected=X.drop(droplist,axis=1)
X_selected.head()

model=RandomForestClassifier(n_estimators=1000,n_jobs=-1,class_weight='balanced_subsample')
name="RFC"
fit(model,name,data)


I am working on the file "Day4/Transit/MachineLearningWorkShop-TESSSimulatedData.ipynb".
The first task is:
We can compare the prediction with the Manuel selection and the Catalog selection as the following:
Can you write Python code for it?

from sklearn.cross_validation import StratifiedKFold

model=RandomForestClassifier(n_estimators=3000,n_jobs=-1,class_weight='balanced_subsample',oob_score=True)

skf=StratifiedKFold(Y,n_folds=4)
i=1
for train_index,test_index in skf:
    trainX=X.iloc[train_index];testX=X.iloc[test_index]
    trainY=np.array(Y)[train_index];testY=np.array(Y)[test_index]
    #print train_index
    traincatY=np.array(df['CatalogY'])[train_index];testcatY=np.array(df['CatalogY'])[test_index]
    trainmanY=np.array(df['ManuleY'])[train_index];testmanY=np.array(df['ManuleY'])[test_index]
    model.fit(trainX,trainY)

    predY=model.predict_proba(testX)[:,1]
    rocscore = metrics.roc_auc_score(testY, predY)
    precision, recall, thresholds = metrics.precision_recall_curve(testY, predY)
    aucscore=metrics.auc(precision,recall,reorder=True)
    predY=model.predict(testX)
    f1score = metrics.f1_score(testY, predY)
    print "#####################################"
    print "fold %d:" % i
    print "f1 score from train test split %f" % f1score
    print "roc score from train test split %f" % rocscore
    print "auc score from train test split %f" % aucscore
    print "oob score from RF %f" % model.oob_score_

    flag1=(predY==1)*(predY==np.array(testY))

    flag2=(predY==1)*(predY==np.array(testmanY))

    print "predict Transit %d" % len(predY[predY==1])
    print "real Transit %d" % len(testY[testY==1])
    print "real Transit selected by eye %d" % len(testmanY[testmanY==1])
    print "predicted Transit that's real %d" % len(predY[flag1])
    print "predicted Transits selected by eye %d" % len(predY[flag2])
    i+=1

model=GradientBoostingClassifier(n_estimators=3000)

skf=StratifiedKFold(Y,n_folds=4)
i=1
for train_index,test_index in skf:
    trainX=X.iloc[train_index];testX=X.iloc[test_index]
    trainY=np.array(Y)[train_index];testY=np.array(Y)[test_index]
    #print train_index
    traincatY=np.array(df['CatalogY'])[train_index];testcatY=np.array(df['CatalogY'])[test_index]
   ...


I am working on the file "Day4/Transit/MachineLearningWorkShop-TESSSimulatedData.ipynb".
The first task is:
We show the results from some standard algorithms here:
Can you write Python code for it?

model=RandomForestClassifier(n_estimators=1000,class_weight={0:10,1:1},n_jobs=-1)
name="RFC"
fit(model,name,data,cv=False)

model=GradientBoostingClassifier(n_estimators=1000)
name="GBC"
fit(model,name,data,cv=False)

from xgboost import XGBClassifier
model = XGBClassifier(n_estimators=1000)
#model=XGBClassifier(learning_rate=0.1,
#    n_estimators=1000,
#    max_depth=5,
#    min_child_weight=1,
#    gamma=0,
#    subsample=0.8,
#    colsample_bytree=0.8,
#    objective='binary:logistic')
model.fit(trainX,trainY)
#model.plot_importance(bst)
#name="XGBoost"
fit(model,name,data,cv=False)


I am working on the file "Day5/GaussianProcesses/GP_MLWorkshop.ipynb".
The first task is:
In addition to the kernels specified above, additional more complicated kernels can constructed via the linear combination of multiple kernel functions. This is the adopted case for the stellar photometric variability. Namely we have rotationally modulated jitter arising from surface inhomogenities on the star such as cool spots and/or bright plages in the star's chromosphere. The data's correlation is therefore expected to have a periodic component at the stellar rotation period ($P_{\text{rot}}$). However, higher order effects such as the varying lifetimes, contrast, and spatial distribution of active regions will cause the correlations to instead be quasi-periodic. This covariance kernel can commonly constructed via the superposition of the squared exponential kernel
\begin{equation}
k(t,t') = \exp{\left( -\frac{(t-t')^2}{2 \lambda^2} \right)}
\end{equation}
and the periodic covariance kernel
\begin{equation}
k(t,t') = \exp{\left( -\Gamma^2 \sin^2{\left[ \frac{\pi |t-t'|}{P} \right]} \right)}.
\end{equation}
Together the quasi-periodic covariance kernel (with amplitude $a$) is
\begin{equation}
k(t,t') = a^2 \exp{\left( -\frac{(t-t')^2}{2 \lambda^2} -\Gamma^2 \sin^2{\left[ \frac{\pi |t-t'|}{P} \right]} \right)}.
\end{equation}
The quasi-periodic covariance kernel therefore has a set of four hyperparameters $\theta=(a,\lambda,\Gamma,P)$ where $a$ is the amplitude of the correlations, $\lambda$ is the exponentially decaying timescale (i.e. measurements further away from each other in time are less correlated than those that are close together), $\Gamma$ is this weird 'coherence' scale, and $P$ is the periodicity of the periodic component ($P=P_{\text{rot}}$ in the photometric light curve). See http://dan.iel.fm/gp.js/ to play with the effect of varying the hyperparameter values. For the quasi-periodic kernel at the aforementioned link, the order of the hyperparameter sliders is $a,\Gamma,P,\lambda$.
Using the optimized hyperparameters from my previous analysis of the above light curve, we can initialize our...
Can you write Python code for it?

# hyperparameters for the GJ 1132 light curve
a, l, G, Prot = 1.439e-5, 6.522e4, 6.883e-1, 1.224e2
my_gp = george.GP(a * (kernels.ExpSquaredKernel(l) * kernels.ExpSine2Kernel(G, Prot)))


I am working on the file "Day5/GaussianProcesses/GP_MLWorkshop.ipynb".
The first task is:
Now that we have specified the covariance matrix, we can draw samples from the GP prior function (using george.GP.sample). Remember that the GP is an $N$ dimensional multivariate Gaussian whose covariance is desribed by $K$. So despite us randomly sampling from a Gaussian distribution at each $t_i$, the resulting regression model will still be smooth.
Can you write Python code for it?

# Plotting prior samples
Nsamples = 5
fig = plt.figure(figsize=(10,5))
ax = fig.add_subplot(111)
t0 = round(min(self.bjd))
ax.plot(t-t0, y, '.', c='#a50f15', alpha=.7)
tm = np.linspace(t.min(), t.max(), 2e2)
for i in range(Nsamples):
    ax.plot(tm-t0, my_gp.sample(tm), '-')
plt.gca().invert_yaxis()
ax.set_xlabel('BJD - %i'%t0), ax.set_ylabel('Differential Magnitude'), ax.set_xlim((-10,max(self.modelbjd)-t0)), ax.set_ylim((.015,-.015))


I am working on the file "Day5/GaussianProcesses/GP_MLWorkshop.ipynb".
The first task is:
Optimizing the hyperparameters
The first method of GP hyperparmeter optimization is just likelihood maximization and is built-in to george via the scipy.stats.optimize function. This method is admittedly not very robust. Let's compare our previously adopted hyperparameter values to those obtained from the built-in likelihood maximization routine.
Can you write Python code for it?

theta_orig = my_gp.kernel.pars
# create a second GP with non-optimized parameters
theta_nonopt = my_gp.kernel.pars * np.random.uniform(.5, 1.5, 4)
a, l, G, Prot = theta_nonopt
my_2nd_gp = george.GP(a * (kernels.ExpSquaredKernel(l) * kernels.ExpSine2Kernel(G, Prot)))
# Optimize these hyperparameters
success = False
while not success:
    theta_opt, results = my_2nd_gp.optimize(t, y, ey)
    success = results['success']
labels = ['a','l','G','P']
for i in range(4):
    print '\n%s:'%labels[i]
    print 'Non-optimized %s = %.3e'%(labels[i], theta_nonopt[i])
    print 'Optimized %s = %.3e'%(labels[i], np.exp(theta_opt[i]))
    print 'Original %s = %.3e'%(labels[i], theta_orig[i])


I am working on the file "Day5/GaussianProcesses/GP_MLWorkshop.ipynb".
The first task is:
Run the above cell a few times. The optimal hyperparameters tend to vary sometimes by orders of magnitude (i.e. P).
Another common method is to use MCMC sampling the obtain a maximum a posteriori estimate of the hyperparameter values and their uncertainies. To do so one must evaulate the ln likelihood function at each step in their MCMC chain which requires a matrix inversion operation. George has a built-in method to compute the inverse of the covariance matrix via Cholesky decomposition. We can therefore fold the lnlikehood method into our own MCMC code for a given set of sampled hyperparameter values $\theta'$. Note that the dependent variable must already be corrected by the mean function; i.e. feed the residual vector ($\textbf{r} \equiv \textbf{y} - \boldsymbol{\mu}$) to george.
Can you write Python code for it?

# example of computing the likelihood
thetaprime = my_gp.kernel.pars * np.random.uniform(.9, 1.1, 4)
a, l, G, Prot = thetaprime
my_3rd_gp = george.GP(a * (kernels.ExpSquaredKernel(l) * kernels.ExpSine2Kernel(G, Prot)))
my_3rd_gp.compute(t, ey)
mean_func = np.zeros(t.size)    # our mean function is zero
lnl = my_3rd_gp.lnlikelihood(y - mean_func)
# put the above code into your lnlikelihood function which is called at each step in your MCMC chain


I am working on the file "Day5/GaussianProcesses/GP_MLWorkshop.ipynb".
The first task is:
We haven't told the GP about our data yet so the samples from the prior do not trace the observations but they do exhibit the expected covariance properties. Namely, smooth evolution (small $\Gamma$) with a distinct periodicity ($P\sim 122$ days) and a long timescale decay in amplitude ($\lambda \sim 6.5 \times 10^4$ days).
Conditioning the GP on the data
We can compute the posterior function (up to a constant) from which to sample given the input dataset and the likelihood function. Recall from Baye's theorem that
\begin{equation}
\ln{\text{posterior}} \propto \ln{\text{prior}} + \ln{\text{likelihood}}.
\end{equation}
For a GP with specified mean function $\boldsymbol{\mu}$ and covariance matrix $K$, the logarithmic likelihood function for the data $\mathbf{y}$ can be computed analytically via
\begin{equation}
\ln{\mathcal{L}} = -\frac{1}{2} \left( \mathbf{r}^T K^{-1} \mathbf{r} + \ln{\mathrm{det} K} + N \ln{2 \pi} \right),
\end{equation}
where $\mathbf{r} \equiv \mathbf{y} - \boldsymbol{\mu}$ is the residual vector. For our fiducial light curve in differential magnitudes, the data are already centered at zero so we take $\boldsymbol{\mu}$ to be the zero vector.
With the values of $\theta$ specified, we can compute the predictive posterior mean $M$ and covariance $C$ at epochs $t'$ via
\begin{align}
M &= K(t'_i, t_j) K(t_i,t_j)^{-1} y(t_j), \
C &= K(t'_i,t*_j) - K(t'_i, t_j) K(t_i,t_j)^{-1} K(t'_i, t_j)^T
\end{align}
and consequently sample from the predictive posterior distribution. However we are often most interested in the predictive mean (maximum a-posteriori model) and its variance (confidence intervals) at previously unseen epochs $t'$.
Can you write Python code for it?

tprime = np.linspace(t.min(), t.max(), 2e2)
M, C = my_gp.predict(y, tprime)  # give data to evaluate likelihood at prediction epochs tprime
sig = np.sqrt(np.diag(C))

# Plotting posterior mean and one standard deviation
fig = plt.figure(figsize=(10,5))
ax = fig.add_subplot(111)
t0 = round(min(self.bjd))
ax.errorbar(t-t0, y, ey, fmt='o', color='#a50f15', capsize=0, elinewidth=.8)
ax.plot(tprime-t0, M, '-', c='#fb6a4a', lw=2)
ax.fill_between(tprime-t0, M-sig, M+sig, color='#fb6a4a', alpha=.5)
plt.gca().invert_yaxis()
ax.set_xlabel('BJD - %i'%t0), ax.set_ylabel('Differential Magnitude'), ax.set_xlim((-10,max(self.modelbjd)-t0)), ax.set_ylim((.015,-.015))


I am working on the file "Day5/GaussianProcesses/GP_MLWorkshop.ipynb".
The first task is:
Why use GP regression?: parametric models aren't always complete
In the frequentist approach, one may assume a functional form of the model to the dataset.
Once the best-fit parameters of that model have been optimized, the model can be used to predict the values of $y_i$ at previously unseen epochs $t_i$. Recall that this is the primary goal of regression modelling in machine learning. In the above example, the brightness variations appear roughly periodic in time so we might be tempted to model the light curve with a purely sinusoidal function of the form
\begin{equation}
\Psi_i = A \sin{\left( \frac{2\pi t_i}{P_{\text{rot}}} + \phi \right)}.
\end{equation}
With this optimized model we can estimate the probability of obtaining certain values of the star’s brightness $\Psi_i$ the next time we go back to our telescope to observe the star by evaluating the model at those previously unseen epochs. This is great! We’ve obtained a model that we can use for predictive purposes! However, we have assumed a functional form of the model which itself can be thought of as a hyperparameter of the model to the dataset. Indeed this model may not be complete you may suspect to have been the case in this example given that the stellar photometric variability does not appear to be purely sinusoidal (i.e. the decaying photometric amplitude with time). Effects of the varying lifetimes, contrast, and spatial distribution of active regions on the stellar surface over adjacent rotation cycles may result in brightness variations which are not strictly periodic. In this case a purely sinusoidal function clearly does not completely capture the structure present in the star’s light curve and we should pursue a more complete model.
Why use GP regression?: advantages of non-parametricity (is that a real word?)
An alternative approach is to assign a prior probability to every possible function that could potentially model our photometric timeseries. We’ll call this the Bayesian approach because it incorporates our prior knowledge of...
Can you write Python code for it?

# Plotting
fig = plt.figure(figsize=(10,5))
ax = fig.add_subplot(111)
t0 = round(min(self.bjd))
ax.plot(self.bjdtrim-t0, self.magtrim, 'k.', alpha=.1)
ax.errorbar(t-t0, y, ey, fmt='o', color='#a50f15', capsize=0, elinewidth=.8)
ax.plot(self.modelbjd-t0, self.model, '-', c='#fb6a4a', lw=2)
ax.fill_between(self.modelbjd-t0, self.model-2*self.modelerr, self.model+2*self.modelerr, color='#fb6a4a', alpha=.5)
plt.gca().invert_yaxis()
ax.set_xlabel('BJD - %i'%t0), ax.set_ylabel('Differential Magnitude'), ax.set_xlim((-10,max(self.modelbjd)-t0)), ax.set_ylim((.02,-.02))


I am working on the file "Day5/Transit/Findout_Blended_stars.ipynb".
The first task is:
Let's create two stars as if their point spread functions (PSFs) are one-d gaussians with full width half maximum (FWHM) of 1.2 pixels, which means sigma ~0.5 pixel. Star one is twice the brightness as star two, and the two stars are one pixel apart. Note I haven't pixelize the flux in the left figure below.
Can you write Python code for it?

from scipy.stats import norm
FWHM=1.2
x=np.linspace(-3,3,50)
rv = norm(scale=FWHM/2.35)
starone=rv.pdf(x)*1000.
rv = norm(scale=FWHM/2.35)
startwo=(rv.pdf(x-1.0))*500.
fig=plt.figure()
ax=fig.add_subplot(121)
ax.plot(x,starone,label="star1")
ax.plot(x,startwo,label="star2")
ax.plot(x,starone+startwo,label="Total")
ax.set_xlabel("Pixel")
ax.set_ylabel("Flux")
ax.legend()
x_pixel=np.arange(7)-3
def pixelize(x,flux,x_pixel):
    flux_pixel=np.zeros(len(x_pixel))
    for i in xrange(len(x_pixel)):
        index=(x>=x_pixel[i]-0.5)*(x<x_pixel[i]+0.5)
        flux_pixel[i]=np.sum(flux[index])
    return flux_pixel
starone_pixel=pixelize(x,starone,x_pixel)
startwo_pixel=pixelize(x,startwo,x_pixel)
ax2=fig.add_subplot(122)
ax2.plot(x_pixel,starone_pixel,label="star1")
ax2.plot(x_pixel,startwo_pixel,label="star2")
total=starone_pixel+startwo_pixel
ax2.plot(x_pixel,total,label="Total")
ax2.set_xlabel("Pixel")
#ax2.set_ylabel("Flux")
#ax2.legend()
plt.show()


I am working on the file "Day5/Transit/Findout_Blended_stars.ipynb".
The first task is:
Let's first create a Eclipse signal that makes star one dimmer by 10%.
Can you write Python code for it?

fig=plt.figure()
ax=fig.add_subplot(121)
ax.plot(x,starone*0.9,label="star1")
ax.plot(x,startwo,label="star2")
ax.plot(x,starone*0.9+startwo,label="Total")
ax.set_xlabel("Pixel")
ax.set_ylabel("Flux")
ax.legend()
starone_pixel_ec=pixelize(x,starone*0.9,x_pixel)
startwo_pixel_ec=pixelize(x,startwo,x_pixel)
ax2=fig.add_subplot(122)
ax2.plot(x_pixel,starone_pixel_ec,label="star1")
ax2.plot(x_pixel,startwo_pixel_ec,label="star2")
total_ec=starone_pixel_ec+startwo_pixel_ec
ax2.plot(x_pixel,total_ec,label="Total")
ax2.set_xlabel("Pixel")
#ax2.set_ylabel("Flux")
plt.show()


I am working on the file "Day5/Transit/Findout_Blended_stars.ipynb".
The first task is:
Let's first have compare the Manuel selected signals with the catalog selected signals.
As a reminder:
- Manuel Y is those signals selected by human eye as looks like transit shape
- Catalog Y is those stars have signals injected in them at the original input catalog that we determined have enough signal to noise to be recovered
Can you write Python code for it?

#print df.shape
plt.plot(df['ManuleY']+np.random.randn(df.shape[0])*0.1,df['CatalogY']+np.random.randn(df.shape[0])*0.1,'.')
plt.xlim([-0.4,1.3])
plt.ylim([-0.4,1.3])
plt.xlabel('ManuleY')
plt.ylabel('CatalogY')


I am working on the file "Day5/Transit/Findout_Blended_stars.ipynb".
The first task is:
Let's look at the actual dimming of two stars assuming we take a radius equals to one apperture around both stars.
Can you write Python code for it?


I am working on the file "Day5/Transit/Findout_Blended_stars.ipynb".
The first task is:
Note that we added the new auc score which computes the area under the precision recall curve.
Can you write Python code for it?

model=GradientBoostingClassifier(n_estimators=3000)
name="GBC"
fit(model,name,data)

from xgboost import XGBClassifier
model = XGBClassifier(n_estimators=3000)
model.fit(trainX,trainY)
name="XGBoost"
fit(model,name,data)


I am working on the file "Day5/Transit/Findout_Blended_stars.ipynb".
The first task is:
There are four groups of cluster above:
- left bottom corner: manule selection determines most of the stars without a signal correctly.
- right top corner: manule selection determines some of the stars with a catalog signal correctly.
- left top coner: singals that's not sensitive to human eyes.
- right bottom coner: why would signals that's not in catalog appears to be ture to human eye?
The answer to the question is blending. And our task today is to further distinuish the stars affected by blending and those with true astrophysical signals.
Can you write Python code for it?


I am working on the file "Day5/Transit/Findout_Blended_stars.ipynb".
The first task is:
This is a slightly inbalanced sample, with the false sample twice as large as the true samples.
Can you write Python code for it?


I am working on the file "Day6/code/Decision-tree-from-scratch.ipynb".
The first task is:
Build a tree.
Can you write Python code for it?

my_decision_tree_new = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                                min_node_size = 100, min_error_reduction=0.0)

my_decision_tree_old = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                                min_node_size = 0, min_error_reduction=-1)


I am working on the file "Day6/code/Decision-tree-from-scratch.ipynb".
The first task is:
Early stopping condition 3: Minimum gain in error reduction
The function error_reduction takes 2 arguments:

The error before a split, error_before_split.
The error after a split, error_after_split.

This function computes the gain in error reduction, i.e., the difference between the error before the split and that after the split. This function will be used to detect this early stopping condition in the decision_tree_create function.
Can you write Python code for it?

def error_reduction(error_before_split, error_after_split):
    return error_before_split-error_after_split


I am working on the file "Day6/code/Decision-tree-from-scratch.ipynb".
The first task is:
Early stopping methods for decision trees
3 early stopping methods:

Reached a maximum depth. (set by parameter max_depth).
Reached a minimum node size. (set by parameter min_node_size).
Don't split if the gain in error reduction is too small. (set by parameter min_error_reduction).

Early stopping condition 1: Maximum depth
Early stopping condition 2: Minimum node size
The function reached_minimum_node_size takes 2 arguments:

The data (from a node)
The minimum number of data points that a node is allowed to split on, min_node_size.

This function simply calculates whether the number of data points at a given node is less than or equal to the specified minimum node size. This function will be used to detect this early stopping condition in the decision_tree_create function.
Can you write Python code for it?

def reached_minimum_node_size(data, min_node_size):
    if len(data)<=min_node_size:
        return True
    else:
        return False


I am working on the file "Day6/code/Decision-tree-from-scratch.ipynb".
The first task is:
Evaluating the models
Can you write Python code for it?

print "Training data, classification error (model 1):", evaluate_classification_error(model_1, train_data)
print "Training data, classification error (model 2):", evaluate_classification_error(model_2, train_data)
print "Training data, classification error (model 3):", evaluate_classification_error(model_3, train_data)

print "Training data, classification error (model 1):", evaluate_classification_error(model_1, validation_set)
print "Training data, classification error (model 2):", evaluate_classification_error(model_2, validation_set)
print "Training data, classification error (model 3):", evaluate_classification_error(model_3, validation_set)


I am working on the file "Day6/code/Decision-tree-from-scratch.ipynb".
The first task is:
Exploring the effect of min_error
Can you write Python code for it?

model_4 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                               min_node_size = 0, min_error_reduction=-1)
model_5 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                               min_node_size = 0, min_error_reduction=0)
model_6 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                               min_node_size = 0, min_error_reduction=5)

print "Validation data, classification error (model 4):", evaluate_classification_error(model_4, validation_set)
print "Validation data, classification error (model 5):", evaluate_classification_error(model_5, validation_set)
print "Validation data, classification error (model 6):", evaluate_classification_error(model_6, validation_set)

print count_leaves(model_4),count_leaves(model_5),count_leaves(model_6)


I am working on the file "Day6/code/Decision-tree-from-scratch.ipynb".
The first task is:
Exploring the effect of min_node_size
Can you write Python code for it?

model_7 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                               min_node_size = 0, min_error_reduction=-1)
model_8 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                               min_node_size = 2000, min_error_reduction=-1)
model_9 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                               min_node_size = 50000, min_error_reduction=-1)

print "Validation data, classification error (model 7):", evaluate_classification_error(model_7, validation_set)
print "Validation data, classification error (model 8):", evaluate_classification_error(model_8, validation_set)
print "Validation data, classification error (model 9):", evaluate_classification_error(model_9, validation_set)

print count_leaves(model_7),count_leaves(model_8),count_leaves(model_9)


I am working on the file "Day6/code/Decision-tree-from-scratch.ipynb".
The first task is:
Incorporating new early stopping conditions in binary decision tree implementation
Can you write Python code for it?

def decision_tree_create(data, features, target, current_depth = 0,
                         max_depth = 10, min_node_size=1,
                         min_error_reduction=0.0):
    remaining_features = features[:]
    target_values = data[target]
    print "--------------------------------------------------------------------"
    print "Subtree, depth = %s (%s data points)." % (current_depth, len(target_values))

    # Stopping condition 1: All nodes are of the same type.
    if intermediate_node_num_mistakes(target_values) == 0:
        print "Stopping condition 1 reached. All data points have the same target value."
        return create_leaf(target_values)

    # Stopping condition 2: No more features to split on.
    if remaining_features == []:
        print "Stopping condition 2 reached. No remaining features."
        return create_leaf(target_values)

    # Early stopping condition 1: Reached max depth limit.
    if current_depth >= max_depth:
        print "Early stopping condition 1 reached. Reached maximum depth."
        return create_leaf(target_values)

    # Early stopping condition 2: Reached the minimum node size.
    if reached_minimum_node_size(data,min_node_size):
        print "Early stopping condition 2 reached. Reached minimum node size."
        return create_leaf(target_values)

    splitting_feature = best_splitting_feature(data, features, target)
    left_split = data[data[splitting_feature] == 0]
    right_split = data[data[splitting_feature] == 1]
    error_before_split = intermediate_node_num_mistakes(target_values) / float(len(data))
    left_mistakes = intermediate_node_num_mistakes(left_split[target])
    right_mistakes = intermediate_node_num_mistakes(right_split[target])
    error_after_split = (left_mistakes + right_mistakes) / float(len(data))
    if error_reduction(error_before_split,error_after_split)<=min_error_reduction:
        print "Early stopping condition 3 reached. Minimum error reduction."
     ...


I am working on the file "Day6/code/Decision-tree-from-scratch.ipynb".
The first task is:
Making predictions
Can you write Python code for it?

def classify(tree, x, annotate = False):
    if tree['is_leaf']:
        if annotate:
            print "At leaf, predicting %s" % tree['prediction']
        return tree['prediction']
    else:
        split_feature_value = x[tree['splitting_feature']]
        if annotate:
            print "Split on %s = %s" % (tree['splitting_feature'], split_feature_value)
        if split_feature_value == 0:
            return classify(tree['left'], x, annotate)
        else:
            return classify(tree['right'], x, annotate)

validation_set[0]

print 'Predicted class: %s ' % classify(my_decision_tree_new, validation_set[0])

classify(my_decision_tree_new, validation_set[0], annotate = True)

classify(my_decision_tree_old, validation_set[0], annotate = True)


I am working on the file "Day6/code/Decision-tree-from-scratch.ipynb".
The first task is:
Measuring the complexity of the tree
complexity(T) = number of leaves in the tree T
Can you write Python code for it?

def count_leaves(tree):
    if tree['is_leaf']:
        return 1
    return count_leaves(tree['left']) + count_leaves(tree['right'])

print count_leaves(model_1),count_leaves(model_2),count_leaves(model_3)


I am working on the file "Day6/code/Decision-tree-from-scratch.ipynb".
The first task is:
Subsample dataset to make sure classes are balanced
undersample the larger class (safe loans) in order to balance out our dataset.
Can you write Python code for it?

safe_loans_raw = loans[loans[target] == 1]
risky_loans_raw = loans[loans[target] == -1]

percentage = len(risky_loans_raw)/float(len(safe_loans_raw))
safe_loans = safe_loans_raw.sample(percentage, seed = 1)
risky_loans = risky_loans_raw
loans_data = risky_loans.append(safe_loans)

print "Percentage of safe loans                 :", len(safe_loans) / float(len(loans_data))
print "Percentage of risky loans                :", len(risky_loans) / float(len(loans_data))
print "Total number of loans in our new dataset :", len(loans_data)


I am working on the file "Day6/code/Decision-tree-from-scratch.ipynb".
The first task is:
Transform categorical data into binary features
Can you write Python code for it?

loans_data = risky_loans.append(safe_loans)
for feature in features:
    loans_data_one_hot_encoded = loans_data[feature].apply(lambda x: {x: 1})
    loans_data_unpacked = loans_data_one_hot_encoded.unpack(column_name_prefix=feature)
    # Change None's to 0's
    for column in loans_data_unpacked.column_names():
        loans_data_unpacked[column] = loans_data_unpacked[column].fillna(0)
    loans_data.remove_column(feature)
    loans_data.add_columns(loans_data_unpacked)


I am working on the file "Day6/code/Decision-tree-from-scratch.ipynb".
The first task is:
reassign the labels to have +1 for a safe loan, and -1 for a risky (bad) loan.
Can you write Python code for it?

loans['safe_loans'] = loans['bad_loans'].apply(lambda x : +1 if x==0 else -1)
loans = loans.remove_column('bad_loans')


I am working on the file "Day6/code/Decision-tree-from-scratch.ipynb".
The first task is:
use 4 categorical features:
1. grade of the loan
2. the length of the loan term
3. the home ownership status: own, mortgage, rent
4. number of years of employment.
In the dataset, each of these features is a categorical feature. I will convert this to binary data in a subsequent section using 1-hot encoding.
Can you write Python code for it?

features = ['grade',              # grade of the loan
            'term',               # the term of the loan
            'home_ownership',     # home_ownership status: own, mortgage or rent
            'emp_length',         # number of years of employment
           ]
target = 'safe_loans'
loans = loans[features + [target]]