I am working on the file "HW04-Applied ML/HW04-2-RandomForest.ipynb".
The first task is:
Even for the 2 class problem it is hard to predict the colors and the classifier still mostly predicts WWW.
From that results we might conclude that there is just not enough difference between the 'black' and 'white' players to classify them.
Try other classifiers
A quick and short exploration of other classifiers to show that the RandomForest is not the 'wrong' classifier for that problem.
TLDR; They don't do better than the RandomForest.
Can you write Python code for it?

from sklearn.neural_network import MLPClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn import svm
from sklearn.ensemble import AdaBoostClassifier

def make_print_confusion_matrix(clf, clf_name):
    x_train, x_test, y_train, y_test = train_test_split(aggr_rf_input_data, player_colors_2, test_size=0.25)
    clf.fit(x_train, y_train)
    prediction = clf.predict(x_test)
    accuracy = np.mean(cross_val_score(clf, aggr_rf_input_data, player_colors_2, cv=5, n_jobs=3, pre_dispatch='n_jobs+1', verbose=1))
    print(clf_name + ' Accuracy: ',accuracy)
    cm = confusion_matrix(y_test, prediction)
    class_names = ['WWW', 'BBB']
    plot_confusion_matrix(cm, classes=class_names, title='Confusion matrix of '+clf_name)
    plt.show()


I am working on the file "HW04-Applied ML/HW04-2-RandomForest.ipynb".
The first task is:
First we just train and test the preprocessed data with the default values of the Random Forest to see what happens.
For this first model, we will use all the features (color_rating) and then we will observe which are the most important.
Can you write Python code for it?

player_colors = data['color_rating']
rf_input_data = data.drop(['color_rating'], axis=1)
player_colors.head() # values 1 to 5

rf = RandomForestClassifier()
cross_val_score(rf, rf_input_data, player_colors, cv=10, n_jobs=3, pre_dispatch='n_jobs+1', verbose=1)


I am working on the file "HW04-Applied ML/HW04-2-RandomForest.ipynb".
The first task is:
Indeed, some players appear around 200 times, so it is easy to determine the skin-color of the player djibril cisse if he appears both in the training set and in the test set. But in the reality the probability to have 2 djibril cisse with the same birthday and same color skin is almost null.
The reason why this attributes are so important is that some of the rows of one player appear in the train and test set, so the classifier can take those to determine the skin-color.
So we drop those attributes and see what happens.
Can you write Python code for it?

rf_input_data_drop = rf_input_data.drop(['birthday', 'player','playerShort', 'photoID'], axis=1)

rf = RandomForestClassifier()
result = cross_val_score(rf, rf_input_data_drop, player_colors, cv=10, n_jobs=3, pre_dispatch='n_jobs+1', verbose=1)

result


I am working on the file "HW04-Applied ML/HW04-2-RandomForest.ipynb".
The first task is:
Our model predicts almost only 2 categories instead of 5. It predicts mostly WW or W. This is because we have imbalanced data and the balancing did not really help apparently. We can see in the matrix above by looking at the True label that there is clearly a majority of of white player. Let's have a look at the exact distribution.
Can you write Python code for it?

fig, ax  = plt.subplots(1, 2, figsize=(8, 4))

ax[0].hist(aggr_player_colors)
ax[1].hist(aggr_player_colors, bins=3)


I am working on the file "HW04-Applied ML/HW04-2-RandomForest.ipynb".
The first task is:
Quite good results...
Observe the important features
Can you write Python code for it?

def show_important_features_random_forest(X, y, rf=None):
    if rf is None:
        rf = RandomForestClassifier()

    # train the forest
    rf.fit(X, y)

    # find the feature importances
    importances = rf.feature_importances_
    std = np.std([tree.feature_importances_ for tree in rf.estimators_], axis=0)
    indices = np.argsort(importances)[::-1]

    # plot the feature importances
    cols = X.columns
    print("Feature ranking:")
    for f in range(X.shape[1]):
        print("%d. feature n° %d %s (%f)" % (f + 1, indices[f], cols[indices[f]], importances[indices[f]]))

    # Plot the feature importances of the forest
    plt.figure()
    plt.title("Feature importances")
    plt.bar(range(X.shape[1]), importances[indices],
           color="r", yerr=std[indices], align="center")
    plt.xticks(range(X.shape[1]), indices)
    plt.xlim([-1, X.shape[1]])
    plt.show()

show_important_features_random_forest(rf_input_data, player_colors)


I am working on the file "HW04-Applied ML/HW04-2-RandomForest.ipynb".
The first task is:
The accuracy of the classifier dropped a bit, which is no surprise.
Can you write Python code for it?


I am working on the file "HW04-Applied ML/HW04-2-RandomForest.ipynb".
The first task is:
The results are not very impressive...
Can you write Python code for it?


I am working on the file "HW04-Applied ML/HW04-2-RandomForest.ipynb".
The first task is:
Those 2 histograms show the imbalance data. Indeed the 2 first category represent more than 50% of the data. Let's look at the numbers
Can you write Python code for it?

print('Proportion of WW: {:.2f}%'.format(
        100*aggr_player_colors[aggr_player_colors == 1].count()/aggr_player_colors.count()))
print('Proportion of W: {:.2f}%'.format(
        100*aggr_player_colors[aggr_player_colors == 2].count()/aggr_player_colors.count()))
print('Proportion of N: {:.2f}%'.format(
        100*aggr_player_colors[aggr_player_colors == 3].count()/aggr_player_colors.count()))
print('Proportion of B: {:.2f}%'.format(
        100*aggr_player_colors[aggr_player_colors == 4].count()/aggr_player_colors.count()))
print('Proportion of BB: {:.2f}%'.format(
        100*aggr_player_colors[aggr_player_colors == 5].count()/aggr_player_colors.count()))


I am working on the file "HW04-Applied ML/HW04-2-RandomForest.ipynb".
The first task is:
We can see that the most important features are:
    - photoID
    - player
    - the birthday
    - playerShort
The obtained result is weird. From personal experience, those 4 features should to be independant of the skin color and they also should be unique to one player. PhotoID is the id of the photo and thus unique for one player and independent of the skin_color. Same about 'player' and 'playerShort' (both represent the players name). Birthday is not necessarily unique, but should not be that important for the skin color since people all over the world are born all the time.
We have to remember that our data contains dyads between player and referee, so a player can appear several times in our data. It could be the reason why the unique features for the players are imprtant. Let's look at the data:
Can you write Python code for it?


I am working on the file "HW04-Applied ML/HW04-2-RandomForest.ipynb".
The first task is:
We see that our classifier is only a little bit better than the 'stupid' one. The difference between the ternary and binary classification is also small.
Confusion Matrix of the binary classifier:
Can you write Python code for it?

x_train, x_test, y_train, y_test = train_test_split(aggr_rf_input_data, player_colors_2, test_size=0.25)
rf_good.fit(x_train, y_train)
prediction = rf_good.predict(x_test)
accuracy = accuracy_score(y_test, prediction)
cm = confusion_matrix(y_test, prediction)
class_names = ['WWW', 'BBB']
plot_confusion_matrix(cm, classes=class_names, title='Confusion matrix')


I am working on the file "HW04-Applied ML/HW04-2-RandomForest.ipynb".
The first task is:
n_estimators
How many trees to be used. As expected, we see that more trees improve the train and test accuracy, however the test accuracy is bounded and it does not really make sense to use more than 500 trees. (Adding trees also means more computation time).
More trees also mean more overfitting. The train accuracy goes almost to 1 while the test stays around 0.42.
min_samples_leaf
The minimum number of samples required to be at a leaf node. The higher this value, the less overfitting. It effectively limits how good a tree can fit to a given train set.
criterion
The function to measure the quality of a split. You can see that 'entropy' scores higher in the test. So we take it even though gini has a much lover variance.
max_depth
The maximal depth of the tree. The higher the more the tree overfits. It seems that no tree is grown more than 10 deep. So we wont limit it.
max_leaf_nodes
An upper limit on how many leaf the tree can have. The train accuracy grows until about 400 where there is no more gain in more leaf nodes. probably because the trees don't create that big leaf nodes anyway.
min_samples_split
The minimum number of samples required to split an internal node. Has a similar effect and behaviour as min_samples_leaf.
class_weight
Weights associated with classes. Gives more weight to classes with fewer members. It does not seem to have a big influence.
Note that the third option is None which sets all classes weight to 1.
Find a good classifier
The default classifier achieves about 40% accuracy. This is not much considering that about 40% of players are in category 2. This classifier is not better than classifying all players into category 2.
So we are going to find better parameters for the classifier.
Based on the plots above and trial and error, we find good parameters for the RandomForestClassifier and look if feature importance changed.
Can you write Python code for it?

rf_good = RandomForestClassifier(n_estimators=500,
                                    max_depth=None,
                                    criterion='entropy',
                                    min_samples_leaf=2,
                                    min_samples_split=5,
                                    class_weight='balanced_subsample')

aggr_rf_input_data = data_aggregated.drop(['color_rating'], axis=1)
aggr_player_colors = data_aggregated['color_rating']

result = cross_val_score(rf_good, aggr_rf_input_data, aggr_player_colors,
                         cv=10, n_jobs=3, pre_dispatch='n_jobs+1', verbose=1)
print("mean result: ", np.mean(result))
result

show_important_features_random_forest(aggr_rf_input_data, aggr_player_colors, rf=rf_good)


I am working on the file "HW04-Applied ML/HW04-3-UnsupervizedLearning.ipynb".
The first task is:
Aggregation of the data
We aggregate the data the same way we did it for the 1st exercices. So as before you can find the aggregation in the notebook HW04-1-Preprocessing.
Here we just load the CVS containing the aggregation already encoded
Can you write Python code for it?

data_agr = pd.read_csv('CrowdstormingDataJuly1st_aggregated_encoded.csv')
data_agr.head()


I am working on the file "HW04-Applied ML/HW04-3-UnsupervizedLearning.ipynb".
The first task is:
Almost all the points are close to the center so the cluster 0 is almost represented by one point then this is why we have such a high silhouette score
Accuracy
Now have a look at the accuracy that seems pretty good
Can you write Python code for it?

colors = data_agr.color_rating
print('Arruracy: {:.2f} %'.format( 100- wrong_pred(colors, labels) * 100))


I am working on the file "HW04-Applied ML/HW04-3-UnsupervizedLearning.ipynb".
The first task is:
KMeans
Can you write Python code for it?

from sklearn import metrics
from sklearn.cluster import KMeans
np.random.seed(1)


I am working on the file "HW04-Applied ML/HW04-3-UnsupervizedLearning.ipynb".
The first task is:
Remember that we assume that a player with color rating N (value 3 in our case) cannot be misclusterised since we can't determined if it is more white or more black'. So let's see the distribution of white, neutral and black people
Can you write Python code for it?

colors3 = colors.apply(lambda x: mapping(x))
plt.hist(colors3)
print('Proportion of White: {:.2f}%'.format(len(colors3[colors3==0]) / len(colors3) * 100))
print('Proportion of Neutral: {:.2f}%'.format(len(colors3[colors3==0.5]) / len(colors3) * 100))
print('Proportion of Black: {:.2f}%'.format(len(colors3[colors3==1]) / len(colors3) * 100))


I am working on the file "HW04-Applied ML/HW04-3-UnsupervizedLearning.ipynb".
The first task is:
So first of all we can see that we have "separate clusters". But it's not clear at all that this 2 cluster return a silhouette score over 0.90. Let's rememenber what is the silhouette score (according to the documentation of sklearn):
-The Silhouette Coefficient is calculated using the mean intra-cluster distance (a) and the mean nearest-cluster distance (b) for each sample. The Silhouette Coefficient for a sample is (b - a) / max(a, b). To clarify, b is the distance between a sample and the nearest cluster that the sample is not a part of. Note that Silhouette Coefficent is only defined if number of labels is 2 <= n_labels <= n_samples - 1.

We can see that SC (silhouette score) is maximum when $\frac{b-a}{max(a,b)} = 1$. It can only happen when $a = 0$ and $b>0$. If $a=0$, it's mean that the mean intra-cluster distance is 0. It doesn't look that way when we just see the graph above.
With this graph it's not possible to see the concentration of point in each cluster. So let's plot an histogram to see the distribution of each cluster
Can you write Python code for it?

plt.figure(figsize=(2,2))
plt.hist(labels)
print('Number of elements in cluster 0: {}  ({:.2f}%)'.format(len(labels[labels == 0]) ,len(labels[labels == 0]) / len(labels) * 100))
print('Number of elements in cluster 0: {}  ({:.2f}%)'.format(len(labels[labels == 1]) ,len(labels[labels == 1]) / len(labels) * 100))


I am working on the file "HW04-Applied ML/HW04-3-UnsupervizedLearning.ipynb".
The first task is:
We drop the features that are unique to the players and we normalize them. That way all the features will be in [-1;1]. We also remove the color_ratings from the training data.
Can you write Python code for it?

data_agr = data_agr.drop(['playerShort', 'player'], axis=1)
data_train = data_agr.drop(['color_rating'], axis=1)
colors = data_agr['color_rating']
col = data_train.columns
data_train = pd.DataFrame(pp.normalize(data_train))
data_train.columns = col
data_train.head()


I am working on the file "HW04-Applied ML/HW04-3-UnsupervizedLearning.ipynb".
The first task is:
We have a lot of point concentrate close to 0. Let's see what is the value of the center of the cluster 0:
Can you write Python code for it?


I am working on the file "HW04-Applied ML/HW04-3-UnsupervizedLearning.ipynb".
The first task is:
assumption:
Since there are 5 variations of skin colors, there is no obvious choice of where to put the category 3. So we made the following choice.
- categories 1 and 2: light
- category 3: neutral. This category can't be wrongly categorized
- categories 4 and 5: dark
Can you write Python code for it?

def wrong_pred(ratings, labels):
    """returns the percentage of wrong prediction"""
    ratings = ratings.apply(lambda x: mapping(x))
    dif = np.abs(ratings - labels)
    # the difference between the ratings and labels has be 1 to count a as wrong prediction
    return (min(len(dif[dif==1]), len(dif[dif==0]))/len(labels))

def mapping(x):
    if (x < 3):
        return 0
    if (x == 3):
        return 0.5
    return 1


I am working on the file "HW04/Homework4.ipynb".
The first task is:
Firstly, we split the dataset into a training and a testing set. We build a stratified 10-fold cross validation indices generator and finally, we vectorize the (textual) data by fitting a TF-IDF vectorizer on the training set.
Can you write Python code for it?

X = news.data
Y = news.target
# Split data into a training and a testing set using stratified sampling
X_train, X_test, Y_train, Y_test = train_test_split(X, Y,
    test_size=0.1, random_state=7, stratify=Y)

# Create a stratified 10-fold cross-validation indices generator
SKF = StratifiedKFold(10, shuffle=True, random_state=7)

# Fit a TF-IDF vectorizer on the training set only (indeed we don't want to leak information from the testing
# set in the training set)
vectorizer = TfidfVectorizer().fit(X_train)
# We vectorize the textual data using the vectorizer (fitted only on the training set) for the testing and
# training sets
X_train = vectorizer.transform(X_train)
X_test = vectorizer.transform(X_test)


I am working on the file "HW04/Homework4.ipynb".
The first task is:
From this naive analysis, we can see that participating in the job training (JTP) program does not imply a significant increase of the income, it is even doing worst looking at the general trend. Indeed, we see that the median income from people who did not take the job training program is slightly higher from the ones who did take it. A few people (outliers) seemed to have fully benefit from the program, but as it concerns only a very small proportion, we cannot draw similar conclusion for the overall group. We also notice that the two categories are not evenly represented: there is much more people who did not participate in the JTP.
2. A closer look at the data
In this section, we focus the analysis of the data in four different contexts:

social status,
school background,
ethnic origin,
real earnings.
Can you write Python code for it?

def plot_(treat_group, control_group, title, left_column, left_xlabel, left_legend, right_column, right_xlabel, right_legend):

    lower_bound_left = min(min(treat_group[left_column]), min(control_group[left_column]))
    upper_bound_left = max(max(treat_group[left_column]), max(control_group[left_column]))
    range_left = (lower_bound_left, upper_bound_left)

    lower_bound_right = min(min(treat_group[right_column]), min(control_group[right_column]))
    upper_bound_right = max(max(treat_group[right_column]), max(control_group[right_column]))
    range_right = (lower_bound_right, upper_bound_right)

    num_bins=15

    fig, axes = plt.subplots(1, 2, figsize=(15,5), sharey=True)
    fig.suptitle(title, fontsize=14)
    treat_group[left_column].plot.hist(bins=num_bins, range=range_left, ax=axes[0], alpha=0.7, label='treat')
    control_group[left_column].plot.hist(bins=num_bins, range=range_left, ax=axes[0], alpha=0.5, label='control')
    axes[0].legend(loc=left_legend)
    axes[0].set_xlabel(left_xlabel)
    treat_group[right_column].plot.hist(bins=num_bins, range=range_right, ax=axes[1], alpha=0.7, label='treat')
    control_group[right_column].plot.hist(bins=num_bins, range=range_right, ax=axes[1], alpha=0.5, label='control')
    axes[1].legend(loc=right_legend)
    axes[1].set_xlabel(right_xlabel)
    plt.show()

import matplotlib.ticker as ticker

def plot_1binary_(treat_group, control_group, title, left_column, left_xlabel, left_legend, right_column, right_xlabel, right_legend):

    lower_bound_left = min(min(treat_group[left_column]), min(control_group[left_column]))
    upper_bound_left = max(max(treat_group[left_column]), max(control_group[left_column]))
    range_left = (lower_bound_left, upper_bound_left)

    num_bins=15

    fig, axes = plt.subplots(1, 2, figsize=(15,5), sharey=True)
    fig.suptitle(title, fontsize=14)
    treat_group[left_column].plot.hist(bins=num_bins, range=range_left, ax=axes[0], alpha=0.7, label='treat')
   ...


I am working on the file "HW04/Homework4.ipynb".
The first task is:
Here, we explore the importance of the different features. First, we show the top 10 most and least important features. As we can see, words which represent clearly a specific topic/context are the most important ones. For instance "clipper", "bike", "car", "sale", ... are words that clearly mark a specific context (car for automobile). There are however some exceptions like "the", "of", ...

The least significant ones are the outliers like we can see in the top 10 least important features: "62494hj6t8yv", "jarlehto", "6245". Indeed, these words are likely not frequent outliers which are not really representing a specific topic.

Bonus
Lastly, we tried to plot the features importance ordered from most to least importance. We see here an exponentially decaying function which indicates that a lot of features might not be relevant. So we tried to keep the 2000 first most important features and build a new model with the same parameters than the optimal one we found with Grid Search. Doing so we still have model with good accuracy (only 3% below the other ones). Conclusion: by keeping a bit more than 1% of the data we lost only 3% of accuracy. This shows that a few words matter to classify well the topics and there is a lot of words which bring just a tiny part of the information needed.
Can you write Python code for it?

# Get a mapping from index to word
idx_to_word = {v:k for k,v in vectorizer.vocabulary_.items()}

# Get feature immportance
importances = model.feature_importances_

df = pd.DataFrame([{
    'word': idx_to_word[feature],
    'feature': feature,
    'importance': importances[feature]
} for feature in range(len(importances))])

df = df.sort_values('importance', ascending=False).reset_index()
del df['index']

print('Top 10 most important features')
print(df.iloc[:10])
print()
print('Top 10 least important features')
print(df.iloc[-10:])

fig = df.importance.plot(rot=45)
fig.autoscale(tight=False)
plt.title('Plot of the importance of features ordered form the most important to the least')
plt.show()

fig = df.importance.iloc[:10000].plot(rot=45)
fig.autoscale(tight=False)
plt.title('Same plot but "zoomed" to the 10000 first most important features')
plt.show()

kept_words = [idx_to_word[f] for f in np.argsort(importances)[::-1][:2000]]

X = news.data
Y = news.target
# Split data into a training and a testing set using stratified sampling
X_train, X_test, Y_train, Y_test = train_test_split(X, Y,
    test_size=0.1, random_state=7, stratify=Y)

# Create a stratified 10-fold cross-validation indices generator
SKF = StratifiedKFold(10, shuffle=True, random_state=7)

# Fit a TF-IDF vectorizer on the training set only (indeed we don't want to leak information from the testing
# set in the training set)
vectorizer = TfidfVectorizer(vocabulary=kept_words).fit(X_train)
# We vectorize the textual data using the vectorizer (fitted only on the training set) for the testing and
# training sets
X_train = vectorizer.transform(X_train)
X_test = vectorizer.transform(X_test)

model = RandomForestClassifier(max_depth=best_params.max_depth.values[0],
                               n_estimators=best_params.n_estimators.values[0])
model.fit(X_train, Y_train)
pred = model.predict(X_test)
print('Accuracy on the testing set by keeping the first 2000 most important features: {0:.2f}%' .format(accuracy_score(pred, Y_test)*100))


I am working on the file "HW04/Homework4.ipynb".
The first task is:
Here, we fit a Random Forest model with just the default parameters and we compute our accuracy on the testing set.
Can you write Python code for it?

# We start by fitting a Ranfom Forest model on the training set using the default parameters
model = RandomForestClassifier()
model.fit(X_train, Y_train)
# We compute our prediction on the test set
pred = model.predict(X_test)
# We compute our accuracy based on our prediction on the testing set
print('Accuracy on the testing set with default parameters: {0:.2f}%' .format(accuracy_score(pred, Y_test)*100))


I am working on the file "HW04/Homework4.ipynb".
The first task is:
If we compare the black race to the other ethnic groups we see that the category of people who participated in the JTP is much more represented. We have the exact opposite for the hispanic race. We can conlcude that the representation of the different ethnic origins is very badly represented.
Can you write Python code for it?

plot_(treat_group_o, control_group_o, 'Earnings analysis', 're74', 'Real earnings from 1974', 'upper right',
      're75', 'Real earnings from 1975', 'upper right')


I am working on the file "HW04/Homework4.ipynb".
The first task is:
In the histogram of the earnings of 1978, we see that there is more people from the control group that have a very low salary (first bin). In the boxplot, we see that the the mean of the salary his higher in the treat group. The value of the third quantile is also higher. Hence, we can conclude (with more certainty than part 1) that the job training program has, although not so big, a positive effect on salaries.
Question 2
Can you write Python code for it?


I am working on the file "HW04/Homework4.ipynb".
The first task is:
Now we show the confusion matrix on the testing set. Here, it's interesting to see how the different classes are confused by the model. For instance, if we take the talk.religion.misc class, we can see that it is poorly classified (only 54% accuracy) and it's mainly confused with the class soc.religion.christian. This confusion makes sense since these two classes seem to be strongly related. Another example is the talk.politics.misc which is confused the most with talk.politics.gun which also makes sense, the two topics seem to be close and it's harder for the model to discriminate them.
Can you write Python code for it?

cm = confusion_matrix(Y_test, pred)
cm = pd.DataFrame(cm, index=news.target_names,
                  columns=news.target_names)
cm = cm.div(cm.sum(axis=1), axis=0)
plt.figure(figsize=(15, 15))
sns.heatmap(cm, square=True, annot=True, linewidths=0.1)
plt.title('Confusion matrix on testing set')
plt.show()


I am working on the file "HW04/Homework4.ipynb".
The first task is:
Question 1
Measure taken: we didn't created the white race category as it wasn't explicitely stated in the dataset.
1. A naive analysis
Can you write Python code for it?

def plot_re78(treat_group, control_group):

    lower_bound_left = min(min(treat_group['re78']), min(control_group['re78']))
    upper_bound_left = max(max(treat_group['re78']), max(control_group['re78']))
    range_ = (lower_bound_left, upper_bound_left)

    num_bins = 15

    fig, axes = plt.subplots(1, 1, figsize=(15,5), sharey=True)
    fig.suptitle('Real earnings in 1978', fontsize=14)
    treat_group['re78'].plot.hist(bins = num_bins, range=range_, ax=axes, alpha=0.7, label='treat')
    control_group['re78'].plot.hist(bins = num_bins, range=range_, ax=axes, alpha=0.5, label='control')
    axes.legend(loc='upper right')
    axes.set_xlabel('Income')
    plt.show()

def boxplot_re78(treat_group, control_group):
    merge = pd.concat([treat_group['re78'], control_group['re78']], axis=1)
    merge.columns = ['treat', 'control']

    fig, axes = plt.subplots(1, 1, figsize=(15,10))
    fig.suptitle("Boxplots", fontsize=14)
    merge.plot.box(ax=axes, sym='k.')
    axes.set_ylabel('Income')
    plt.show()

df = pd.read_csv('lalonde.csv')

treat_group_o = df[df['treat'] == 1]
control_group_o = df[df['treat'] == 0]

plot_re78(treat_group_o, control_group_o)
boxplot_re78(treat_group_o, control_group_o)

num_people_treat_o = treat_group_o['re78'].shape[0]
num_people_control_o = control_group_o['re78'].shape[0]

print('Number of people in treat group: ', num_people_treat_o, '\nNumber of people in control group: ', num_people_control_o)


I am working on the file "HW04/Homework4.ipynb".
The first task is:
The earnings do not fit very well between the treat and control groups.
All these observations allow us to realize that the sample selected for this analysis is very poorly represented in terms of the selected features. There is too much disparity. These inequalities skew the conclusions we can draw from the results obtained in the naive analysis. A much more sophisticated analysis is necessary to couteract these inequities.
3. A propensity score model
Can you write Python code for it?

from sklearn import linear_model
logistic = linear_model.LogisticRegression()

# the features
X = df[['age', 'educ', 'black', 'hispan', 'married', 'nodegree', 're74', 're75']]
# the output
y = df['treat']

model = logistic.fit(X, y)
# accuracy of the model
score = model.score(X,y)
# probability of each datapoint to be in treat group
probs_distr = model.predict_proba(X)[:,1]

print('Accuracy of logistic regression model: {0:.2f}%' .format(score*100))


I am working on the file "HW04/Homework4.ipynb".
The first task is:
The matches are not perfect yet but the difference between them has been greatly decreased. As a result, the conclusions drawn from the analysis will not be as biased as in the previous case.
5. Balancing the groups further
Looking at all the features, we see that the one that needs to be improved in priority is the black feature.
Can you write Python code for it?

num_black_treat = treat_group_o.black.sum()
num_black_control = control_group_o.black.sum()

print('There is {} black people in the treat group, and {} black people in the control group.'
      .format(num_black_treat, num_black_control))

num_people_treat = treat_group_o.shape[0]
num_people_control = control_group_o.shape[0]

print('There is {} people in the treat group and {} people in the control group in total.'
     . format(num_people_treat, num_people_control))

num_black = min(num_black_treat, num_black_control)
num_no_black = min(num_people_treat-num_black_treat, num_people_control-num_black_control)

group_size = num_black+num_no_black

print('This means that we will have {} people from each group.' .format(group_size))

# create a graph
G_black = nx.Graph()

# add all the indexes as nodes in the graph
for idx in df.index:
    G_black.add_node(idx)

# build the graph with weight between nodes as the distance we want to optimize making sure we match only subjects
# that have the same 'black' value
for idx_t in idx_treat:
    for idx_c in idx_control:
        if df.loc[idx_t, 'black'] == df.loc[idx_c, 'black']:
            w = 1 - (np.abs(probs_distr[idx_t] - probs_distr[idx_c]))
            G_black.add_edge(idx_t, idx_c, weight=w)

# dictionary with pairs where the weight is maximum
tot_matching = nx.max_weight_matching(G_black)

# take the first 'group_size' pairs of key-value
eq_matching = take(group_size, tot_matching.items())

idx_treat_sel = [x[0] for x in eq_matching]
idx_control_sel = [x[1] for x in eq_matching]

# define the new treat and control groups
treat_group_b = df.loc[idx_treat_sel]
control_group_b = df.loc[idx_control_sel]


I am working on the file "HW04/Homework4.ipynb".
The first task is:
We commented the cell below since its execution take a long time. We stored the results into a csv. If you want to re-run the code, you can just uncomment this cell. We optimized the parameters of the Random Forest model using a Grid Search with 10-fold stratified cross validation. Then, we show the results in the form of a heatmap. For computing time consideration, we restricted ourselves to a domain from 0 to 200 with a 20 step for the max_depth and n_estimators parameters. The best results we get was at max_dept=140 and n_estimators=200. Unfortunately, the optimum value were on the "border" of the grid, this suggests that we could have even better parameters by extending the domain of our grid. We didn't do it since it take a long time especially for models with high-valued parameters (it increases the complexity and thus the time to fit).
Can you write Python code for it?

## We choose a domain for each parameter we want to optimize
## For computation time consideration we restricted ourselves to a grid from 0 to 200 for each parameter
## with a step of 20
#tuned_parameters = {
#    'n_estimators': list(range(0, 201, 20))[1:],
#    'max_depth': list(range(0, 201, 20))[1:]
#}
#
## We then build a Grid Search object with our stratified cross validation indices generator precedently created
## So for each combination of parameters of the grid we'll run a 10-fold cross validation
#clf = GridSearchCV(RandomForestClassifier(), tuned_parameters, cv=SKF,
#                   scoring='accuracy', n_jobs=15, refit=False,
#                   return_train_score=True, verbose=2)
#
## We run the grid search
#clf.fit(X_train, Y_train)
#
## We save it since it's a relatively long operation
#pd.DataFrame(clf.cv_results_).to_csv('grid_search.csv')

# Code to show the results of the grid search through a hit map
res = pd.read_csv('grid_search.csv')
params = list(map(eval, res.params.values))
params = pd.DataFrame(params)
hm = pd.concat([params, res.mean_test_score, res.rank_test_score], axis=1)
best_params = hm[hm['rank_test_score'] == 1][['max_depth', 'n_estimators', 'mean_test_score']]

max_depth_labs = hm['max_depth'].unique()
nb_est_labs = hm['n_estimators'].unique()
mat = np.zeros(shape=(len(max_depth_labs), len(nb_est_labs)))
for i, depth in enumerate(max_depth_labs):
    for j, n_est in enumerate(nb_est_labs):
        mat[i, j] = hm[(hm['max_depth']==depth) & (hm['n_estimators']==n_est)]['mean_test_score'].values[0]
mat = pd.DataFrame(mat, index=hm.max_depth.unique(), columns=hm.n_estimators.unique())

plt.figure(figsize=(10, 10))
sns.heatmap(mat, square=True, annot=True)
plt.title('Heatmap: mean of cross-validation accuracy')
plt.show()

best_params

# Here we refit the model on the best parameter found using grid search
model = RandomForestClassifier(max_depth=best_params.max_depth.values[0],
                              ...


I am working on the file "HW04/Homework4.ipynb".
The first task is:
We see that there is a much better fit between the two categories compared to the same plot of part 1.
Can you write Python code for it?

# social status
plot_1binary_(treat_group, control_group, 'Age and Marriage analysis', 'age', 'Age', 'upper right',
      'married', 'Married', 'upper center')

# school background
plot_1binary_(treat_group, control_group, 'Education and School analysis', 'educ', 'Year of education', 'upper right',
      'nodegree', 'No earned school degree', 'upper center')
# ethnic origin
plot_2binary_(treat_group, control_group, 'Ethnic origin analysis', 'black', 'Black race', 'upper center',
      'hispan', 'Hispanic race', 'upper center')

# real earnings
plot_(treat_group, control_group, 'Earnings analysis', 're74', 'Real earnings from 1974', 'upper right',
      're75', 'Real earnings from 1975', 'upper right')


I am working on the file "HW04/Homework4.ipynb".
The first task is:
We see that there is not a very similar distribution of ages. For the distribution of married people among the groups, there is an even number of married and single people who did not participate in the JTP, but there is a significant difference among the ones who did participate.
Can you write Python code for it?

plot_1binary_(treat_group_o, control_group_o, 'Education and School analysis', 'educ', 'Year of education', 'upper right',
      'nodegree', 'No earned school degree', 'upper center')


I am working on the file "HW05-TamingText/Part2.ipynb".
The first task is:
Aggregate by countries
We groupe by country and compute the mean of each score
Can you write Python code for it?

sentiments = pd.read_csv('mailScore2.csv', index_col='Id')
sentiments.head()

def aggScoreByCountry(country):
    condition = sentiments.apply(lambda x: np.any(country.isin(x.SubjectBody.split())), axis=1)
    sent = sentiments[condition]
    if len(sent) == 0:
        print(country.Name, -999)
        return pd.Series({'Compound_':-999, 'Mean_':-999, 'Appearance': int(len(sent))})
    compound_ = np.mean(sent.Compound_)
    mean_ = np.mean(sent.Mean_)
    print(country.Name, compound_)
    return pd.Series({'Compound_': compound_, 'Mean_': mean_, 'Appearance': int(len(sent))})

countries = pd.concat([countries, countries.apply(lambda x: aggScoreByCountry(x), axis=1)],axis=1)
countries.to_csv('countriesScore.csv')


I am working on the file "HW05-TamingText/Part2.ipynb".
The first task is:
Analyse Emails
Can you write Python code for it?


I am working on the file "HW05-TamingText/Part2.ipynb".
The first task is:
Analysis
We will do a sentiment analysis on each sentense and then compute a socre for each country
We will compare different module:
    - nltk.sentiment.vader that attribute a score to each sentence
    - liuhu that has a set of positive word and one of neg word. We count the positif word and neg word in each sentence and compute the mean
Vader (That part takes time)
Can you write Python code for it?

sentiments.head()

from nltk.sentiment.vader import SentimentIntensityAnalyzer
sid = SentimentIntensityAnalyzer()
def sentiment_analysis(row):
    score = sid.polarity_scores(row)
    return pd.Series({'Pos': score['pos'], 'Neg': score['neg'], 'Compound_':score['compound'] })

sentiments = pd.concat([sentiments, sentiments.SubjectBody.apply(sentiment_analysis)], axis=1)

sentiments.to_csv('mailScore.csv')
sentiments.head()


I am working on the file "HW05-TamingText/Part2.ipynb".
The first task is:
Drop all country that have a score of -999 (they never appear in the mails)
Can you write Python code for it?

countries = countries[countries.Compound_ != -999]
len(countries)


I am working on the file "HW05-TamingText/Part2.ipynb".
The first task is:
First we create a dataframe with one line by countries and we count for each countries its occurences in the mail.
Since a country can be reference in many way (Switzerland, switzerland, CH), we need to consider all the possible form.
We may have a problem with word that have many meaning like US(country) and us (pronoun) so we can't just take all the country name in loer case and all the mail in lower case and just compare.
We first try to use that technique:
    1. the country name can appear either in lower case, with the first letter in uppercase or all in uppercase
    2. alpha_2 and alpha_3 are always used in uppercase
But we still have a lot of problem. Indeed a lot of mail contain sentences all in upper cas (see below):
       - SUBJECT TO AGREEMENT ON SENSITIVE INFORMATION & REDACTIONS. NO FOIA WAIVER. STATE-SCB0045012
For example this sentence will match for Togo because of the TO and also for norway because of the NO. An other example is Andorra that appears in 55 mails thanks to AND
At first we also wanted to keep the upper case since it can be helpfull to do the sentiment analysis. Look at those 2 sentence and their corresponding score:
    - VADER is very smart, handsome, and funny.: compound: 0.8545, neg: 0.0, neu: 0.299, pos: 0.701,
    - VADER is VERY SMART, handsome, and FUNNY.: compound: 0.9227, neg: 0.0, neu: 0.246, pos: 0.754,
The score is not the same. But since we have a lot of information in Upper case and it nothing to do with sentiment, we will put all mails in lower case. And we will also remove the stopwords.
We know that we remove the occurance of USA under 'us' but it will also remove the 'and', 'can', 'it',...
Can you write Python code for it?


I am working on the file "HW05-TamingText/Part2.ipynb".
The first task is:
Import data
Can you write Python code for it?

folder = 'hillary-clinton-emails/'

emails = pd.read_csv(folder + 'Emails.csv', index_col='Id')
emails.head(5)


I am working on the file "HW05-TamingText/Part2.ipynb".
The first task is:
It's a lot of country. We will also use a thresold for the appearance. We only keep mails that are mentioned in a minimum number of emails
Can you write Python code for it?

minimum_appearance = 15
countries_min = countries[countries.Appearance >= minimum_appearance]
len(countries_min)


I am working on the file "HW05-TamingText/Part2.ipynb".
The first task is:
Liuhu
Can you write Python code for it?

from nltk.corpus import opinion_lexicon
sentimentsLihuh = pd.read_csv('mailScore.csv', index_col='Id')
#transform the array of positiv and negatif word in dict
dicPosNeg =  dict()
for word in opinion_lexicon.positive():
    dicPosNeg[word] = 1

for word in opinion_lexicon.negative():
    dicPosNeg[word] = -1

def sentiment_liuhu(sentence):
    counter = []
    for word in sentence.split():
        value = dicPosNeg.get(word, -999)
        if value != -999:
            counter.append(value)

    if len(counter) == 0 :
        return pd.Series({'Sum_': int(0), 'Mean_': int(0) })
    return pd.Series({'Sum_': np.sum(counter), 'Mean_': np.mean(counter) })

sentimentsLihuh = pd.concat([sentimentsLihuh, sentimentsLihuh.SubjectBody.apply(sentiment_liuhu)], axis=1)

sentimentsLihuh.to_csv('mailScore2.csv')
sentimentsLihuh


I am working on the file "HW05-TamingText/Part2.ipynb".
The first task is:
Now let's try to combine the subject and the body and drop the mail that have both subject= NaN and body = Nan
Can you write Python code for it?

subBodyNan = emails[np.logical_and(emails.ExtractedBodyText.isnull(),emails.ExtractedSubject.isnull())]
print('Number of email where both subject and body is NaN: {}({:.2f})'.format(len(subBodyNan), len(subBodyNan)/ len(emails)))


I am working on the file "HW05-TamingText/Part2.ipynb".
The first task is:
Sentiment analysis
We explain before our precessing. Now we will do the sentiment analysis only on the subject and the body
So we will only consider the subject and the body
Can you write Python code for it?

sentiments = pd.DataFrame(emails_country.SubjectBody)
sentiments.head()

sentiments.shape


I am working on the file "HW05-TamingText/Part2.ipynb".
The first task is:
Tokenize and remove stopwords
Can you write Python code for it?

from gensim import corpora, models, utils
from nltk.corpus import stopwords
sw = stopwords.words('english') + ['re', 'fw', 'fvv', 'fwd']
sw = sw + ['pm', "a", "about", "above", "above", "across", "after", "afterwards", "again", "against", "all", "almost", "alone", "along", "already", "also","although","always","am","among", "amongst", "amoungst", "amount",  "an", "and", "another", "any","anyhow","anyone","anything","anyway", "anywhere", "are", "around", "as",  "at", "back","be","became", "because","become","becomes", "becoming", "been", "before", "beforehand", "behind", "being", "below", "beside", "besides", "between", "beyond", "bill", "both", "bottom","but", "by", "call", "can", "cannot", "cant", "co", "con", "could", "couldnt", "cry", "de", "describe", "detail", "do", "done", "down", "due", "during", "each", "eg", "eight", "either", "eleven","else", "elsewhere", "empty", "enough", "etc", "even", "ever", "every", "everyone", "everything", "everywhere", "except", "few", "fifteen", "fify", "fill", "find", "fire", "first", "five", "for", "former", "formerly", "forty", "found", "four", "from", "front", "full", "further", "get", "give", "go", "had", "has", "hasnt", "have", "he", "hence", "her", "here", "hereafter", "hereby", "herein", "hereupon", "hers", "herself", "him", "himself", "his", "how", "however", "hundred", "ie", "if", "in", "inc", "indeed", "interest", "into", "is", "it", "its", "itself", "keep", "last", "latter", "latterly", "least", "less", "ltd", "made", "many", "may", "me", "meanwhile", "might", "mill", "mine", "more", "moreover", "most", "mostly", "move", "much", "must", "my", "myself", "name", "namely", "neither", "never", "nevertheless", "next", "nine", "no", "nobody", "none", "noone", "nor", "not", "nothing", "now", "nowhere", "of", "off", "often", "on", "once", "one", "only", "onto", "or", "other", "others", "otherwise", "our", "ours", "ourselves", "out", "over", "own","part", "per", "perhaps", "please", "put", "rather", "re", "same", "see", "seem", "seemed", "seeming", "seems", "serious",...


I am working on the file "HW05-TamingText/Part3.ipynb".
The first task is:
Now we generate the LDA models with different numbers of topics.
Can you write Python code for it?

import pprint
pp = pprint.PrettyPrinter(depth=2)
for i in range(5, 50, 10):
    print('------------------------------------')
    print(i, 'topics')
    lda = models.LdaModel(corpus, num_topics=i, id2word = dictionary)
    pp.pprint(lda.print_topics(lda.num_topics))
    print()


I am working on the file "HW05-TamingText/Part3.ipynb".
The first task is:
To improve the result of the lda model, we group the mails by Subject. We filter the subjects to remove the keywords 're', 'fw', 'fvv', 'fwd'
Can you write Python code for it?

def filt(row):
    t = utils.simple_preprocess(row.ExtractedSubject)
    filt = list(filter(lambda x: x not in sw, t))
    return ' '.join(filt)
texts['ExtractedSubject'] = texts.apply(filt, axis=1)
texts = texts.groupby(by='ExtractedSubject', as_index=False).apply(lambda x: (x + ' ').sum())

texts.head()


I am working on the file "HW05-TamingText/Part3.ipynb".
The first task is:
To use the LDA model, we need to transform every document into a list of tuple (ID, term frequency).
Can you write Python code for it?

dictionary = corpora.Dictionary(documents)
corpus = [dictionary.doc2bow(doc) for doc in documents]


I am working on the file "HW05-TamingText/Part3.ipynb".
The first task is:
we concat the e-mail body wit the subject.
Can you write Python code for it?

texts.ExtractedBodyText.fillna('',inplace=True)
texts.ExtractedSubject.fillna('',inplace=True)
texts['SubjectBody'] = texts.ExtractedBodyText +  ' ' + texts.ExtractedSubject
mails = texts.SubjectBody


I am working on the file "HW05-TamingText/Part3.ipynb".
The first task is:
when we choose the number of topics to be 25, the topics seem to be the most meaningfull.
Can you write Python code for it?

lda = models.LdaModel(corpus, num_topics=25, id2word = dictionary)
pp.pprint(lda.print_topics(lda.num_topics))


I am working on the file "HW0_solutions.ipynb".
The first task is:
If any of these libraries are missing or out of date, you will need to install them and restart IPython
Hello matplotlib
The notebook integrates nicely with Matplotlib, the primary plotting package for python. This should embed a figure of a sine wave:
Can you write Python code for it?

#this line prepares IPython for working with matplotlib
%matplotlib inline

# this actually imports matplotlib
import matplotlib.pyplot as plt

x = np.linspace(0, 10, 30)  #array of 30 points from 0 to 10
y = np.sin(x)
z = y + np.random.normal(size=30) * .2
plt.plot(x, y, 'ro-', label='A sine wave')
plt.plot(x, z, 'b-', label='Noisy sine')
plt.legend(loc = 'lower right')
plt.xlabel("X axis")
plt.ylabel("Y axis")


I am working on the file "HW0_solutions.ipynb".
The first task is:
If that last cell complained about the %matplotlib line, you need to update IPython to v1.0, and restart the notebook. See the installation page
Hello Numpy
The Numpy array processing library is the basis of nearly all numerical computing in Python. Here's a 30 second crash course. For more details, consult Chapter 4 of Python for Data Analysis, or the Numpy User's Guide
Can you write Python code for it?

print "Make a 3 row x 4 column array of random numbers"
x = np.random.random((3, 4))
print x
print

print "Add 1 to every element"
x = x + 1
print x
print

print "Get the element at row 1, column 2"
print x[1, 2]
print

# The colon syntax is called "slicing" the array.
print "Get the first row"
print x[0, :]
print

print "Get every 2nd column of the first row"
print x[0, ::2]
print


I am working on the file "HW0_solutions.ipynb".
The first task is:
Next, write a function that simulates the contestant's guesses for nsim simulations. Call this function simulate_guess. The specs:
Can you write Python code for it?

"""
Function
--------
simulate_guess

Return any strategy for guessing which door a prize is behind. This
could be a random strategy, one that always guesses 2, whatever.

Parameters
----------
nsim : int
    The number of simulations to generate guesses for

Returns
-------
guesses : array
    An array of guesses. Each guess is a 0, 1, or 2

Example
-------
>>> print simulate_guess(5)
array([0, 0, 0, 0, 0])
"""
#your code here

def simulate_guess(nsim):
    return np.zeros(nsim, dtype=np.int)


I am working on the file "HW0_solutions.ipynb".
The first task is:
Next, write a function, goat_door, to simulate randomly revealing one of the goat doors that a contestant didn't pick.
Can you write Python code for it?

"""
Function
--------
goat_door

Simulate the opening of a "goat door" that doesn't contain the prize,
and is different from the contestants guess

Parameters
----------
prizedoors : array
    The door that the prize is behind in each simulation
guesses : array
    THe door that the contestant guessed in each simulation

Returns
-------
goats : array
    The goat door that is opened for each simulation. Each item is 0, 1, or 2, and is different
    from both prizedoors and guesses

Examples
--------
>>> print goat_door(np.array([0, 1, 2]), np.array([1, 1, 1]))
>>> array([2, 2, 0])
"""
#your code here

def goat_door(prizedoors, guesses):

    #strategy: generate random answers, and
    #keep updating until they satisfy the rule
    #that they aren't a prizedoor or a guess
    result = np.random.randint(0, 3, prizedoors.size)
    while True:
        bad = (result == prizedoors) | (result == guesses)
        if not bad.any():
            return result
        result[bad] = np.random.randint(0, 3, bad.sum())


I am working on the file "HW0_solutions.ipynb".
The first task is:
Now, put it together. Simulate 10000 games where contestant keeps his original guess, and 10000 games where the contestant switches his door after a  goat door is revealed. Compute the percentage of time the contestant wins under either strategy. Is one strategy better than the other?
Can you write Python code for it?

#your code here

nsim = 10000

#keep guesses
print "Win percentage when keeping original door"
print win_percentage(simulate_prizedoor(nsim), simulate_guess(nsim))

#switch
pd = simulate_prizedoor(nsim)
guess = simulate_guess(nsim)
goats = goat_door(pd, guess)
guess = switch_guess(guess, goats)
print "Win percentage when switching doors"
print win_percentage(pd, guess).mean()


I am working on the file "HW0_solutions.ipynb".
The first task is:
Print the maximum, minimum, and mean of the array. This does not require writing a loop. In the code cell below, type x.m<TAB>, to find built-in operations for common array statistics like this
Can you write Python code for it?

#your code here
print "Max is  ", x.max()
print "Min is  ", x.min()
print "Mean is ", x.mean()


I am working on the file "HW0_solutions.ipynb".
The first task is:
Python Libraries
We will be using a several different libraries throughout this course. If you've successfully completed the installation instructions, all of the following statements should run.
Can you write Python code for it?

#IPython is what you are using now to run the notebook
import IPython
print "IPython version:      %6.6s (need at least 1.0)" % IPython.__version__

# Numpy is a library for working with Arrays
import numpy as np
print "Numpy version:        %6.6s (need at least 1.7.1)" % np.__version__

# SciPy implements many different numerical algorithms
import scipy as sp
print "SciPy version:        %6.6s (need at least 0.12.0)" % sp.__version__

# Pandas makes working with data tables easier
import pandas as pd
print "Pandas version:       %6.6s (need at least 0.11.0)" % pd.__version__

# Module for plotting
import matplotlib
print "Mapltolib version:    %6.6s (need at least 1.2.1)" % matplotlib.__version__

# SciKit Learn implements several Machine Learning algorithms
import sklearn
print "Scikit-Learn version: %6.6s (need at least 0.13.1)" % sklearn.__version__

# Requests is a library for getting data from the Web
import requests
print "requests version:     %6.6s (need at least 1.2.3)" % requests.__version__

# Networkx is a library for working with networks
import networkx as nx
print "NetworkX version:     %6.6s (need at least 1.7)" % nx.__version__

#BeautifulSoup is a library to parse HTML and XML documents
import BeautifulSoup
print "BeautifulSoup version:%6.6s (need at least 3.2)" % BeautifulSoup.__version__

#MrJob is a library to run map reduce jobs on Amazon's computers
import mrjob
print "Mr Job version:       %6.6s (need at least 0.4)" % mrjob.__version__

#Pattern has lots of tools for working with data from the internet
import pattern
print "Pattern version:      %6.6s (need at least 2.6)" % pattern.__version__


I am working on the file "HW0_solutions.ipynb".
The first task is:
The Monty Hall Problem
Here's a fun and perhaps surprising statistical riddle, and a good way to get some practice writing python functions
In a gameshow, contestants try to guess which of 3 closed doors contain a cash prize (goats are behind the other two doors). Of course, the odds of choosing the correct door are 1 in 3. As a twist, the host of the show occasionally opens a door after a contestant makes his or her choice. This door is always one of the two the contestant did not pick, and is also always one of the goat doors (note that it is always possible to do this, since there are two goat doors). At this point, the contestant has the option of keeping his or her original choice, or swtiching to the other unopened door. The question is: is there any benefit to switching doors? The answer surprises many people who haven't heard the question before.
We can answer the problem by running simulations in Python. We'll do it in several parts.
First, write a function called simulate_prizedoor. This function will simulate the location of the prize in many games -- see the detailed specification below:
Can you write Python code for it?

"""
Function
--------
simulate_prizedoor

Generate a random array of 0s, 1s, and 2s, representing
hiding a prize between door 0, door 1, and door 2

Parameters
----------
nsim : int
    The number of simulations to run

Returns
-------
sims : array
    Random array of 0s, 1s, and 2s

Example
-------
>>> print simulate_prizedoor(3)
array([0, 0, 2])
"""
def simulate_prizedoor(nsim):
    #compute here
    return answer
#your code here

def simulate_prizedoor(nsim):
    return np.random.randint(0, 3, (nsim))


I am working on the file "HW0_solutions.ipynb".
The first task is:
Write a function, switch_guess, that represents the strategy of always switching a guess after the goat door is opened.
Can you write Python code for it?

"""
Function
--------
switch_guess

The strategy that always switches a guess after the goat door is opened

Parameters
----------
guesses : array
     Array of original guesses, for each simulation
goatdoors : array
     Array of revealed goat doors for each simulation

Returns
-------
The new door after switching. Should be different from both guesses and goatdoors

Examples
--------
>>> print switch_guess(np.array([0, 1, 2]), np.array([1, 2, 1]))
>>> array([2, 0, 0])
"""
#your code here

def switch_guess(guesses, goatdoors):
    result = np.zeros(guesses.size)
    switch = {(0, 1): 2, (0, 2): 1, (1, 0): 2, (1, 2): 1, (2, 0): 1, (2, 1): 0}
    for i in [0, 1, 2]:
        for j in [0, 1, 2]:
            mask = (guesses == i) & (goatdoors == j)
            if not mask.any():
                continue
            result = np.where(mask, np.ones_like(result) * switch[(i, j)], result)
    return result


I am working on the file "HW1.ipynb".
The first task is:
Briefly summarize these graphs -- how accurate is the typical poll a day before the election? How often does a prediction one month before the election mispredict the actual winner?
Your summary here
Generally, these polls seem to be within about 5% of the election outcome. The accuracy is higher shortly before the election. (within 2% or so). Only for about 20% of the time, a poll one month before the election mispredict the actual winner.

Part 3: Analysis
Problem 5
You are (finally!) in a position to do some quantitative analysis.
We have provided an error_data function that builds upon the functions you have written. It computes a new DataFrame with information about polling errors.
Use error_data, find_governer_races, and pd.concat to construct a Data Frame summarizing the forecast errors
from all the Governor races
Hint
It's best to set ignore_index=True in pd.concat
Can you write Python code for it?

def party_from_color(color):
    if color in ['#0000CC', '#3B5998']:
        return 'democrat'
    if color in ['#FF0000', '#D30015']:
        return 'republican'
    return 'other'

def error_data(url):
    """
    Given a Governor race URL, download the poll data and race result,
    and construct a DataFrame with the following columns:

    candidate: Name of the candidate
    forecast_length: Number of days before the election
    percentage: The percent of poll votes a candidate has.
                Normalized to that the canddidate percentages add to 100%
    error: Difference between percentage and actual race reulst
    party: Political party of the candidate

    The data are resampled as necessary, to provide one data point per day
    """

    id = id_from_url(url)
    xml = get_poll_xml(id)

    colors = plot_colors(xml)
    if len(colors) == 0:
        return pd.DataFrame()

    df = rcp_poll_data(xml)
    result = race_result(url)

    #remove non-letter characters from columns
    df = df.rename(columns={c: _strip(c) for c in df.columns})
    for k, v in result.items():
        result[_strip(k)] = v

    candidates = [c for c in df.columns if c is not 'date']

    #turn into a timeseries...
    df.index = df.date

    #...so that we can resample at regular, daily intervals
    df = df.resample('D').mean()
    df = df.dropna()

    #compute forecast length in days
    #(assuming that last forecast happens on the day of the election, for simplicity)
    forecast_length = (df.index.max() - df.index).values

    forecast_length = forecast_length / np.timedelta64(1, 'D')  # convert to number of days

    errors = {}
    normalized = {}
    poll_lead = {}

    for c in candidates:
        #turn raw percentage into percentage of poll votes
        corr = df[c].values / df[candidates].sum(axis=1).values * 100.
        err = corr - result[_strip(c)]

        normalized[c] = corr
        errors[c] = err

    n =...


I am working on the file "HW1.ipynb".
The first task is:
Here are some other functions we'll use later. plot_colors contains hints about parsing XML data.
Can you write Python code for it?

# "r"egular "e"xpressions is kind of a mini-language to
# do pattern matching on text
import re

def _strip(s):
    """This function removes non-letter characters from a word

    for example _strip('Hi there!') == 'Hi there'
    """
    return re.sub(r'[\W_]+', '', s)

def plot_colors(xml):
    """
    Given an XML document like the link above, returns a python dictionary
    that maps a graph title to a graph color.

    Both the title and color are parsed from attributes of the <graph> tag:
    <graph title="the title", color="#ff0000"> -> {'the title': '#ff0000'}

    These colors are in "hex string" format. This page explains them:
    http://coding.smashingmagazine.com/2012/10/04/the-code-side-of-color/

    Example
    -------
    >>> plot_colors(get_poll_xml(1044))
    {u'Approve': u'#000000', u'Disapprove': u'#FF0000'}
    """
    dom = web.Element(xml)
    result = {}
    for graph in dom.by_tag('graph'):
        title = _strip(graph.attributes['title'])
        result[title] = graph.attributes['color']
    return result

_strip('Hi there!')
plot_colors(get_poll_xml(1044))


I am working on the file "HW1.ipynb".
The first task is:
Here's a histogram of the error of every polling measurement in the data
Can you write Python code for it?

errors.error.hist(bins=50)
plt.xlabel("Polling Error")
plt.ylabel('N')


I am working on the file "HW1.ipynb".
The first task is:
If you've done everything right so far, the following code should reproduce the graph on this page
Can you write Python code for it?

poll_plot(1044)
plt.title("Obama Job Approval")


I am working on the file "HW1.ipynb".
The first task is:
Introduction
This was the XKCD comic after the 2012 Presidential election:

The comic refers to the fact that Nate Silver's statistical model (which is based mostly on combining information from pre-election polls) correctly predicted the outcome of the 2012 presidential race in all 50 states.
Polling data isn't a perfect predictor for the future, and some polls are more accurate than others. This means that election forecastors must consider prediction uncertainty when building models.
In this first assignment, you will perform a simple analysis of polling data about the upcoming Governor races. The assignment has three main parts:
First you will build some tools to download historical polling data from the web, and parse it into a more convenient format.
Next you will use these tools to aggregate and visualize several past Governor races
Finally you will run a bootstrap analysis to estimate the probable outcome of current Governor races, given the level of precision of historical polls.

Part 1: Collect and Clean
The Real Clear Politics website archives many political polls. In addition, they combine related polls to form an "RCP average" estimate of public opinion over time. For example, the chart on this page shows historical polling data for the Obama-Romney presidential race. The chart is an average of the polling data table below the chart.
The data used to generate plots like this are stored as XML pages, with URLs like:
http://charts.realclearpolitics.com/charts/[id].xml
Here, [id] is a unique integer, found at the end of the URL of the page that displays the graph. The id for the Obama-Romney race is 1171:
http://charts.realclearpolitics.com/charts/1171.xml
Opening this page in Google Chrome or Firefox will show you the XML content in an easy-to-read format. Notice that XML tags are nested inside each other, hierarchically (the jargony term for this is the "Document Object Model", or...
Can you write Python code for it?

"""
Function
--------
get_poll_xml

Given a poll_id, return the XML data as a text string

Inputs
------
poll_id : int
    The ID of the poll to fetch

Returns
-------
xml : str
    The text of the XML page for that poll_id

Example
-------
>>> get_poll_xml(1044)
u'<?xml version="1.0" encoding="UTF-8"?><chart><series><value xid=\'0\'>1/27/2009</value>
...etc...
"""
#your code here
def get_poll_xml(poll_id):
    text = requests.get("http://charts.realclearpolitics.com/charts/%s.xml" %poll_id).text
    return text
get_poll_xml(1044)


I am working on the file "HW1.ipynb".
The first task is:
Now that this is done, we can easily visualize many historical Governer races. The solid line plots the poll history, the dotted line reports the actual result.
If this code block fails, you probably have a bug in one of your functions.
Can you write Python code for it?

#page = requests.get('http://www.realclearpolitics.com/epolls/2010/governor/2010_elections_governor_map.html').text.encode('ascii', 'ignore')
page = 'http://www.realclearpolitics.com/epolls/2010/governor/2010_elections_governor_map.html'
for race in find_governor_races(page):
    plot_race(race)
    plt.show()


I am working on the file "HW1.ipynb".
The first task is:
Part 2: Aggregate and Visualize
Problem 3
Unfortunately, these data don't have any error bars. If a candidate leads by 10% in the RCP average, is she a shoo-in to win? Or is this number too close to call? Does a 10% poll lead mean more 1 day before a race than it does 1 week before? Without error estimates, these questions are impossible to answer.
To get a sense of how accurate the RCP polls are, you will gather data from many previous Governor races, where the outcome is known.
This url has links to many governer races.
http://www.realclearpolitics.com/epolls/2010/governor/2010_elections_governor_map.html
Notice that each link to a governor race has the following URL pattern:
http://www.realclearpolitics.com/epolls/[YEAR]/governor/[STATE]/[TITLE]-[ID].html
Write a function that scans html for links to URLs like this
Hint The fnmatch function is useful for simple string matching tasks.
Can you write Python code for it?

"""
    Function
    --------
    find_governor_races

    Find and return links to RCP races on a page like
    http://www.realclearpolitics.com/epolls/2010/governor/2010_elections_governor_map.html

    Parameters
    ----------
    html : str
        The HTML content of a page to scan

    Returns
    -------
    A list of urls for Governer race pages

    Example
    -------
    For a page like

    find_governor_races would return
    ['http://www.realclearpolitics.com/epolls/2010/governor/ma/massachusetts_governor_baker_vs_patrick_vs_cahill-1154.html',
     'http://www.realclearpolitics.com/epolls/2010/governor/ca/california_governor_whitman_vs_brown-1113.html']
"""
#your code here
def is_gov_race(l):
    """return True if a URL refers to a Governor race"""
    pattern = 'http://www.realclearpolitics.com/epolls/????/governor/??/*-*.html'
    return fnmatch(l, pattern)

url1 = 'http://www.realclearpolitics.com/epolls/2010/governor/2010_elections_governor_map.html'

def find_governor_races(html):
    #tree = bs(get_html(url1))

    tree = bs(urllib2.urlopen(html).read())
    links =[]

    selects = tree.find_all(class_ ="search_by_race")
    for select in selects:
        for option in select.find_all('option'):
            links.append(str(option.get('value')))

    links = [l for l in links if is_gov_race(l)]
    #eliminate duplicates!
    links = list(set(links))
    return  links


I am working on the file "HW1.ipynb".
The first task is:
Problem 2
Even though get_poll_xml pulls data from the web into Python, it does so as a block of text. This still isn't very useful. Use the web module in pattern to parse this text, and extract data into a pandas DataFrame.
Hints

You might want create python lists for each column in the XML. Then, to turn these lists into a DataFrame, run

pd.DataFrame({'column_label_1': list_1, 'column_label_2':list_2, ...})

use the pandas function pd.to_datetime to convert strings into dates
Can you write Python code for it?

"""
    Function
    ---------
    rcp_poll_data

    Extract poll information from an XML string, and convert to a DataFrame

    Parameters
    ----------
    xml : str
        A string, containing the XML data from a page like
        get_poll_xml(1044)

    Returns
    -------
    A pandas DataFrame with the following columns:
        date: The date for each entry
        title_n: The data value for the gid=n graph (take the column name from the `title` tag)

    This DataFrame should be sorted by date

    Example
    -------
    Consider the following simple xml page:

    <chart>
    <series>
    <value xid="0">1/27/2009</value>
    <value xid="1">1/28/2009</value>
    </series>
    <graphs>
    <graph gid="1" color="#000000" balloon_color="#000000" title="Approve">
    <value xid="0">63.3</value>
    <value xid="1">63.3</value>
    </graph>
    <graph gid="2" color="#FF0000" balloon_color="#FF0000" title="Disapprove">
    <value xid="0">20.0</value>
    <value xid="1">20.0</value>
    </graph>
    </graphs>
    </chart>

    Given this string, rcp_poll_data should return
    result = pd.DataFrame({'date': pd.to_datetime(['1/27/2009', '1/28/2009']),
                           'Approve': [63.3, 63.3], 'Disapprove': [20.0, 20.0]})
"""
#your code here
def rcp_poll_data(xml):
    dom = web.Element(xml)

    result = {}
    dates = dom.by_tag('series')[0]
    dates = {date.attributes['xid']: str(date.content) for date in dates.by_tag('value')}

    keys = dates.keys();
    result['date'] = pd.to_datetime([dates[k] for k in keys])

    #retrieve graph attributes
    for graph in dom.by_tag('graph'):
        name = graph.attributes['title']
        data = {x.attributes['xid'] : float(x.content) if x.content else np.nan for x in graph.by_tag('value')}
        result[name] = [data[k] for k in keys]

    result = pd.DataFrame(result);
    result = result.sort_values(by=['date'])
    return result

rcp_poll_data(get_poll_xml(1044))

  ...


I am working on the file "HW1.ipynb".
The first task is:
Problem 4
At this point, you have functions to find a collection of governor races, download historical polling data from each one,
parse them into a numerical DataFrame, and plot this data.
The main question we have about these data are how accurately they predict election outcomes. To answer this question, we
need to grab the election outcome data.
Write a function that looks up and returns the election result on a page like this one.
Remember to look at the HTML source!
You can do this by selection view->developer->view source in Chrome, or Tools -> web developer -> page source in Firefox. Altenatively, you can right-click on a part of the page, and select "inspect element"
Can you write Python code for it?

"""
    Function
    --------
    race_result

    Return the actual voting results on a race page

    Parameters
    ----------
    url : string
        The website to search through

    Returns
    -------
    A dictionary whose keys are candidate names,
    and whose values is the percentage of votes they received.

    If necessary, normalize these numbers so that they add up to 100%.

    Example
    --------
    >>> url = 'http://www.realclearpolitics.com/epolls/2010/governor/ca/california_governor_whitman_vs_brown-1113.html'
    >>> race_result(url)
    {'Brown': 56.0126582278481, 'Whitman': 43.9873417721519}
"""
#your code here
url = 'http://www.realclearpolitics.com/epolls/2010/governor/ca/california_governor_whitman_vs_brown-1113.html'
""""
def race_result(url):

    result = {}
    soup = bs(urllib2.urlopen(url).read())
    rows = [row for row in soup.find("table", class_ ="data").find_all("tr")]
    name_columns = [col.get_text() for col in rows[0].find_all("th")]
    res_columns = [col.get_text() for col in rows[1].find_all("td")]

    res1 = float(res_columns[3])
    res2 = float(res_columns[4])
    sum =  res1 + res2
    tot = sum / 100
    result[name_columns[3]] = res1 / tot
    result[name_columns[4]] = res2 / tot

    return result
race_result(url)
"""
def race_result(url):

    dom = web.Element(requests.get(url).text)

    table = dom.by_tag('div#polling-data-rcp')[0]
    result_data = table.by_tag('tr.final')[0]
    td = result_data.by_tag('td')

    results = [float(t.content) for t in td[3:-1]]
    tot = sum(results) / 100

    #get table headers
    headers = table.by_tag('th')
    labels = [str(t.content).split('(')[0].strip() for t in headers[3:-1]]

    return {l:r / tot for l, r in zip(labels, results)}


I am working on the file "HW1.ipynb".
The first task is:
Problem 6
Compute the standard deviation of the polling errors. How much uncertainty is there in the typical RCP poll?
Can you write Python code for it?


I am working on the file "HW1.ipynb".
The first task is:
Q1. Write a program to print your personal info  as the following format:
Name: [Last Name] [First Name]
LSU email: xx@lsu.edu
Can you write Python code for it?

print("Name: Wang Bin")
print("LSU email: bxw1153@lsu.edu")


I am working on the file "HW1.ipynb".
The first task is:
Q2. Write ONE line code to print the following test content as the following format (hint: using \n \t):
[Last Name]  [First Name]
             Test2    Test3
Test4        Test5    Test6
Can you write Python code for it?


I am working on the file "HW1.ipynb".
The first task is:
Q3. Write a program to finish following task:
1: Define a variable of x = the number of letters in your name
e.g. ${x=15}$
2: Compute and print the solution of the following equation (hint: input, str(val))
$${{e}^{2}}+\frac{\sqrt{\ln 5+{{\log }_{10}}x}}{2}$$
e.g. The solution of the above equation with x of 15 is 55.432645265528734
Can you write Python code for it?

import numpy as np
x=15
print("The solution of the above equation with x of " + str(x) + " is " +str(np.exp(2)**2+np.sqrt(np.log(5)+np.log10(6))/2) )


I am working on the file "HW1.ipynb".
The first task is:
Q4.
1 Use array to define three array/list to store well name, daily oil production and location
|Well Name| Oil Pro (stb/day)|(Latitude,Longitude)|
|------|------|------|
|   LA12  | 334| (123.11,343.31)|
|   LA21  | 534| (713.61,546.11)|
|   LA44  | 274| (353.91,566.81)|
|   LA62  | 1334| (223.11,333.31)|
|   LA71  | 7834| (213.61,346.31)|
|   LA24  | 974| (823.91,796.21)|
|   LA46  | 8334| (333.11,503.33)|
|   LA78  | 6534| (163.61,596.17)|
|   LA84  | 4274| (353.93,586.86)|
2 Use the value from the defined array, write/using code to print out a SUMMARY REPORT of this oil field
2.3 The number of well in this oil field
2.2 Total daily oil production rate
2.3 The index, well name, oil production rate, and location in the above list of the nearest well to LA44
Summary Report Example
[Last Name] Oil Field
Number of Wells: 23
Total daily production rate: 33341 stb/day
The nearest well to LA44 is: [index] [Well Name] [Oil Production rate] [Location]
Can you write Python code for it?


I am working on the file "HW1.ipynb".
The first task is:
The above result  is in line with the conclusion from our visual inspection before. The polls shortly before the election(< 7 days) is more accurate than than polls more than a month before the election
Problem 8
Bootstrap resampling is a general purpose way to use empirical data like the errors DataFrame to estimate uncertainties. For example, consider the Viriginia Governor Race. If we wanted to estimate how likey it is that McAuliffe will win given the current RCP data, the approch would be:

Pick a large number N of experiments to run (say N=1000).
For each experiment, randomly select a value from errors.error. We are assuming that these numbers represent a reasonable error distribution for the current poll data.
Assume that the error on McAullife's current polling score is given by this number (and, by extension, the error on Cuccinelli's poll score is the opposite). Calculate who actually wins the election in this simulation.
Repeat N times, and calculate the percentage of simulations where either candidate wins.

Bootstrapping isn't foolproof: it makes the assumption that the previous Governor race errors are representative of the Virginia race, and it does a bad job at estimating very rare events (with only ~30 races in the errors DataFrame, it would be hard to accurately predict probabilities for 1-in-a-million scenarios). Nevertheless, it's a versatile technique.
Use bootstrap resampling to estimate how likely it is that each candidate could win the following races.

Virginia Governor
New Jersey Governor

Summarize your results in a paragraph. What conclusions do you draw from the bootstrap analysis, and what assumptions did you make in reaching this conclusion. What are some limitations of this analysis?
Can you write Python code for it?

#your code here
def bootstrap_result(s1, s2, errors, nsample = 10000):
    """
    Given the current polling data for 2 candidates, return the
    bootstrap-estimate for the win probability of each candidate

    Parameters
    ----------
    s1 : float
       The current proportion of poll votes for candidate 1
    s2 : float
       The current proportio of poll votes for candidate 2
    errors : DataFrame
       The errors DataFrame
    nsample : int
       The number of bootstrap iteraionts. Default=1000

    Returns
    -------
    p1, p2
    The probability that each candidate will win, based on the bootstrap simulations
    """

    #normalize s1, s2
    tot = (s1 + s2)
    s1 = 100 * s1 / tot
    s2 = 100 * s2 / tot

    indices = np.random.randint(0, errors.shape[0], nsample)
    errors =  errors.error.irow(indices).values

    s1_actual = s1 - errors
    s2_actual = s2 + errors

    p1 = (s1_actual > s2_actual).mean()
    p2 = 1 - p1

    return p1, p2

#Look up the data as of 10/03/2013
#virginia
nsample = 10000
mcauliffe, cuccinelli = 44.2, 39.3

pm, pc = bootstrap_result(mcauliffe, cuccinelli, errors, nsample=nsample)
print "Virginia Race"
print "-------------------------"
print "P(McAuliffe wins)  = %0.2f" % pm
print "P(Cuccinelli wins) = %0.2f" % pc

#new jersey
print "\n\n"
print "New Jersey Race"
print "-----------------------"
christie, buono = 57.2, 33
pc, pb = bootstrap_result(christie, buono, errors, nsample=nsample)
print "P(Christie wins) = %0.2f" % pc
print "P(Buono wins)    = %0.2f" % pb


I am working on the file "HW1.ipynb".
The first task is:
The output from rcp_poll_data is much more useful for analysis. For example, we can plot with it:
Can you write Python code for it?

def poll_plot(poll_id):
    """
    Make a plot of an RCP Poll over time

    Parameters
    ----------
    poll_id : int
        An RCP poll identifier
    """

    # hey, you wrote two of these functions. Thanks for that!
    xml = get_poll_xml(poll_id)
    data = rcp_poll_data(xml)
    colors = plot_colors(xml)

    #remove characters like apostrophes
    data = data.rename(columns = {c: _strip(c) for c in data.columns})

    #normalize poll numbers so they add to 100%
    norm = data[colors.keys()].sum(axis=1) / 100
    for c in colors.keys():
        data[c] /= norm

    for label, color in colors.items():
        plt.plot(data.date, data[label], color=color, label=label)

    plt.xticks(rotation=70)
    plt.legend(loc='best')
    plt.xlabel("Date")
    plt.ylabel("Normalized Poll Percentage")


I am working on the file "HW1/Q7_lasso/Q7-4-1_useful_features_regularization_path.ipynb".
The first task is:
In the Yelp Question in HW1, please normalize the data so that it has the same L2 norm. We will grade it either way, but please state clearly what you did to treat the yelp data, which is currently not normalized.
http://stackoverflow.com/questions/7140738/numpy-divide-along-axis
Can you write Python code for it?

#data_splits = (4000, 5000) # HW setting
data_splits = (2000, 2500)# faster setting

A_train = A[:data_splits[0], :]; y_train = y[:data_splits[0]]
A_val = A[data_splits[0]:data_splits[1], :]; y_val = y[data_splits[0]:data_splits[1]]
A_test = A[data_splits[1]:, :]; y_test = y[data_splits[1]:]

A_train.shape


I am working on the file "HW1/Q7_lasso/Q7-4-1_useful_features_regularization_path.ipynb".
The first task is:
Star data
Can you write Python code for it?

# Load a text file of integers:
y = np.loadtxt("yelp_data/star_labels.txt", dtype=np.int)
# Load a text file with strings identifying the 2500 features:
featureNames = open("yelp_data/star_features.txt").read().splitlines()
# Load a matrix market matrix with 45000 samples of 2500 features, convert it to csc format:
A = sp.csc_matrix(io.mmread("yelp_data/star_data.mtx"))


I am working on the file "HW1/Q7_lasso/Q7-4-1_useful_features_regularization_path.ipynb".
The first task is:
Train models for varying lambda values.  Calculate training error for each model.
Can you write Python code for it?

result = RegularizationPathTrainTest(X_train=A_train[0:100, 0:50], y_train=y_train[0:100], feature_names=featureNames, lam_max=1,
                                     X_val=A_val[0:100, 0:50], y_val=y_val[0:100,], steps=2, frac_decrease=0.05,
                                    delta = 0.001)

result.results_df

result.analyze_path()

result.results_df

result = RegularizationPathTrainTest(X_train=A_train, y_train=y_train, feature_names=featureNames, lam_max=100,
                                     X_val=A_val, y_val=y_val, steps=5, frac_decrease=0.7, delta=0.01)

result.analyze_path()

print(A_train.shape)
print(A_val.shape)
print(A_test.shape)

#Assuming df is a pandas data frame with columns 'x', 'y', and 'label'
fig, ax = plt.subplots(1, 1, figsize=(5, 4))
colors = {1: 'gray', 10: 'b'}
data = result.results_df.copy()
plt.semilogx(data['lam'], data['RMSE (validation)'], linestyle='--', marker='o', color='g')
plt.semilogx(data['lam'], data['RMSE (training)'], linestyle='--', marker='o', color='#D1D1D1')
#for key,grp in data.groupby('sigma'):
#    print (key)
#    plt.semilogx(grp.lam, grp.recall, linestyle='--', marker='o',
#             color=colors[key], label='sigma = {}'.format(key)) #, t, t**2, 'bs', t, t**3, 'g^')
plt.legend(loc = 'best')
plt.xlabel('lambda')
plt.ylabel('RMSE')
#ax.set_ylim([0.55, 1.05])

#Assuming df is a pandas data frame with columns 'x', 'y', and 'label'
fig, ax = plt.subplots(1, 1, figsize=(4,3))
colors = {1: 'gray', 10: 'b'}
data = result.results_df.copy()
plt.semilogx(data['lam'], data['# nonzero coefficients'], linestyle='--', marker='o', color='b')
#for key,grp in data.groupby('sigma'):
#    print (key)
#    plt.semilogx(grp.lam, grp.recall, linestyle='--', marker='o',
#             color=colors[key], label='sigma = {}'.format(key)) #, t, t**2, 'bs', t, t**3, 'g^')
plt.legend(loc = 'best')
plt.xlabel('lambda')
plt.ylabel('num nonzero coefficients')
#ax.set_ylim([0.55, 1.05])

assert False


I am working on the file "HW1/Q7_lasso/Q7-4-1_useful_features_regularization_path.ipynb".
The first task is:
Upvote data
Can you write Python code for it?

# Load a text file of integers:
y = np.loadtxt("yelp_data/upvote_labels.txt", dtype=np.int)
# Load a text file with strings identifying the 1000 features:
featureNames = open("yelp_data/upvote_features.txt").read().splitlines()
featureNames = np.array(featureNames)
# Load a csv of floats, which are the values of 1000 features (columns) for 6000 samples (rows):
A = np.genfromtxt("yelp_data/upvote_data.csv", delimiter=",")
norms = np.apply_along_axis(np.linalg.norm,0,A)
A = A / norms
# print(np.apply_along_axis(np.linalg.norm,0,A))

# Randomize input order
np.random.seed(12345)
shuffler = np.arange(len(y))
np.random.shuffle(shuffler)
A = A[shuffler,:]
y = y[shuffler]


I am working on the file "HW1_sd3462.ipynb".
The first task is:
1. Load the data set "data/ads_dataset.tsv" into a Python Pandas data frame called ads.
Can you write Python code for it?


I am working on the file "HW1_sd3462.ipynb".
The first task is:
2. How many unique users are in this file? (hint: consider the 'cut' command and use pipe operator '|')
Can you write Python code for it?

!cut Datasets/advertising_events.csv -d ',' -f 1 | sort | uniq | wc -l


I am working on the file "HW1_sd3462.ipynb".
The first task is:
3. Rank all domains by the number of visits they received in descending order. (hint: consider the 'cut', 'uniq' and 'sort' commands and the pipe operator).
Can you write Python code for it?

!cut Datasets/advertising_events.csv -d ',' -f 3 | sort | uniq -c | sort -nr


I am working on the file "HW1_sd3462.ipynb".
The first task is:
4. Using the results returned from getDfSummary(), which fields, if any, contain missing NaN values?
Can you write Python code for it?


I am working on the file "HW1_sd3462.ipynb".
The first task is:
5. For the fields with missing values, does it look like the data is missing at random? Are there any other fields that correlate perfectly, or predict that the data is missing? What would be an appropriate method for filling in missing values?
Hint: create another data frame that has just the records with a missing value. Get a summary of this data frame using getDfSummary() and compare the differences. Do some feature distributions change dramatically?
Can you write Python code for it?

#ads[result.loc[result['Number_NaN']>0].index]
#Analyse the results (returned by getDfSummary() function) to identify columns containing null values.
result[result.Number_NaN>0]

# Based on the results obtained from the above code, we can infer that only one column has NaN values.
#We use the dropna() function to remove the NaN values and use the getDfSummary function to see how different values change.
new=ads.dropna()
getDfSummary(new)

#Based on the results obtained by calling the getDfSummary function, we observe that the "isbuyer" column has only one distinct value which is 1
#It is safe to assume that when 'buy_freq' is not null, 'isbuyer' is always 1.

#We can check out how well these columns correlate by using the corr() function
new.isbuyer.corr(new['buy_freq'])

#Running the above command returns nan. Since we have removed all NaN values, we look at other factors resulting in this.
#We can observe that when 'buy_freq' is not null, the field of 'isbuyer' demonstrates 0 standard deviation.
#Due to this, the denominator of the correlation formula becomes 0 thus resulting in NaN output
#We can check corrlation between different columns as well using the below command
new.corr()

#Apart from 'isbuyer', we can observe that 'buy_freq' demonstrates strong correlation with 'multiple_buy'
new[['buy_freq','multiple_buy']]
#The above code informs that for 'buy_freq'>1, 'multiple_buy' is 1

#We can safely assume that when 'multiple_buy' is 0 and 'isbuyer' is 0, 'buy_freq' will be 0
##ads.loc[(ads['isbuyer']==0) & ads['multiple_buy']<1]=0
ads.loc[(ads['multiple_buy']<1) & (ads['isbuyer']==0),['buy_freq']]=0
ads.isnull().sum()
#We have elminated all NaN values by observing correlation values


I am working on the file "HW1_sd3462.ipynb".
The first task is:
6. Which variables are binary?
Can you write Python code for it?

result=getDfSummary(ads)
result.loc[(result.Number_Distinct==2)]

#We can use Number_Distinct as an indicator of binary variables since the value of the below four variables can either be 0 or 1.
#Number_Distinct of the below four variables is 2 and max and min is 1 and 0 respectively.


I am working on the file "HW1_sd3462.ipynb".
The first task is:
Part 3: Dealing with data Pythonically
Can you write Python code for it?

# You might find these packages useful. You may import any others you want!
import pandas as pd
import numpy as np


I am working on the file "HW2.ipynb".
The first task is:
1.11 Simulate elections assuming a bias of 1% and 5%, and plot histograms for each one.
Can you write Python code for it?

#your code here
model = biased_gallup_poll(gallup_2012, 1)
model = model.join(electoral_votes)
prediction = simulate_election(model, 10000)
plot_simulation(prediction)
plt.show()

model = biased_gallup_poll(gallup_2012, 5)
model = model.join(electoral_votes)
prediction = simulate_election(model, 10000)
plot_simulation(prediction)
plt.show()


I am working on the file "HW2.ipynb".
The first task is:
1.14 Finally, given that we know the actual outcome of the 2012 race, and what you saw from the 2008 race would you trust the results of the an election forecast based on the 2012 Gallup party affiliation poll?
Answer
This was a disaster. The 8% calibration completey destroys the accuracy of our prediction in 2012. Our calibration made the assumptions that a) the bias in 2008 was the same as 2012, and b) the bias in each state was the same.
There are several ways in which these assumptions may have been violated. Gallup may have changed their methodology to account for this bias already, leading to a different bias in 2012 from what there was in 2008. The state-by-state biases may have also been different -- voters in highly conservative states may have responded to polls differently from voters in libreral states, for instance. It might have been better to callibrate the bias on a state-wide or clustered basis.
Question 2: Logistic Considerations
In the previous forecast, we used the strategy of taking some side-information about an election (the partisan affiliation poll) and relating that to the predicted outcome of the election. We tied these two quantities together using a very simplistic assumption, namely that the vote outcome is deterministically related to estimated partisan affiliation.
In this section, we use a more sophisticated approach to link side information -- usually called features or predictors -- to our prediction. This approach has several advantages, including the fact that we may use multiple features to perform our predictions. Such data may include demographic data, exit poll data, and data from previous elections.
First, we'll construct a new feature called PVI, and use it and the Gallup poll to build predictions. Then, we'll use logistic regression to estimate win probabilities, and use these probabilities to build a prediction.
The Partisan Voting Index
The Partisan Voting Index (PVI) is defined as the excessive swing towards a party in the previous election in a given state. In...
Can you write Python code for it?


I am working on the file "HW2.ipynb".
The first task is:
1.3 Now, write a function called plot_simulation to visualize the simulation. This function should:

Build a histogram from the result of simulate_election
Overplot the "victory threshold" of 269 votes as a vertical black line (hint: use axvline)
Overplot the result (Obama winning 332 votes) as a vertical red line
Compute the number of votes at the 5th and 95th quantiles, and display the difference (this is an estimate of the outcome's uncertainty)
Display the probability of an Obama victory
Can you write Python code for it?

"""
Function
--------
plot_simulation

Inputs
------
simulation: Numpy array with n_sim (see simulate_election) elements
   Each element stores the number of electoral college votes Obama wins in each simulation.

Returns
-------
Nothing
"""

#your code here
def plot_simulation(simulation):
    plt.hist(simulation, bins=np.arange(200, 538, 1),
             label='simulations', align='left', normed=True)
    plt.axvline(332, 0, .5, color='r', label='Actual Outcome')
    plt.axvline(269, 0, .5, color='k', label='Victory Threshold')
    p05 = np.percentile(simulation, 5.)
    p95 = np.percentile(simulation, 95.)
    iq = int(p95 - p05)
    pwin = ((simulation >= 269).mean() * 100)
    plt.title("Chance of Obama Victory: %0.2f%%, Spread: %d votes" % (pwin, iq))
    plt.legend(frameon=False, loc='upper left')
    plt.xlabel("Obama Electoral College Votes")
    plt.ylabel("Probability")
    remove_border()


I am working on the file "HW2.ipynb".
The first task is:
1.7 Attempt to validate the predictive model using the above simulation histogram. Does the evidence contradict the predictive model?
Predictive model suggests zero probability for the actual outcome(in red). This contradicts the evidence and hence we should reject the preditive model.
Adding Polling Uncertainty to the Predictive Model
The model above is brittle -- it includes no accounting for uncertainty, and thus makes predictions with 100% confidence. This is clearly wrong -- there are numerous sources of uncertainty in estimating election outcomes from a poll of affiliations.
The most obvious source of error in the Gallup data is the finite sample size -- Gallup did not poll everybody in America, and thus the party affilitions are subject to sampling errors. How much uncertainty does this introduce?
On their webpage discussing these data, Gallup notes that the sampling error for the states is between 3 and 6%, with it being 3% for most states. (The calculation of the sampling error itself is an exercise in statistics. Its fun to think of how you could arrive at the sampling error if it was not given to you. One way to do it would be to assume this was a two-choice situation and use binomial sampling error for the non-unknown answers, and further model the error for those who answered 'Unknown'.)
1.8 Use Gallup's estimate of 3% to build a Gallup model with some uncertainty. Assume that the Dem_Adv column represents the mean of a Gaussian, whose standard deviation is 3%. Build the model in the function uncertain_gallup_model. Return a forecast where the probability of an Obama victory is given by the probability that a sample from the Dem_Adv Gaussian is positive.
Hint
The probability that a sample from a Gaussian with mean $\mu$ and standard deviation $\sigma$ exceeds a threhold $z$ can be found using the the Cumulative Distribution Function of a Gaussian:
$$
CDF(z) = \frac1{2}\left(1 + {\rm erf}\left(\frac{z - \mu}{\sqrt{2 \sigma^2}}\right)\right)
$$
Can you write Python code for it?

"""
Function
--------
uncertain_gallup_model

A forecast that predicts an Obama (Democratic) victory if the random variable drawn
from a Gaussian with mean Dem_Adv and standard deviation 3% is >0

Inputs
------
gallup : DataFrame
    The Gallup dataframe above

Returns
-------
model : DataFrame
    A dataframe with the following column
     * Obama: probability that the state votes for Obama.
    model.index should be set to gallup.index (that is, it should be indexed by state name)
"""
# your code here
from scipy.special import erf
def uncertain_gallup_model(gallup):
    sigma = 3.0
    prob = 0.5*(1 + erf(gallup.Dem_Adv/np.sqrt(2*sigma**2)))
    return pd.DataFrame(dict(Obama= prob), index =gallup.index)


I am working on the file "HW2.ipynb".
The first task is:
1.9  Attempt to validate the above model using the histogram. Does the predictive distribution appear to be consistent with the real data? Comment on the accuracy and precision of the prediction.
Your answers here
Biases
While accounting for uncertainty is one important part of making predictions, we also want to avoid systematic errors. We call systematic over- or under-estimation of an unknown quantity bias. In the case of this forecast, our predictions would be biased if the estimates from this poll systematically over- or under-estimate vote proportions on election day. There are several reasons this might happen:

Gallup is wrong. The poll may systematically over- or under-estimate party affiliation. This could happen if the people who answer Gallup phone interviews might not be a representative sample of people who actually vote, Gallup's methodology is flawed, or if people lie during a Gallup poll.
Our assumption about party affiliation is wrong. Party affiliation may systematically over- or under-estimate vote proportions. This could happen if people identify with one party, but strongly prefer the candidate from the other party, or if undecided voters do not end up splitting evenly between Democrats and Republicans on election day.
Our assumption about equilibrium is wrong. This poll was released in August, with more than two months left for the elections. If there is a trend in the way people change their affiliations during this time period (for example, because one candidate is much worse at televised debates), an estimate in August could systematically miss the true value in November.

One way to account for bias is to calibrate our model by estimating the bias and adjusting for it. Before we do this, let's explore how sensitive our prediction is to bias.
1.10 Implement a biased_gallup forecast, which assumes the vote share for the Democrat on election day will be equal to Dem_Adv shifted by a fixed negative amount. We will call this shift the "bias", so a bias of 1% means that the expected vote...
Can you write Python code for it?

"""
Function
--------
biased_gallup_poll

Subtracts a fixed amount from Dem_Adv, beofore computing the uncertain_gallup_model.
This simulates correcting a hypothetical bias towards Democrats
in the original Gallup data.

Inputs
-------
gallup : DataFrame
    The Gallup party affiliation data frame above
bias : float
    The amount by which to shift each prediction

Examples
--------
>>> model = biased_gallup(gallup, 1.)
>>> model.ix['Flordia']
>>> .460172
"""
#your code here
def biased_gallup_poll(gallup, bias):
    g2 = gallup.copy()
    g2.Dem_Adv -= bias
    return uncertain_gallup_model(g2)


I am working on the file "HW2.ipynb".
The first task is:
2.1 Build a new DataFrame called e2008. The dataframe e2008 must have the following columns:

a column named pvi with the contents of the partisan vote index pvi08
a column named Dem_Adv which has the Democratic advantage from the frame prediction_08 of the last question with the mean subtracted out
a column named obama_win which has a 1 for each state Obama won in 2008, and 0 otherwise
a column named Dem_Win which has the 2008 election Obama percentage  minus McCain percentage, also from the frame  prediction_08
The DataFrame should be indexed and sorted by State
Can you write Python code for it?

#your code here
e2008 = pd.DataFrame(dict(pvi=pvi08,Dem_Adv = (prediction_08.Dem_Adv - prediction_08.Dem_Adv.mean()),
        obama_win = (prediction_08.Dem_Win > 0).astype(float), Dem_Win= prediction_08.Dem_Win))
e2008.head()


I am working on the file "HW2.ipynb".
The first task is:
2.4 *Carry out a logistic fit using the cv_and_fit function developed above. As your featurelist use the features we have: Dem_Adv and pvi.
Can you write Python code for it?

#your code here

res, clf = cv_and_fit(e2008, e2012, ['Dem_Adv', 'pvi'])
predict2012_logistic = res.join(electoral_votes)
predict2012_logistic.head()


I am working on the file "HW2.ipynb".
The first task is:
2.5 As before, plot a histogram and map of the simulation results, and interpret the results in terms of accuracy and precision.
Can you write Python code for it?

#code to make the histogram
#your code here

prediction_2012 = simulate_election(predict2012_logistic, 10000)
plot_simulation(prediction_2012)
plt.xlim(200,450)

#code to make the map
#your code here
make_map(predict2012_logistic.Obama, "P(Obama): Logistic")
plt.show()


I am working on the file "HW2.ipynb".
The first task is:
3.1 Using this data, compute a new data frame that averages the obama_spread for each state. Also compute the standard deviation of the obama_spread in each state, and the number of polls for each state.
Define a function state_average which returns this dataframe
Hint
pd.GroupBy could come in handy
Can you write Python code for it?

groups = multipoll.groupby('State')
mean = groups.mean()['obama_spread']
std = groups.std()['obama_spread']
size = groups.count()['Pollster']
print std[std.isnull()]
std[std.isnull()] = 0.05 * mean[std.isnull()]

averages = pd.DataFrame(dict(N = size, poll_mean = mean, poll_std = std))
averages

"""
Function
--------
state_average

Inputs
------
multipoll : DataFrame
   The multipoll data above

Returns
-------
averages : DataFrame
  A dataframe, indexed by State, with the following columns:
     N: Number of polls averaged together
     poll_mean: The average value for obama_spread for all polls in this state
     poll_std: The standard deviation of obama_spread

Notes
-----
For states where poll_std isn't finite (because N is too small), estimate the
poll_std value as .05 * poll_mean
"""
#your code here
def state_average(multipoll):
    groups = multipoll.groupby('State')
    mean = groups.mean()['obama_spread']
    std = groups.std()['obama_spread']
    size = groups.count()['Pollster']
    std[std.isnull()] = 0.5 * mean[std.isnull()]

    averages = pd.DataFrame(dict(N = size, poll_mean = mean, poll_std = std))
    return averages


I am working on the file "HW2.ipynb".
The first task is:
3.5 Put this all together -- compute a new estimate of poll_mean and poll_std for each state, apply the default_missing function to handle missing rows, build a forecast with aggregated_poll_model, run 10,000 simulations, and plot the results, both as a histogram and as a map.
Can you write Python code for it?

#your code here
average = weighted_state_average(multipoll).join(electoral_votes, how='outer')
default_missing(average)
model= aggregated_poll_model(average)
predict_poll_weighted = simulate_election(model, 10000)
plot_simulation(predict_poll_weighted)
plt.xlim(250, 400)

#your map code here
make_map(model.Obama, "P(Obama): Weighted Polls")