I am working on the file "Classification2.ipynb".
The first task is:
Splitting Criteria
Now let's talk about the best ways to split the data.  This is actually a really hard problem, which you can read about more in Ivezic $\S$ 9.7.1.
One way is to use the information content or entropy, $E(x)$, of the data
$$ E(x) = -\sum_i p_i(x) \ln (p_i(x)),$$
where $i$ is the class and $p_i(x)$ is the probability of that class
given the training data.
Another commonly used "loss function" (especially for categorical classification) is the Gini coefficient:
$$ G = \sum_i^k p_i(1-p_i).$$
It essentially estimates the probability of incorrect classification by choosing both a point and (separately) a class randomly from the data.
Obviously in constructing a decision treee, if your choice of stopping criteria is too loose, further splitting just ends up adding noise.  So here is an example using cross-validation in order to optimize the depth of the tree (and to avoid overfitting).
Note that here we aren't classifying the objects into discrete categories, rather we are classifying them into a continuous category.  That is, we are doing regression.  In this particular case, we are using the colors of galaxies in order to predict their redshifts (distances).
Can you write Python code for it?

# Ivezic, Figure 9.14
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt

from sklearn.tree import DecisionTreeRegressor
from astroML.datasets import fetch_sdss_specgals

#------------------------------------------------------------
# Fetch data and prepare it for the computation
data = fetch_sdss_specgals()

# put magnitudes in a matrix
mag = np.vstack([data['modelMag_%s' % f] for f in 'ugriz']).T
z = data['z']

# train on ~60,000 points
mag_train = mag[::10]
z_train = z[::10]

# test on ~6,000 separate points
mag_test = mag[1::100]
z_test = z[1::100]

#------------------------------------------------------------
# Compute the cross-validation scores for several tree depths
depth = np.arange(1, 21)
rms_test = np.zeros(len(depth))
rms_train = np.zeros(len(depth))
i_best = 0
z_fit_best = None

for i, d in enumerate(depth):
    clf = DecisionTreeRegressor(max_depth=d, random_state=0)
    clf.fit(mag_train, z_train)

    z_fit_train = clf.predict(mag_train)
    z_fit_test = clf.predict(mag_test)
    rms_train[i] = np.mean(np.sqrt((z_fit_train - z_train) ** 2))
    rms_test[i] = np.mean(np.sqrt((z_fit_test - z_test) ** 2))

    if rms_test[i] <= rms_test[i_best]:
        i_best = i
        z_fit_best = z_fit_test

best_depth = depth[i_best]

#------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(10, 5))
fig.subplots_adjust(wspace=0.25,
                    left=0.1, right=0.95,
                    bottom=0.15, top=0.9)

# first panel: cross-validation
ax = fig.add_subplot(121)
ax.plot(depth, rms_test, '-k', label='cross-validation')
ax.plot(depth, rms_train, '--k', label='training set')
ax.set_xlabel('depth of...


I am working on the file "Classification2.ipynb".
The first task is:
That's doing the Cross Validation by hand, let's try it automatically using data like the first example that we started with today.
Can you write Python code for it?

X, y = make_blobs(n_samples=500, centers=3,
                  random_state=0, cluster_std=1.50)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='spring')

from sklearn.grid_search import GridSearchCV

clf = DecisionTreeClassifier()
drange = np.arange(1,21)

grid = GridSearchCV(clf, param_grid={'max_depth': drange}, cv=5)
grid.fit(X, y)

print("best parameter choice:", grid.best_params_)


I am working on the file "Classification2.ipynb".
The first task is:
The "leaf (terminal) nodes" record the fraction of points that have one classification or the other
Application of the tree to classification is simple (a series of binary decisions). The  fraction of points from the training set classified as one class or the other (in the leaf node) defines the class associated with that leaf node.
The binary splitting makes this extremely efficient. Tthe trick is to ask the right questions.
So, decision trees are simple to interpret (just a set of questions).
Scikit-learn implements the DecisionTreeClassifier as follows:
Can you write Python code for it?

import numpy as np
from sklearn.tree import DecisionTreeClassifier
X = np.random.random((100,2))
y = (X[:,0] + X[:,1] > 1).astype(int)
dtree = DecisionTreeClassifier(max_depth=6)
dtree.fit(X,y)
y_pred = dtree.predict(X)


I am working on the file "Classification_Turner.ipynb".
The first task is:
PART 1: Sorting Organisms by One Characteristic

We will be looking at the characterists of 75 unique organisms in our Dichotomous Key.  The imput below will provide us with a some of the possible organisms you may discover and the different Organism Characteristics/Conditions in our data set.
Can you write Python code for it?

# Animal options in Dichotomous Key
# Displays all row titles as an array
key.organism

# Conditions/Questions for finding the correct animal
# Displays all column titles as an array
key.columns


I am working on the file "Classification_Turner.ipynb".
The first task is:
PART 1: Sorting Organisms by One Characteristic
3.Organisms are classified by shared characteristics.  Is it possible for something to be eukaryotic and prokaryotic at the same time?  Why might this kind of trait be helpful for scientists?
4.How many different organisms in our list of 75 have wings?  Are they all similar?  Are “wings” a good characteristic to use for classification?
5.Which characteristic gave you the largest category?  Which gave you the smallest?  Why might this be the case?
PART 2: Sorting Organisms by Many Characteristics

These are the conditions or the characteristics in which ceratin answers are categorized for certain organisms. Each characteristic/condition has a yes/no except for the Kingdoms. Change the conditionals in the code below to change what organism(s) are displayed. For most, the only change needs to be the 'yes' or 'no'.

Capitalization matters so be careful. You also must put in only allowed answers in every condition or the code will break!

Use and modify the section of code below to answer questions 6-8 in your coding booklet.
Can you write Python code for it?

# This conditional allows us to query a column and if the data within that cell matches it will display the animal(s).
#if you are unsure of what to put try making that column a comment by adding # in front of it.

key[
#physical characteristics
    (key['fur'] == 'yes') & \
    (key['feathers'] == 'no') & \
    (key['poisonous'] == 'no') & \
    (key['scales'] == 'no') & \
    (key['multicellular'] == 'yes') & \
    (key['fins'] == 'no') & \
    (key['wings'] == 'no') & \
    (key['vertebrate'] == 'yes') & \

#environmental characteristics
    (key['marine'] == 'no') & \
    (key['terrestrial'] == 'yes') & \

#feeding characteristics

    #decomposers get their food by breaking down decaying organisms
    (key['decomposer'] == 'no') & \

    #carnivores get their food by eating animals
    (key['carnivore'] == 'no') & \

    #herbivores get their food by eating plants
    (key['herbivore'] == 'yes') & \

    #omnivores get their food by eating both plants and animals
    (key['omnivore'] == 'no') & \

    #photosynthesis is the process of making food using energy from sunlight
    (key['photosynthesis'] == 'no') & \

    #autotrophs are organisms that generate their own food inside themselves
    (key['autotroph'] == 'no') & \

#possible kingdoms include: animalia, plantae, fungi
    (key['kingdom'] == 'animalia') & \

#cell type
    (key['eukaryotic'] == 'yes') & \
    (key['prokaryotic'] == 'no')
    ]


I am working on the file "Classification_Turner.ipynb".
The first task is:
PART 2: Sorting Organisms by Many Characteristics
6.Set the list to the characteristics of a cow.  What are some other organisms that are sorted this way?  What new traits would make the cow the only result?
7.How would the list of characteristics differ between a whale and dolphin?
8.A zoologist is exploring the jungle when she spots a small, hairy animal.  As she follows the animal she sees it eat nuts from a tree and some insects off the ground.  She also observes that even though it appears to have wings, it seems to prefer to move along the ground.  According to your key, what organism is it most similar to, how do the observed characteristics differ from what you know about this animal?
Part 3 & 4: Scientific Classification of Organisms & Unstructured Coding

Use and modify the section of code below to answer questions 9-13 in your coding booklet.
Can you write Python code for it?

#sort your organisms by their taxonomical classification

# This conditional allows us to query a column and if the data within that cell matches,
# it will display the corresponding animal(s)
key2[(key2['kingdom'] == 'animalia')]


I am working on the file "Classification_Turner.ipynb".
The first task is:
PART 3: Scientific Classification of Organisms
9.Are all organisms in the same kingdom classified in the same phylum?
10.If organisms are in the same order (like rodentia), describe how their kingdom, phylum, and classes compare.  Explain.
PART 4: Unstructured Coding
11.What are two examples of organisms in kingdom plantae?
12.What are two examples of organisms in kingdom fungi?
13.Find a few poisonous animals by changing the conditional statements.
Can you write Python code for it?

#Done?? Insert a image for one of the organisms you found using the dichotomous key.
from IPython.display import Image
from IPython.core.display import HTML
Image(url= 'https://lms.mrc.ac.uk/wp-content/uploads/insert-pretty-picture-here1.jpg')


I am working on the file "Classification_With_NN_1.ipynb".
The first task is:
Let's start the training using GPU.
Can you write Python code for it?

!./caffe/build/tools/caffe train -solver digits_solver.prototxt 2> digits_train.log

!python ./caffe/tools/extra/parse_log.py digits_train.log .

import pandas as pd

# columns: NumIters,Seconds,TrainingLoss,LearningRate
df_train = pd.read_csv( 'digits_train.log.train' )
# columns: NumIters,Seconds,TestAccuracy,TestLoss
df_test = pd.read_csv( 'digits_train.log.test' )

fig, (ax1, ax2) = plt.subplots( 2, sharex=True, figsize=(10, 12) )

ax1.plot( df_train.NumIters, df_train.TrainingLoss, 'b-' )

ax1.set_xlabel( 'Iterations' )
ax1.set_ylabel( 'Loss' )

ax3 = ax1.twinx()
ax3.plot( df_train.NumIters, df_train.LearningRate, 'g-' )
ax3.set_ylabel( 'Learning Rate' )

ax2.plot( df_test.NumIters, df_test.TestAccuracy, 'r-' )
ax2.set_xlabel( 'Iterations' )
ax2.set_ylabel( 'Test Accuracy' )

plt.show()

print( 'Final accuracy: {0}'.format( df_test.tail(1).TestAccuracy.iloc[0] ) )


# and we decided to choose the best parameters
model_params = {
    'vgg16-keras':    [ {'C':0.0001} ],
    'vgg19-keras':    [ {'C':0.001}  ],
    'resnet50-keras': [ {'C':0.001}   ],
    'Inception_v3':   [ {'C':0.01}   ],
    'incv3-keras':    [ {'C':0.001}  ]
}

from sklearn.svm import LinearSVC

# C - chosen experimentally (see explanation below)
results = dict()

for model_name in model_params:
    print('model = ', model_name)
    X_training = data[model_name]['features_training']
    X_testing = data[model_name]['features_testing']
    print( 'X_training size = {}'.format(X_training.shape))
#     print( 'X_testing size = {}'.format(X_testing.shape))
#     print( 'y_training size = {}'.format(y_training.shape))
#     print( 'y_testing size = {}'.format(y_testing.shape))
    results[model_name] = []
    for params in model_params[model_name]:
        clf = LinearSVC(**params, verbose=0)
        clf.fit( X_training, y_training )
        y_pred = clf.predict( X_testing )
        score = sum( y_pred == y_testing )
        print('features={:>16}, C={:8f} => score={:5d}'.format(model_name,params['C'],score))
        results[model_name].append({'pred': y_pred, 'score': score, 'clf': clf})

from sklearn.externals import joblib

for model_name in model_params:
    joblib.dump(results[model_name][0]['clf'], \
                'classifiers/{score}-{name}.pkl'.format(score=results[model_name][0]['score'], name=model_name))

!ls -l classifiers/*.pkl

best_model = 'resnet50-keras'
X_training = data[best_model]['features_training']
X_testing  = data[best_model]['features_testing']

clf = results[best_model][0]['clf']
print( 'Best accuracy = {}'.format( clf.score( X_testing, y_testing ) ) )
y_predictions = clf.predict( X_testing )


I am working on the file "Classification_using_CNN_codes.ipynb".
The first task is:
Saving parameters
We simply save the matrix with weights and bias vector for linear classifier.
Can you write Python code for it?


I am working on the file "Classification_using_CNN_codes.ipynb".
The first task is:
So we obtained 91.58% accuracy on testing dataset using LinearSVC classifier on top of features extracted with ResNET50 convolutional neural network.
Some misclassifications
Can you write Python code for it?

import myutils
from sklearn.metrics import confusion_matrix
labels = myutils.load_CIFAR_classnames()
conf_matrix = confusion_matrix( y_testing, y_predictions )

print( 'Confusion matrix:\n', conf_matrix )
print( labels )

i,j = 3,0
img_idx = [ k for k in range(10000) if y_testing[k]==i and y_predictions[k]==j ]
print( 'We have, e.g., {c} {iname}s predicted to be {jname}'.format(\
       c=conf_matrix[i,j], iname=labels[i], jname=labels[j]) )
# print(img_idx)

_, data_testing = myutils.load_CIFAR_dataset(shuffle=False)
from matplotlib import pyplot as plt
%matplotlib inline

fig = plt.figure(figsize=(18,2));
for _i in range(conf_matrix[i,j]):
    a=fig.add_subplot(1,conf_matrix[i,j],_i+1)
    plt.imshow(data_testing[img_idx[_i]][0])
    plt.axis('off')


I am working on the file "Classification_using_CNN_codes.ipynb".
The first task is:
k-nearest neighbors classifier
Let us note that simple kNN classifier
(with k=10), trained with 5000 training features (CNN codes from Inception_v3) gives 83.45% accuracy on whole 10000 testing images.
Remark that computing predictions with this classifier is very complex and it is not recommended for classificcation of images.
Here is the code to compute the score on testing dataset.
```python
from sklearn.neighbors import KNeighborsClassifier
kNN_clf = KNeighborsClassifier(n_neighbors=10)
kNN_clf.fit(X_training, y_training)
print( 'Classification score = ', kNN_clf.score( X_testing, y_testing ) )
Classification score =  0.8345
```
Logistic regression
Finally we used <tt>Logistic regression</tt> with default parameters. We trained the model with all the training data and obtained 90.37% accuracy on testing dataset.
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression
clf = LogisticRegression()
clf.fit(X_training, y_training)
print( 'Linear regression accuracy = ', clf.score( X_testing, y_testing ) )


I am working on the file "Classification_using_HOG_features.ipynb".
The first task is:
Classification
We simply use LinearSVC from scikit-learn library.
Can you write Python code for it?

from sklearn.svm import LinearSVC

# parameter C chosen experimentally (see explanation below)
C = 1.0

clf = LinearSVC(C=C)


I am working on the file "Classification_using_HOG_features.ipynb".
The first task is:
Feature extraction
We use HOG descriptor from scikit-image library.
Can you write Python code for it?

# Configuring HOG descriptor
# see http://scikit-image.org/docs/dev/api/skimage.feature.html#skimage.feature.hog

# Configuration of HOG descriptor
normalize = True          #  True ==> yields a little bit better score
                          #
block_norm = 'L2-Hys'     # or 'L1'
orientations = 9          #
pixels_per_cell = [8, 8]  #  see section 'Additional remarks' for some explanation
cells_per_block = [2, 2]  #

def extractFeature(img, vis=False):
    from skimage.feature import hog
    return hog(img, orientations, pixels_per_cell, cells_per_block, block_norm, visualise=vis, transform_sqrt=normalize)


I am working on the file "Classification_using_HOG_features.ipynb".
The first task is:
Our results for different parameters
TODO:
* show misclassifications
Additional remarks
Tuning C parameter
Can you write Python code for it?

for C in [ 0.001, 0.01, 0.1, 1.0, 1.2, 1.5, 2.0, 10.0 ]:
    clf = LinearSVC(C=C)
    clf.fit(X_training, y_training)
    print( 'normalize={norm}, C={C}, score={score}'.format(norm=normalize, C=C, score=clf.score( X_testing, y_testing )) )


I am working on the file "Classification_using_HOG_features.ipynb".
The first task is:
PCA 2D decomposition
Can you write Python code for it?

from sklearn import decomposition
pca = decomposition.PCA(n_components=2)
pca.fit(X_training)
X = pca.transform(X_training)
print(pca.explained_variance_ratio_)

plt.figure( figsize=(15,15) )
plt.scatter( X[:, 0], X[:, 1], c=y_training, cmap='tab10' )
# plt.colorbar()
plt.show()

# TODO: remove outliers


I am working on the file "Classification_using_HOG_features.ipynb".
The first task is:
SVC(kernel=linear) and SVC()
Can you write Python code for it?

from sklearn.svm import SVC

svc_lin_clf = SVC(kernel='linear', C=1)
svc_lin_clf.fit(X_training, y_training)
svc_lin_clf.score(X_testing, y_testing)

from sklearn.svm import SVC
svc_clf = SVC(C=1)
svc_clf.fit(X_training, y_training)
svc_clf.score(X_testing, y_testing)


I am working on the file "Classification_using_HOG_features.ipynb".
The first task is:
Short look at the training data
Can you write Python code for it?

print( 'X_training shape is {}'.format( X_training.shape ) )
print( 'y_training shape is {}'.format( y_training.shape ) )
print( 'X_testing shape is {}'.format( X_testing.shape ) )
print( 'y_testing shape is {}'.format( y_testing.shape ) )

import pandas as pd
print( 'X_training data description')
pd.DataFrame( X_training ).describe()

print( 'y_training data description')
pd.DataFrame( y_training ).describe()


I am working on the file "Classification_using_HOG_features.ipynb".
The first task is:
Testing and score
We obtain 0.4914 score on CIFAR10 testing dataset.
Can you write Python code for it?


I am working on the file "Classification_using_HOG_features.ipynb".
The first task is:
Training
Can you write Python code for it?

# this may take some time
clf.fit(X_training, y_training)


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Build the word count vector for each review
Let us explore a specific example of a baby product.
Can you write Python code for it?


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Comparing models
We will now compare the accuracy of the sentiment_model and the simple_model using the get_classification_accuracy method you implemented above.
First, compute the classification accuracy of the sentiment_model on the train_data:
Can you write Python code for it?


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Data preperation
We will use a dataset consisting of baby product reviews on Amazon.com.
Can you write Python code for it?


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
For each review, we will use the word_count column and trim out all words that are not in the significant_words list above. We will use the SArray dictionary trim by keys functionality. Note that we are performing this on both the training and test set.
Can you write Python code for it?

train_data['word_count_subset'] = train_data['word_count'].dict_trim_by_keys(significant_words, exclude=False)
test_data['word_count_subset'] = test_data['word_count'].dict_trim_by_keys(significant_words, exclude=False)


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Let's sort the coefficients (in descending order) by the value to obtain the coefficients with the most positive effect on the sentiment.
Can you write Python code for it?


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Next, we will compute the classification accuracy of the simple_model on the test_data:
Can you write Python code for it?


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Now compute the accuracy of the majority class classifier on test_data.
Quiz Question: Enter the accuracy of the majority class classifier model on the test_data. Round your answer to two decimal places (e.g. 0.76).
Can you write Python code for it?

num_positive  = (test_data['sentiment'] == +1).sum()
num_negative = (test_data['sentiment'] == -1).sum()
print num_positive
print num_negative

num_positive * 1. / len(test_data)


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Now, compute the classification accuracy of the simple_model on the train_data:
Can you write Python code for it?


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Now, let us explore what the sample example above looks like after these 2 transformations. Here, each entry in the word_count column is a dictionary where the key is the word and the value is a count of the number of times the word occurs.
Can you write Python code for it?


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Now, let us see a preview of what the dataset looks like.
Can you write Python code for it?


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Now, we can see that the dataset contains an extra column called sentiment which is either positive (+1) or negative (-1).
Split data into training and test sets
Let's perform a train/test split with 80% of the data in the training set and 20% of the data in the test set. We use seed=1 so that everyone gets the same result.
Can you write Python code for it?

train_data, test_data = products.random_split(.8, seed=1)
print len(train_data)
print len(test_data)


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Now, we will assign reviews with a rating of 4 or higher to be positive reviews, while the ones with rating of 2 or lower are negative. For the sentiment column, we use +1 for the positive class label and -1 for the negative class label.
Can you write Python code for it?

products['sentiment'] = products['rating'].apply(lambda rating : +1 if rating > 3 else -1)
products


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Now, we will perform 2 simple data transformations:

Remove punctuation using Python's built-in string functionality.
Transform the reviews into word-counts.

Aside. In this notebook, we remove all punctuations for the sake of simplicity. A smarter approach to punctuations would preserve phrases such as "I'd", "would've", "hadn't" and so forth. See this page for an example of smart handling of punctuations.
Can you write Python code for it?

def remove_punctuation(text):
    import string
    return text.translate(None, string.punctuation)

review_without_puctuation = products['review'].apply(remove_punctuation)
products['word_count'] = graphlab.text_analytics.count_words(review_without_puctuation)


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Quiz Question: Are the positive words in the simple_model (let us call them positive_significant_words) also positive words in the sentiment_model?
Can you write Python code for it?

joined = coefficients_words.join(sentiment_model.coefficients, 'index')

print (joined['value']>0)
print (joined['value.1']>0)


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Quiz Question: Consider the coefficients of simple_model. There should be 21 of them, an intercept term + one for each word in significant_words. How many of the 20 coefficients (corresponding to the 20 significant_words and excluding the intercept term) are positive for the simple_model?
Can you write Python code for it?

coefficients = simple_model.coefficients

coefficients_words = coefficients[coefficients['name'] != '(intercept)']

(coefficients_words['value'] > 0).sum()


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Quiz Question: Of the three data points in sample_test_data, which one (first, second, or third) has the lowest probability of being classified as a positive review?
Find the most positive (and negative) review
We now turn to examining the full test dataset, test_data, and use GraphLab Create to form predictions on all of the test data points for faster performance.
Using the sentiment_model, find the 20 reviews in the entire test_data with the highest probability of being classified as a positive review. We refer to these as the "most positive reviews."
To calculate these top-20 reviews, use the following steps:
1.  Make probability predictions on test_data using the sentiment_model. (Hint: When you call .predict to make predictions on the test data, use option output_type='probability' to output the probability rather than just the most likely class.)
2.  Sort the data according to those predictions and pick the top 20. (Hint: You can use the .topk method on an SFrame to find the top k rows sorted according to the value of a specified column.)
Can you write Python code for it?

probabilities = sentiment_model.predict(test_data, output_type='probability')

test_data['prediction'] = probabilities

top20 = test_data.topk('prediction', 20)['name']

for product in ['Snuza Portable Baby Movement Monitor', 'MamaDoo Kids Foldable Play Yard Mattress Topper, Blue',
               'Britax Decathlon Convertible Car Seat, Tiffany', 'Safety 1st Exchangeable Tip 3 in 1 Thermometer']:
    print product, product in top20


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Quiz Question: What is the accuracy of the sentiment_model on the test_data? Round your answer to 2 decimal places (e.g. 0.76).
Quiz Question: Does a higher accuracy value on the training_data always imply that the classifier is better?
Learn another classifier with fewer words
There were a lot of words in the model we trained above. We will now train a simpler logistic regression model using only a subet of words that occur in the reviews. For this assignment, we selected a 20 words to work with. These are:
Can you write Python code for it?

significant_words = ['love', 'great', 'easy', 'old', 'little', 'perfect', 'loves',
      'well', 'able', 'car', 'broke', 'less', 'even', 'waste', 'disappointed',
      'work', 'product', 'money', 'would', 'return']

len(significant_words)


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Quiz Question: Which model (sentiment_model or simple_model) has higher accuracy on the TRAINING set?
Now, we will repeat this excercise on the test_data. Start by computing the classification accuracy of the sentiment_model on the test_data:
Can you write Python code for it?


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Quiz Question: Which of the following products are represented in the 20 most negative reviews?  [multiple choice]
Compute accuracy of the classifier
We will now evaluate the accuracy of the trained classifer. Recall that the accuracy is given by
$$
\mbox{accuracy} = \frac{\mbox{# correctly classified examples}}{\mbox{# total examples}}
$$
This can be computed as follows:

Step 1: Use the trained model to compute class predictions (Hint: Use the predict method)
Step 2: Count the number of data points when the predicted class labels match the ground truth labels (called true_labels below).
Step 3: Divide the total number of correct predictions by the total number of data points in the dataset.

Complete the function below to compute the classification accuracy:
Can you write Python code for it?

def get_classification_accuracy(model, data, true_labels):
    # First get the predictions
    ## YOUR CODE HERE
    predictions = model.predict(data)

    # Compute the number of correctly classified examples
    ## YOUR CODE HERE
    correct = (predictions == true_labels).sum()

    # Then compute accuracy by dividing num_correct by total number of examples
    ## YOUR CODE HERE
    accuracy = correct * 1. / len(data)

    return accuracy


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
Quiz Question: Which of the following products are represented in the 20 most positive reviews? [multiple choice]
Now, let us repeat this excercise to find the "most negative reviews." Use the prediction probabilities to find the  20 reviews in the test_data with the lowest probability of being classified as a positive review. Repeat the same steps above but make sure you sort in the opposite order.
Can you write Python code for it?

worst20 = test_data.sort('prediction')[0:20]['name']

for product in ['The First Years True Choice P400 Premium Digital Monitor, 2 Parent Unit',
                'JP Lizzy Chocolate Ice Classic Tote Set',
                'Peg-Perego Tatamia High Chair, White Latte',
                'Safety 1st High-Def Digital Monitor']:
    print product, product in worst20


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
That review seems pretty positive.
Now, let's see what the next row of the sample_test_data looks like. As we could guess from the sentiment (-1), the review is quite negative.
Can you write Python code for it?


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
The word_count column had been working with before looks like the following:
Can you write Python code for it?


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
There are a total of 121713 coefficients in the model. Recall from the lecture that positive weights $w_j$ correspond to weights that cause positive sentiment, while negative weights correspond to negative sentiment.
Fill in the following block of code to calculate how many weights are positive ( >= 0). (Hint: The 'value' column in SFrame weights must be positive ( >= 0)).
Can you write Python code for it?

num_positive_weights = len(weights[weights['value']>=0])
num_negative_weights = len(weights[weights['value']<0])

print "Number of positive weights: %s " % num_positive_weights
print "Number of negative weights: %s " % num_negative_weights


I am working on the file "Classifier_1_linear_regression.ipynb".
The first task is:
We can compute the classification accuracy using the get_classification_accuracy function you implemented earlier.
Can you write Python code for it?


I am working on the file "Classifier_2_logistic_regression.ipynb".
The first task is:
Aside. How the link function works with matrix algebra
Since the word counts are stored as columns in feature_matrix, each $i$-th row of the matrix corresponds to the feature vector $h(\mathbf{x}_i)$:
$$
[\text{feature_matrix}] =
\left[
\begin{array}{c}
h(\mathbf{x}_1)^T \
h(\mathbf{x}_2)^T \
\vdots \
h(\mathbf{x}_N)^T
\end{array}
\right] =
\left[
\begin{array}{cccc}
h_0(\mathbf{x}_1) & h_1(\mathbf{x}_1) & \cdots & h_D(\mathbf{x}_1) \
h_0(\mathbf{x}_2) & h_1(\mathbf{x}_2) & \cdots & h_D(\mathbf{x}_2) \
\vdots & \vdots & \ddots & \vdots \
h_0(\mathbf{x}_N) & h_1(\mathbf{x}_N) & \cdots & h_D(\mathbf{x}_N)
\end{array}
\right]
$$
By the rules of matrix multiplication, the score vector containing elements $\mathbf{w}^T h(\mathbf{x}_i)$ is obtained by multiplying feature_matrix and the coefficient vector $\mathbf{w}$.
$$
[\text{score}] =
[\text{feature_matrix}]\mathbf{w} =
\left[
\begin{array}{c}
h(\mathbf{x}_1)^T \
h(\mathbf{x}_2)^T \
\vdots \
h(\mathbf{x}_N)^T
\end{array}
\right]
\mathbf{w}
= \left[
\begin{array}{c}
h(\mathbf{x}_1)^T\mathbf{w} \
h(\mathbf{x}_2)^T\mathbf{w} \
\vdots \
h(\mathbf{x}_N)^T\mathbf{w}
\end{array}
\right]
= \left[
\begin{array}{c}
\mathbf{w}^T h(\mathbf{x}_1) \
\mathbf{w}^T h(\mathbf{x}_2) \
\vdots \
\mathbf{w}^T h(\mathbf{x}_N)
\end{array}
\right]
$$
Checkpoint
Just to make sure you are on the right track, we have provided a few examples. If your predict_probability function is implemented correctly, then the outputs will match:
Can you write Python code for it?

dummy_feature_matrix = np.array([[1.,2.,3.], [1.,-1.,-1]])
dummy_coefficients = np.array([1., 3., -1.])

correct_scores      = np.array( [ 1.*1. + 2.*3. + 3.*(-1.),          1.*1. + (-1.)*3. + (-1.)*(-1.) ] )
correct_predictions = np.array( [ 1./(1+np.exp(-correct_scores[0])), 1./(1+np.exp(-correct_scores[1])) ] )

print 'The following outputs must match '
print '------------------------------------------------'
print 'correct_predictions           =', correct_predictions
print 'output of predict_probability =', predict_probability(dummy_feature_matrix, dummy_coefficients)


I am working on the file "Classifier_2_logistic_regression.ipynb".
The first task is:
Estimating conditional probability with link function
Recall from lecture that the link function is given by:
$$
P(y_i = +1 | \mathbf{x}_i,\mathbf{w}) = \frac{1}{1 + \exp(-\mathbf{w}^T h(\mathbf{x}_i))},
$$
where the feature vector $h(\mathbf{x}_i)$ represents the word counts of important_words in the review  $\mathbf{x}_i$. Complete the following function that implements the link function:
Can you write Python code for it?

'''
produces probablistic estimate for P(y_i = +1 | x_i, w).
estimate ranges between 0 and 1.
'''
import math
def predict_probability(feature_matrix, coefficients):
    # Take dot product of feature_matrix and coefficients
    # YOUR CODE HERE
    scores = np.dot(feature_matrix, coefficients)

    # Compute P(y_i = +1 | x_i, w) using the link function
    # YOUR CODE HERE
    predictions = 1 / (1 + math.e ** (-scores))

    # return predictions
    return predictions


I am working on the file "Classifier_2_logistic_regression.ipynb".
The first task is:
Load review dataset
For this assignment, we will use a subset of the Amazon product review dataset. The subset was chosen to contain similar numbers of positive and negative reviews, as the original dataset consisted primarily of positive reviews.
Can you write Python code for it?


I am working on the file "Classifier_2_logistic_regression.ipynb".
The first task is:
Note: For this assignment, we eliminated class imbalance by choosing
a subset of the data with a similar number of positive and negative reviews.
Apply text cleaning on the review data
In this section, we will perform some simple feature cleaning using SFrames. The last assignment used all words in building bag-of-words features, but here we limit ourselves to 193 words (for simplicity). We compiled a list of 193 most frequent words into a JSON file.
Now, we will load these words from this JSON file:
Can you write Python code for it?

import json
with open('important_words.json', 'r') as f: # Reads the list of most frequent words
    important_words = json.load(f)
important_words = [str(s) for s in important_words]

print important_words


I am working on the file "Classifier_2_logistic_regression.ipynb".
The first task is:
Now, complete the following code block for Step 2 to compute the class predictions using the scores obtained above:
Can you write Python code for it?


I am working on the file "Classifier_2_logistic_regression.ipynb".
The first task is:
Now, we will perform 2 simple data transformations:

Remove punctuation using Python's built-in string functionality.
Compute word counts (only for important_words)

We start with Step 1 which can be done as follows:
Can you write Python code for it?

def remove_punctuation(text):
    import string
    return text.translate(None, string.punctuation)

products['review_clean'] = products['review'].apply(remove_punctuation)


I am working on the file "Classifier_2_logistic_regression.ipynb".
The first task is:
Now, word_coefficient_tuples contains a sorted list of (word, coefficient_value) tuples. The first 10 elements in this list correspond to the words that are most positive.
Ten "most positive" words
Now, we compute the 10 words that have the most positive coefficient values. These words are associated with positive sentiment.
Can you write Python code for it?


I am working on the file "Classifier_2_logistic_regression.ipynb".
The first task is:
Now, write some code to compute the number of product reviews that contain the word perfect.
Hint:
* First create a column called contains_perfect which is set to 1 if the count of the word perfect (stored in column perfect) is >= 1.
* Sum the number of 1s in the column contains_perfect.
Can you write Python code for it?

np.sum?

(products['perfect'] >= 1).sum()


I am working on the file "Classifier_2_logistic_regression.ipynb".
The first task is:
One column of this dataset is 'sentiment', corresponding to the class label with +1 indicating a review with positive sentiment and -1 indicating one with negative sentiment.
Can you write Python code for it?


I am working on the file "Classifier_2_logistic_regression.ipynb".
The first task is:
Quiz Question. How many reviews contain the word perfect?
Convert SFrame to NumPy array
As you have seen previously, NumPy is a powerful library for doing matrix manipulation. Let us convert our data to matrices and then implement our algorithms with matrices.
First, make sure you can perform the following import.
Can you write Python code for it?


I am working on the file "Classifier_2_logistic_regression.ipynb".
The first task is:
Quiz question:  How many reviews were predicted to have positive sentiment?
Measuring accuracy
We will now measure the classification accuracy of the model. Recall from the lecture that the classification accuracy can be computed as follows:
$$
\mbox{accuracy} = \frac{\mbox{# correctly classified data points}}{\mbox{# total data points}}
$$
Complete the following code block to compute the accuracy of the model.
Can you write Python code for it?

predictions = np.array([+1 if score > 0 else -1 for score in scores])

(predictions == +1).sum()

(predictions != products['sentiment']).sum()

num_mistakes =  (predictions != products['sentiment']).sum()# YOUR CODE HERE
accuracy = 1 - num_mistakes * 1. / len(products) # YOUR CODE HERE
print "-----------------------------------------------------"
print '# Reviews   correctly classified =', len(products) - num_mistakes
print '# Reviews incorrectly classified =', num_mistakes
print '# Reviews total                  =', len(products)
print "-----------------------------------------------------"
print 'Accuracy = %.2f' % accuracy


I am working on the file "Classifier_2_logistic_regression.ipynb".
The first task is:
Quiz question: As each iteration of gradient ascent passes, does the log likelihood increase or decrease?
Predicting sentiments
Recall from lecture that class predictions for a data point $\mathbf{x}$ can be computed from the coefficients $\mathbf{w}$ using the following formula:
$$
\hat{y}_i =
\left{
\begin{array}{ll}
      +1 & \mathbf{x}_i^T\mathbf{w} > 0 \
      -1 & \mathbf{x}_i^T\mathbf{w} \leq 0 \
\end{array}
\right.
$$
Now, we will write some code to compute class predictions. We will do this in two steps:
* Step 1: First compute the scores using feature_matrix and coefficients using a dot product.
* Step 2: Using the formula above, compute the class predictions from the scores.
Step 1 can be implemented as follows:
Can you write Python code for it?

# Compute the scores as a dot product between feature_matrix and coefficients.
scores = np.dot(feature_matrix, coefficients)


I am working on the file "Classifier_2_logistic_regression.ipynb".
The first task is:
Quiz question: Which word is not present in the top 10 "most positive" words?
Ten "most negative" words
Next, we repeat this exercise on the 10 most negative words.  That is, we compute the 10 words that have the most negative coefficient values. These words are associated with negative sentiment.
Can you write Python code for it?


I am working on the file "Classifier_2_logistic_regression.ipynb".
The first task is:
The SFrame products now contains one column for each of the 193 important_words. As an example, the column perfect contains a count of the number of times the word perfect occurs in each of the reviews.
Can you write Python code for it?


I am working on the file "Classifier_2_logistic_regression.ipynb".
The first task is:
We now provide you with a function that extracts columns from an SFrame and converts them into a NumPy array. Two arrays are returned: one representing features and another representing class labels. Note that the feature matrix includes an additional column 'intercept' to take account of the intercept term.
Can you write Python code for it?

def get_numpy_data(data_sframe, features, label):
    data_sframe['intercept'] = 1
    features = ['intercept'] + features
    features_sframe = data_sframe[features]
    feature_matrix = features_sframe.to_numpy()
    label_sarray = data_sframe[label]
    label_array = label_sarray.to_numpy()
    return(feature_matrix, label_array)


I am working on the file "Classifier_2_logistic_regression_with_L2.ipynb".
The first task is:
Adding  L2 penalty
Let us now work on extending logistic regression with L2 regularization. As discussed in the lectures, the L2 regularization is particularly useful in preventing overfitting. In this assignment, we will explore L2 regularization in detail.
Recall from lecture and the previous assignment that for logistic regression without an L2 penalty, the derivative of the log likelihood function is:
$$
\frac{\partial\ell}{\partial w_j} = \sum_{i=1}^N h_j(\mathbf{x}_i)\left(\mathbf{1}[y_i = +1] - P(y_i = +1 | \mathbf{x}_i, \mathbf{w})\right)
$$
 Adding L2 penalty to the derivative
It takes only a small modification to add a L2 penalty. All terms indicated in red refer to terms that were added due to an L2 penalty.

Recall from the lecture that the link function is still the sigmoid:
$$
P(y_i = +1 | \mathbf{x}_i,\mathbf{w}) = \frac{1}{1 + \exp(-\mathbf{w}^T h(\mathbf{x}_i))},
$$
We add the L2 penalty term to the per-coefficient derivative of log likelihood:
$$
\frac{\partial\ell}{\partial w_j} = \sum_{i=1}^N h_j(\mathbf{x}_i)\left(\mathbf{1}[y_i = +1] - P(y_i = +1 | \mathbf{x}_i, \mathbf{w})\right) \color{red}{-2\lambda w_j }
$$

The per-coefficient derivative for logistic regression with an L2 penalty is as follows:
$$
\frac{\partial\ell}{\partial w_j} = \sum_{i=1}^N h_j(\mathbf{x}i)\left(\mathbf{1}[y_i = +1] - P(y_i = +1 | \mathbf{x}_i, \mathbf{w})\right) \color{red}{-2\lambda w_j }
$$
and for the intercept term, we have
$$
\frac{\partial\ell}{\partial w_0} = \sum{i=1}^N h_0(\mathbf{x}_i)\left(\mathbf{1}[y_i = +1] - P(y_i = +1 | \mathbf{x}_i, \mathbf{w})\right)
$$
Note: As we did in the Regression course, we do not apply the L2 penalty on the intercept. A large intercept does not necessarily indicate overfitting because the intercept is not associated with any particular feature.
Write a function that computes the derivative of log likelihood with respect to a single coefficient $w_j$. Unlike its counterpart in the last assignment, the function accepts five arguments:
 * errors vector containing...
Can you write Python code for it?

def feature_derivative_with_L2(errors, feature, coefficient, l2_penalty, feature_is_constant):

    # Compute the dot product of errors and feature
    ## YOUR CODE HERE
    derivative = np.dot(errors, feature)

    # add L2 penalty term for any feature that isn't the intercept.
    if not feature_is_constant:
        ## YOUR CODE HERE
        derivative = derivative - 2 * l2_penalty * coefficient

    return derivative


I am working on the file "Classifier_2_logistic_regression_with_L2.ipynb".
The first task is:
Are you running this notebook on an Amazon EC2 t2.micro instance? (If you are using your own machine, please skip this section)
It has been reported that t2.micro instances do not provide sufficient power to complete the conversion in acceptable amount of time. For interest of time, please refrain from running get_numpy_data function. Instead, download the binary file containing the four NumPy arrays you'll need for the assignment. To load the arrays, run the following commands:
arrays = np.load('module-4-assignment-numpy-arrays.npz')
feature_matrix_train, sentiment_train = arrays['feature_matrix_train'], arrays['sentiment_train']
feature_matrix_valid, sentiment_valid = arrays['feature_matrix_valid'], arrays['sentiment_valid']
Building on logistic regression with no L2 penalty assignment
Let us now build on Module 3 assignment. Recall from lecture that the link function for logistic regression can be defined as:
$$
P(y_i = +1 | \mathbf{x}_i,\mathbf{w}) = \frac{1}{1 + \exp(-\mathbf{w}^T h(\mathbf{x}_i))},
$$
where the feature vector $h(\mathbf{x}_i)$ is given by the word counts of important_words in the review $\mathbf{x}_i$.
We will use the same code as in this past assignment to make probability predictions since this part is not affected by the L2 penalty.  (Only the way in which the coefficients are learned is affected by the addition of a regularization term.)
Can you write Python code for it?

'''
produces probablistic estimate for P(y_i = +1 | x_i, w).
estimate ranges between 0 and 1.
'''
def predict_probability(feature_matrix, coefficients):
    # Take dot product of feature_matrix and coefficients
    ## YOUR CODE HERE
    scores = np.dot(feature_matrix, coefficients)

    # Compute P(y_i = +1 | x_i, w) using the link function
    ## YOUR CODE HERE
    predictions = 1 / (1 + np.exp(-scores))

    return predictions


I am working on the file "Classifier_2_logistic_regression_with_L2.ipynb".
The first task is:
Below, we compare the accuracy on the training data and validation data for all the models that were trained in this assignment.  We first calculate the accuracy values and then build a simple report summarizing the performance for the various models.
Can you write Python code for it?

train_accuracy = {}
train_accuracy[0]   = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_0_penalty)
train_accuracy[4]   = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_4_penalty)
train_accuracy[10]  = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_10_penalty)
train_accuracy[1e2] = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_1e2_penalty)
train_accuracy[1e3] = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_1e3_penalty)
train_accuracy[1e5] = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_1e5_penalty)

validation_accuracy = {}
validation_accuracy[0]   = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_0_penalty)
validation_accuracy[4]   = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_4_penalty)
validation_accuracy[10]  = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_10_penalty)
validation_accuracy[1e2] = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_1e2_penalty)
validation_accuracy[1e3] = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_1e3_penalty)
validation_accuracy[1e5] = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_1e5_penalty)

# Build a simple report
for key in sorted(validation_accuracy.keys()):
    print "L2 penalty = %g" % key
    print "train accuracy = %s, validation_accuracy = %s" % (train_accuracy[key], validation_accuracy[key])
    print "--------------------------------------------------------------------------------"


I am working on the file "Classifier_2_logistic_regression_with_L2.ipynb".
The first task is:
Compare coefficients
We now compare the coefficients for each of the models that were trained above. We will create a table of features and learned coefficients associated with each of the different L2 penalty values.
Below is a simple helper function that will help us create this table.
Can you write Python code for it?

table = graphlab.SFrame({'word': ['(intercept)'] + important_words})
def add_coefficients_to_table(coefficients, column_name):
    table[column_name] = coefficients
    return table


I am working on the file "Classifier_2_logistic_regression_with_L2.ipynb".
The first task is:
Convert SFrame to NumPy array
Just like in the second assignment of the previous module, we provide you with a function that extracts columns from an SFrame and converts them into a NumPy array. Two arrays are returned: one representing features and another representing class labels.
Note: The feature matrix includes an additional column 'intercept' filled with 1's to take account of the intercept term.
Can you write Python code for it?

import numpy as np

def get_numpy_data(data_sframe, features, label):
    data_sframe['intercept'] = 1
    features = ['intercept'] + features
    features_sframe = data_sframe[features]
    feature_matrix = features_sframe.to_numpy()
    label_sarray = data_sframe[label]
    label_array = label_sarray.to_numpy()
    return(feature_matrix, label_array)


I am working on the file "Classifier_2_logistic_regression_with_L2.ipynb".
The first task is:
Explore effects of L2 regularization
Now that we have written up all the pieces needed for regularized logistic regression, let's explore the benefits of using L2 regularization in analyzing sentiment for product reviews. As iterations pass, the log likelihood should increase.
Below, we train models with increasing amounts of regularization, starting with no L2 penalty, which is equivalent to our previous logistic regression implementation.
Can you write Python code for it?

my_logistic_regression_with_L2(feature_matrix_train, sentiment_train,
                                                     initial_coefficients=np.zeros(194),
                                                     step_size=5e-6, l2_penalty=0, max_iter=501)

# run with L2 = 0
coefficients_0_penalty = logistic_regression_with_L2(feature_matrix_train, sentiment_train,
                                                     initial_coefficients=np.zeros(194),
                                                     step_size=5e-6, l2_penalty=0, max_iter=501)

# run with L2 = 4
coefficients_4_penalty = logistic_regression_with_L2(feature_matrix_train, sentiment_train,
                                                      initial_coefficients=np.zeros(194),
                                                      step_size=5e-6, l2_penalty=4, max_iter=501)

# run with L2 = 10
coefficients_10_penalty = logistic_regression_with_L2(feature_matrix_train, sentiment_train,
                                                      initial_coefficients=np.zeros(194),
                                                      step_size=5e-6, l2_penalty=10, max_iter=501)

# run with L2 = 1e2
coefficients_1e2_penalty = logistic_regression_with_L2(feature_matrix_train, sentiment_train,
                                                       initial_coefficients=np.zeros(194),
                                                       step_size=5e-6, l2_penalty=1e2, max_iter=501)

# run with L2 = 1e3
coefficients_1e3_penalty = logistic_regression_with_L2(feature_matrix_train, sentiment_train,
                                                       initial_coefficients=np.zeros(194),
                                                       step_size=5e-6, l2_penalty=1e3, max_iter=501)

# run with L2 = 1e5
coefficients_1e5_penalty = logistic_regression_with_L2(feature_matrix_train, sentiment_train,
                                                       initial_coefficients=np.zeros(194),
                                                       step_size=5e-6,...


I am working on the file "Classifier_2_logistic_regression_with_L2.ipynb".
The first task is:
Just like we did previously, we will work with a hand-curated list of important words extracted from the review data. We will also perform 2 simple data transformations:

Remove punctuation using Python's built-in string functionality.
Compute word counts (only for the important_words)

Refer to Module 3 assignment for more details.
Can you write Python code for it?

# The same feature processing (same as the previous assignments)
# ---------------------------------------------------------------
import json
with open('important_words.json', 'r') as f: # Reads the list of most frequent words
    important_words = json.load(f)
important_words = [str(s) for s in important_words]

def remove_punctuation(text):
    import string
    return text.translate(None, string.punctuation)

# Remove punctuation.
products['review_clean'] = products['review'].apply(remove_punctuation)

# Split out the words into individual columns
for word in important_words:
    products[word] = products['review_clean'].apply(lambda s : s.split().count(word))


I am working on the file "Classifier_2_logistic_regression_with_L2.ipynb".
The first task is:
Load and process review dataset
For this assignment, we will use the same subset of the Amazon product review dataset that we used in Module 3 assignment. The subset was chosen to contain similar numbers of positive and negative reviews, as the original dataset consisted of mostly positive reviews.
Can you write Python code for it?


I am working on the file "Classifier_2_logistic_regression_with_L2.ipynb".
The first task is:
Now, let us take a look at what the dataset looks like (Note: This may take a few minutes).
Can you write Python code for it?


I am working on the file "Classifier_2_logistic_regression_with_L2.ipynb".
The first task is:
Now, let's run the function add_coefficients_to_table for each of the L2 penalty strengths.
Can you write Python code for it?

add_coefficients_to_table(coefficients_0_penalty, 'coefficients [L2=0]')
add_coefficients_to_table(coefficients_4_penalty, 'coefficients [L2=4]')
add_coefficients_to_table(coefficients_10_penalty, 'coefficients [L2=10]')
add_coefficients_to_table(coefficients_1e2_penalty, 'coefficients [L2=1e2]')
add_coefficients_to_table(coefficients_1e3_penalty, 'coefficients [L2=1e3]')
add_coefficients_to_table(coefficients_1e5_penalty, 'coefficients [L2=1e5]')


I am working on the file "Classifier_2_logistic_regression_with_L2.ipynb".
The first task is:
Quiz question: Does the term with L2 regularization increase or decrease $\ell\ell(\mathbf{w})$?
The logistic regression function looks almost like the one in the last assignment, with a minor modification to account for the L2 penalty.  Fill in the code below to complete this modification.
Can you write Python code for it?

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def compute_derivatives_with_L2(features, indicator, weights, L2):
    """
    indicator can only consists of 1(positive) and 0 (negative)
    """
    scores = np.dot(features, weights)
    predictions = sigmoid(scores)
    differences = indicator - predictions
    return np.dot(differences, features) - 2 * L2 * weights

def my_logistic_regression_with_L2(feature_matrix, sentiment, initial_coefficients, step_size, l2_penalty, max_iter):
    coefficients = np.array(initial_coefficients) # make sure it's a numpy array
    for itr in xrange(max_iter):
        # Predict P(y_i = +1|x_i,w) using your predict_probability() function
        ## YOUR CODE HERE
        predictions = predict_probability(feature_matrix, coefficients)

        # Compute indicator value for (y_i = +1)
        indicator = (sentiment==+1)

        # Compute the derivatives
        derivatives = compute_derivatives_with_L2(feature_matrix, indicator, coefficients, l2_penalty)

        # Update weights
        coefficients = coefficients + derivatives * step_size

        # Checking whether log likelihood is increasing
        if itr <= 15 or (itr <= 100 and itr % 10 == 0) or (itr <= 1000 and itr % 100 == 0) \
        or (itr <= 10000 and itr % 1000 == 0) or itr % 10000 == 0:
            lp = compute_log_likelihood_with_L2(feature_matrix, sentiment, coefficients, l2_penalty)
            print 'iteration %*d: log likelihood of observed labels = %.8f' % \
                (int(np.ceil(np.log10(max_iter))), itr, lp)
    return coefficients

def logistic_regression_with_L2(feature_matrix, sentiment, initial_coefficients, step_size, l2_penalty, max_iter):
    coefficients = np.array(initial_coefficients) # make sure it's a numpy array
    for itr in xrange(max_iter):
        # Predict P(y_i = +1|x_i,w) using your predict_probability() function
        ## YOUR CODE HERE
        predictions = predict_probability(feature_matrix, coefficients)

        # Compute...


I am working on the file "Classifier_2_logistic_regression_with_L2.ipynb".
The first task is:
Run the following cell to generate the plot. Use the plot to answer the following quiz question.
Can you write Python code for it?

make_coefficient_plot(table, positive_words, negative_words, l2_penalty_list=[0, 4, 10, 1e2, 1e3, 1e5])


I am working on the file "Classifier_2_logistic_regression_with_L2.ipynb".
The first task is:
Using the coefficients trained with L2 penalty 0, find the 5 most positive words (with largest positive coefficients). Save them to positive_words. Similarly, find the 5 most negative words (with largest negative coefficients) and save them to negative_words.
Quiz Question. Which of the following is not listed in either positive_words or negative_words?
Can you write Python code for it?

word_weights = zip(coefficients_0_penalty[1:],important_words)

sorted_word_weights = sorted(word_weights, reverse=True)

positive_words = [word for (weight, word) in sorted_word_weights[:5]]

negative_words = [word for (weight, word) in sorted_word_weights[-5:]]

positive_words

negative_words


I am working on the file "Classifier_3_decision_tree_assignment_1.ipynb".
The first task is:
Checkpoint: We should see that big_model has even better performance on the training set than decision_tree_model did on the training set.
Quiz Question: How does the performance of big_model on the validation set compare to decision_tree_model on the validation set? Is this a sign of overfitting?
Quantifying the cost of mistakes
Every mistake the model makes costs money. In this section, we will try and quantify the cost of each mistake made by the model.
Assume the following:

False negatives: Loans that were actually safe but were predicted to be risky. This results in an oppurtunity cost of losing a loan that would have otherwise been accepted.
False positives: Loans that were actually risky but were predicted to be safe. These are much more expensive because it results in a risky loan being given.
Correct predictions: All correct predictions don't typically incur any cost.

Let's write code that can compute the cost of mistakes made by the model. Complete the following 4 steps:
1. First, let us compute the predictions made by the model.
1. Second, compute the number of false positives.
2. Third, compute the number of false negatives.
3. Finally, compute the cost of mistakes made by the model by adding up the costs of true positives and false positives.
First, let us make predictions on validation_data using the decision_tree_model:
Can you write Python code for it?


I am working on the file "Classifier_3_decision_tree_assignment_1.ipynb".
The first task is:
Evaluating accuracy of the decision tree model
Recall that the accuracy is defined as follows:
$$
\mbox{accuracy} = \frac{\mbox{# correctly classified examples}}{\mbox{# total examples}}
$$
Let us start by evaluating the accuracy of the small_model and decision_tree_model on the training data
Can you write Python code for it?

print small_model.evaluate(train_data)['accuracy']
print decision_tree_model.evaluate(train_data)['accuracy']


I am working on the file "Classifier_3_decision_tree_assignment_1.ipynb".
The first task is:
Explore label predictions
Now, we will use our model  to predict whether or not a loan is likely to default. For each row in the sample_validation_data, use the decision_tree_model to predict whether or not the loan is classified as a safe loan.
Hint: Be sure to use the .predict() method.
Can you write Python code for it?


I am working on the file "Classifier_3_decision_tree_assignment_1.ipynb".
The first task is:
Exploring some features
Let's quickly explore what the dataset looks like. First, let's print out the column names to see what features we have in this dataset.
Can you write Python code for it?


I am working on the file "Classifier_3_decision_tree_assignment_1.ipynb".
The first task is:
False negatives are predictions where the model predicts -1 but the true label is +1. Complete the following code block for the number of false negatives:
Can you write Python code for it?

((predictions == -1) & (validation_data['safe_loans'] == 1)).sum()


I am working on the file "Classifier_3_decision_tree_assignment_1.ipynb".
The first task is:
False positives are predictions where the model predicts +1 but the true label is -1. Complete the following code block for the number of false positives:
Can you write Python code for it?

((predictions == 1) & (validation_data['safe_loans'] == -1)).sum()


I am working on the file "Classifier_3_decision_tree_assignment_1.ipynb".
The first task is:
In the view that is provided by GraphLab Create, you can see each node, and each split at each node. This visualization is great for considering what happens when this model predicts the target of a new data point.
Note: To better understand this visual:
* The root node is represented using pink.
* Intermediate nodes are in green.
* Leaf nodes in blue and orange.
Can you write Python code for it?


I am working on the file "Classifier_3_decision_tree_assignment_1.ipynb".
The first task is:
Let's visualize the small tree here to do the traversing for this data point.
Can you write Python code for it?


I am working on the file "Classifier_3_decision_tree_assignment_1.ipynb".
The first task is:
Load LendingClub dataset
We will be using a dataset from the LendingClub. A parsed and cleaned form of the dataset is availiable here. Make sure you download the dataset before running the following command.
Can you write Python code for it?


I am working on the file "Classifier_3_decision_tree_assignment_1.ipynb".
The first task is:
Making predictions
Let's consider two positive and two negative examples from the validation set and see what the model predicts. We will do the following:
* Predict whether or not a loan is safe.
* Predict the probability that a loan is safe.
Can you write Python code for it?

validation_safe_loans = validation_data[validation_data[target] == 1]
validation_risky_loans = validation_data[validation_data[target] == -1]

sample_validation_data_risky = validation_risky_loans[0:2]
sample_validation_data_safe = validation_safe_loans[0:2]

sample_validation_data = sample_validation_data_safe.append(sample_validation_data_risky)
sample_validation_data


I am working on the file "Classifier_3_decision_tree_assignment_1.ipynb".
The first task is:
Note: In the tree visualization above, the values at the leaf nodes are not class predictions but scores (a slightly advanced concept that is out of the scope of this course). You can read more about this here.  If the score is $\geq$ 0, the class +1 is predicted.  Otherwise, if the score < 0, we predict class -1.
Quiz Question: Based on the visualized tree, what prediction would you make for this data point?
Now, let's verify your prediction by examining the prediction made using GraphLab Create.  Use the .predict function on small_model.
Can you write Python code for it?


I am working on the file "Classifier_3_decision_tree_assignment_1.ipynb".
The first task is:
Note: There are many approaches for dealing with imbalanced data, including some where we modify the learning algorithm. These approaches are beyond the scope of this course, but some of them are reviewed in this paper. For this assignment, we use the simplest possible approach, where we subsample the overly represented class to get a more balanced dataset. In general, and especially when the data is highly imbalanced, we recommend using more advanced methods.
Split data into training and validation sets
We split the data into training and validation sets using an 80/20 split and specifying seed=1 so everyone gets the same results.
Note: In previous assignments, we have called this a train-test split. However, the portion of data that we don't train on will be used to help select model parameters (this is known as model selection). Thus, this portion of data should be called a validation set. Recall that examining performance of various potential models (i.e. models with different parameters) should be on validation set, while evaluation of the final selected model should always be on test data. Typically, we would also save a portion of the data (a real test set) to test our final model on or use cross-validation on the training set to select our final model. But for the learning purposes of this assignment, we won't do that.
Can you write Python code for it?

train_data, validation_data = loans_data.random_split(.8, seed=1)


I am working on the file "Classifier_3_decision_tree_assignment_1.ipynb".
The first task is:
Now, let us evaluate big_model on the training set and validation set.
Can you write Python code for it?

print big_model.evaluate(train_data)['accuracy']
print big_model.evaluate(validation_data)['accuracy']


I am working on the file "Classifier_3_decision_tree_assignment_1.ipynb".
The first task is:
Now, let's verify that the resulting percentage of safe and risky loans are each nearly 50%.
Can you write Python code for it?

print "Percentage of safe loans                 :", len(safe_loans) / float(len(loans_data))
print "Percentage of risky loans                :", len(risky_loans) / float(len(loans_data))
print "Total number of loans in our new dataset :", len(loans_data)


I am working on the file "Classifier_3_decision_tree_assignment_1.ipynb".
The first task is:
This feature describes whether the loanee is mortaging, renting, or owns a home. We can see that a small percentage of the loanees own a home.
Exploring the target column
The target column (label column) of the dataset that we are interested in is called bad_loans. In this column 1 means a risky (bad) loan 0 means a safe  loan.
In order to make this more intuitive and consistent with the lectures, we reassign the target to be:
* +1 as a safe  loan,
* -1 as a risky (bad) loan.
We put this in a new column called safe_loans.
Can you write Python code for it?

# safe_loans =  1 => safe
# safe_loans = -1 => risky
loans['safe_loans'] = loans['bad_loans'].apply(lambda x : +1 if x==0 else -1)
loans = loans.remove_column('bad_loans')


I am working on the file "Classifier_3_decision_tree_assignment_1.ipynb".
The first task is:
Use decision tree to build a classifier
Now, let's use the built-in GraphLab Create decision tree learner to create a loan prediction model on the training data. (In the next assignment, you will implement your own decision tree learning algorithm.)  Our feature columns and target column have already been decided above. Use validation_set=None to get the same results as everyone else.
Can you write Python code for it?

decision_tree_model = graphlab.decision_tree_classifier.create(train_data, validation_set=None,
                                target = target, features = features)


I am working on the file "Classifier_3_decision_tree_assignment_1.ipynb".
The first task is:
You should have:
* Around 81% safe loans
* Around 19% risky loans
It looks like most of these loans are safe loans (thankfully). But this does make our problem of identifying risky loans challenging.
Features for the classification algorithm
In this assignment, we will be using a subset of features (categorical and numeric). The features we will be using are described in the code comments below. If you are a finance geek, the LendingClub website has a lot more details about these features.
Can you write Python code for it?

features = ['grade',                     # grade of the loan
            'sub_grade',                 # sub-grade of the loan
            'short_emp',                 # one year or less of employment
            'emp_length_num',            # number of years of employment
            'home_ownership',            # home_ownership status: own, mortgage or rent
            'dti',                       # debt to income ratio
            'purpose',                   # the purpose of the loan
            'term',                      # the term of the loan
            'last_delinq_none',          # has borrower had a delinquincy
            'last_major_derog_none',     # has borrower had 90 day or worse rating
            'revol_util',                # percent of available credit being used
            'total_rec_late_fee',        # total late fees received to day
           ]

target = 'safe_loans'                   # prediction target (y) (+1 means safe, -1 is risky)

# Extract the feature columns and target column
loans = loans[features + [target]]


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
Because there are several steps in this assignment, we have introduced some stopping points where you can check your code and make sure it is correct before proceeding. To test your intermediate_node_num_mistakes function, run the following code until you get a Test passed!, then you should proceed. Otherwise, you should spend some time figuring out where things went wrong.
Can you write Python code for it?

# Test case 1
example_labels = graphlab.SArray([-1, -1, 1, 1, 1])
if intermediate_node_num_mistakes(example_labels) == 2:
    print 'Test passed!'
else:
    print 'Test 1 failed... try again!'

# Test case 2
example_labels = graphlab.SArray([-1, -1, 1, 1, 1, 1, 1])
if intermediate_node_num_mistakes(example_labels) == 2:
    print 'Test passed!'
else:
    print 'Test 2 failed... try again!'

# Test case 3
example_labels = graphlab.SArray([-1, -1, -1, -1, -1, 1, 1])
if intermediate_node_num_mistakes(example_labels) == 2:
    print 'Test passed!'
else:
    print 'Test 3 failed... try again!'


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
Building the tree
With the above functions implemented correctly, we are now ready to build our decision tree. Each node in the decision tree is represented as a dictionary which contains the following keys and possible values:
{
   'is_leaf'            : True/False.
   'prediction'         : Prediction at the leaf node.
   'left'               : (dictionary corresponding to the left tree).
   'right'              : (dictionary corresponding to the right tree).
   'splitting_feature'  : The feature that this node splits on.
}

First, we will write a function that creates a leaf node given a set of target values. Fill in the places where you find ## YOUR CODE HERE. There are three places in this function for you to fill in.
Can you write Python code for it?

def create_leaf(target_values):

    # Create a leaf node
    leaf = {'splitting_feature' : None,
            'left' : None,
            'right' : None,
            'is_leaf': True    }   ## YOUR CODE HERE

    # Count the number of data points that are +1 and -1 in this node.
    num_ones = len(target_values[target_values == +1])
    num_minus_ones = len(target_values[target_values == -1])

    # For the leaf node, set the prediction to be the majority class.
    # Store the predicted class (1 or -1) in leaf['prediction']
    if num_ones > num_minus_ones:
        leaf['prediction'] = +1       ## YOUR CODE HERE
    else:
        leaf['prediction'] = -1       ## YOUR CODE HERE

    # Return the leaf node
    return leaf


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
Decision tree implementation
In this section, we will implement binary decision trees from scratch. There are several steps involved in building a decision tree. For that reason, we have split the entire assignment into several sections.
Function to count number of mistakes while predicting majority class
Recall from the lecture that prediction at an intermediate node works by predicting the majority class for all data points that belong to this node.
Now, we will write a function that calculates the number of missclassified examples when predicting the majority class. This will be used to help determine which feature is the best to split on at a given node of the tree.
Note: Keep in mind that in order to compute the number of mistakes for a majority classifier, we only need the label (y values) of the data points in the node.
 Steps to follow :
*  Step 1: Calculate the number of safe loans and risky loans.
*  Step 2: Since we are assuming majority class prediction, all the data points that are not in the majority class are considered mistakes.
*  Step 3: Return the number of mistakes.
Now, let us write the function intermediate_node_num_mistakes which computes the number of misclassified examples of an intermediate node given the set of labels (y values) of the data points contained in the node. Fill in the places where you find ## YOUR CODE HERE. There are three places in this function for you to fill in.
Can you write Python code for it?

def intermediate_node_num_mistakes(labels_in_node):
    # Corner case: If labels_in_node is empty, return 0
    if len(labels_in_node) == 0:
        return 0

    # Count the number of 1's (safe loans)
    ## YOUR CODE HERE
    safe = (labels_in_node == 1).sum()

    # Count the number of -1's (risky loans)
    ## YOUR CODE HERE
    risky = (labels_in_node == -1).sum()

    # Return the number of mistakes that the majority classifier makes.
    ## YOUR CODE HERE
    return min(safe, risky)


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
Exploring the left subtree of the left subtree
Can you write Python code for it?


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
Function to pick best feature to split on
The function best_splitting_feature takes 3 arguments:
1. The data (SFrame of data which includes all of the feature columns and label column)
2. The features to consider for splits (a list of strings of column names to consider for splits)
3. The name of the target/label column (string)
The function will loop through the list of possible features, and consider splitting on each of them. It will calculate the classification error of each split and return the feature that had the smallest classification error when split on.
Recall that the classification error is defined as follows:
$$
\mbox{classification error} = \frac{\mbox{# mistakes}}{\mbox{# total examples}}
$$
Follow these steps:
* Step 1: Loop over each feature in the feature list
* Step 2: Within the loop, split the data into two groups: one group where all of the data has feature value 0 or False (we will call this the left split), and one group where all of the data has feature value 1 or True (we will call this the right split). Make sure the left split corresponds with 0 and the right split corresponds with 1 to ensure your implementation fits with our implementation of the tree building process.
* Step 3: Calculate the number of misclassified examples in both groups of data and use the above formula to compute the classification error.
* Step 4: If the computed error is smaller than the best error found so far, store this feature and its error.
This may seem like a lot, but we have provided pseudocode in the comments in order to help you implement the function correctly.
Note: Remember that since we are only dealing with binary features, we do not have to consider thresholds for real-valued features. This makes the implementation of this function much easier.
Fill in the places where you find ## YOUR CODE HERE. There are five places in this function for you to fill in.
Can you write Python code for it?

def best_splitting_feature(data, features, target):

    best_feature = None # Keep track of the best feature
    best_error = 10     # Keep track of the best error so far
    # Note: Since error is always <= 1, we should intialize it with something larger than 1.

    # Convert to float to make sure error gets computed correctly.
    num_data_points = float(len(data))

    # Loop through each feature to consider splitting on that feature
    for feature in features:

        # The left split will have all data points where the feature value is 0
        left_split = data[data[feature] == 0]

        # The right split will have all data points where the feature value is 1
        ## YOUR CODE HERE
        right_split = data[data[feature] == 1]

        # Calculate the number of misclassified examples in the left split.
        # Remember that we implemented a function for this! (It was called intermediate_node_num_mistakes)
        # YOUR CODE HERE
        left_mistakes = intermediate_node_num_mistakes(left_split[target])

        # Calculate the number of misclassified examples in the right split.
        ## YOUR CODE HERE
        right_mistakes = intermediate_node_num_mistakes(right_split[target])

        # Compute the classification error of this split.
        # Error = (# of mistakes (left) + # of mistakes (right)) / (# of data points)
        ## YOUR CODE HERE
        error = (left_mistakes + right_mistakes) / num_data_points

        # If this is the best error we have found so far, store the feature as best_feature and the error as best_error
        ## YOUR CODE HERE
        if error < best_error:
            best_feature = feature
            best_error = error

    return best_feature # Return the best feature we found


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
Here is a recursive function to count the nodes in your tree:
Can you write Python code for it?

def count_nodes(tree):
    if tree['is_leaf']:
        return 1
    return 1 + count_nodes(tree['left']) + count_nodes(tree['right'])


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
Let's add some annotations to our prediction to see what the prediction path was that lead to this predicted class:
Can you write Python code for it?


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
Let's explore what the dataset looks like.
Can you write Python code for it?


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
Like the previous assignment, we reassign the labels to have +1 for a safe loan, and -1 for a risky (bad) loan.
Can you write Python code for it?

loans['safe_loans'] = loans['bad_loans'].apply(lambda x : +1 if x==0 else -1)
loans = loans.remove_column('bad_loans')


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
Load the lending club dataset
We will be using the same LendingClub dataset as in the previous assignment.
Can you write Python code for it?


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
Note: There are many approaches for dealing with imbalanced data, including some where we modify the learning algorithm. These approaches are beyond the scope of this course, but some of them are reviewed in "Learning from Imbalanced Data" by Haibo He and Edwardo A. Garcia, IEEE Transactions on Knowledge and Data Engineering 21(9) (June 26, 2009), p. 1263–1284. For this assignment, we use the simplest possible approach, where we subsample the overly represented class to get a more balanced dataset. In general, and especially when the data is highly imbalanced, we recommend using more advanced methods.
Transform categorical data into binary features
In this assignment, we will implement binary decision trees (decision trees for binary features, a specific case of categorical variables taking on two values, e.g., true/false). Since all of our features are currently categorical features, we want to turn them into binary features.
For instance, the home_ownership feature represents the home ownership status of the loanee, which is either own, mortgage or rent. For example, if a data point has the feature
{'home_ownership': 'RENT'}
we want to turn this into three features:
{
   'home_ownership = OWN'      : 0,
   'home_ownership = MORTGAGE' : 0,
   'home_ownership = RENT'     : 1
 }
Since this code requires a few Python and GraphLab tricks, feel free to use this block of code as is. Refer to the API documentation for a deeper understanding.
Can you write Python code for it?

loans_data = risky_loans.append(safe_loans)
for feature in features:
    loans_data_one_hot_encoded = loans_data[feature].apply(lambda x: {x: 1})
    loans_data_unpacked = loans_data_one_hot_encoded.unpack(column_name_prefix=feature)

    # Change None's to 0's
    for column in loans_data_unpacked.column_names():
        loans_data_unpacked[column] = loans_data_unpacked[column].fillna(0)

    loans_data.remove_column(feature)
    loans_data.add_columns(loans_data_unpacked)


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
Now, let's consider the first example of the test set and see what my_decision_tree model predicts for this data point.
Can you write Python code for it?

test_data[0]

print 'Predicted class: %s ' % classify(my_decision_tree, test_data[0])


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
Now, let's use this function to evaluate the classification error on the test set.
Can you write Python code for it?


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
Quiz Question: What is the feature that is used for the split at the root node?
Exploring the intermediate left subtree
The tree is a recursive dictionary, so we do have access to all the nodes! We can use
* my_decision_tree['left'] to go left
* my_decision_tree['right'] to go right
Can you write Python code for it?


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
Quiz question: What is the path of the first 3 feature splits considered along the left-most branch of my_decision_tree?
Quiz question: What is the path of the first 3 feature splits considered along the right-most branch of my_decision_tree?
Can you write Python code for it?


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
Run the following test code to check your implementation. Make sure you get 'Test passed' before proceeding.
Can you write Python code for it?

small_data_decision_tree = decision_tree_create(train_data, features, 'safe_loans', max_depth = 3)
if count_nodes(small_data_decision_tree) == 13:
    print 'Test passed!'
else:
    print 'Test failed... try again!'
    print 'Number of nodes found                :', count_nodes(small_data_decision_tree)
    print 'Number of nodes that should be there : 13'


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
Subsample dataset to make sure classes are balanced
Just as we did in the previous assignment, we will undersample the larger class (safe loans) in order to balance out our dataset. This means we are throwing away many data points. We use seed=1 so everyone gets the same results.
Can you write Python code for it?

safe_loans_raw = loans[loans[target] == 1]
risky_loans_raw = loans[loans[target] == -1]

# Since there are less risky loans than safe loans, find the ratio of the sizes
# and use that percentage to undersample the safe loans.
percentage = len(risky_loans_raw)/float(len(safe_loans_raw))
safe_loans = safe_loans_raw.sample(percentage, seed = 1)
risky_loans = risky_loans_raw
loans_data = risky_loans.append(safe_loans)

print "Percentage of safe loans                 :", len(safe_loans) / float(len(loans_data))
print "Percentage of risky loans                :", len(risky_loans) / float(len(loans_data))
print "Total number of loans in our new dataset :", len(loans_data)


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
This column is set to 1 if the loan grade is A and 0 otherwise.
Checkpoint: Make sure the following answers match up.
Can you write Python code for it?

print "Total number of grade.A loans : %s" % loans_data['grade.A'].sum()
print "Expexted answer               : 6422"


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
To test your best_splitting_feature function, run the following code:
Can you write Python code for it?

if best_splitting_feature(train_data, features, 'safe_loans') == 'term. 36 months':
    print 'Test passed!'
else:
    print 'Test failed... try again!'


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
Train-test split
We split the data into a train test split with 80% of the data in the training set and 20% of the data in the test set. We use seed=1 so that everyone gets the same result.
Can you write Python code for it?

train_data, test_data = loans_data.random_split(.8, seed=1)


I am working on the file "Classifier_3_decision_tree_assignment_2.ipynb".
The first task is:
We have provided a function that learns the decision tree recursively and implements 3 stopping conditions:
1. Stopping condition 1: All data points in a node are from the same class.
2. Stopping condition 2: No more features to split on.
3. Additional stopping condition: In addition to the above two stopping conditions covered in lecture, in this assignment we will also consider a stopping condition based on the max_depth of the tree. By not letting the tree grow too deep, we will save computational effort in the learning process.
Now, we will write down the skeleton of the learning algorithm. Fill in the places where you find ## YOUR CODE HERE. There are seven places in this function for you to fill in.
Can you write Python code for it?

def decision_tree_create(data, features, target, current_depth = 0, max_depth = 10):
    remaining_features = features[:] # Make a copy of the features.

    target_values = data[target]
    print "--------------------------------------------------------------------"
    print "Subtree, depth = %s (%s data points)." % (current_depth, len(target_values))

    # Stopping condition 1
    # (Check if there are mistakes at current node.
    # Recall you wrote a function intermediate_node_num_mistakes to compute this.)
    if intermediate_node_num_mistakes(data[target]) == 0:  ## YOUR CODE HERE
        print "Stopping condition 1 reached."
        # If not mistakes at current node, make current node a leaf node
        return create_leaf(target_values)

    # Stopping condition 2 (check if there are remaining features to consider splitting on)
    if remaining_features == []:   ## YOUR CODE HERE
        print "Stopping condition 2 reached."
        # If there are no remaining features to consider, make current node a leaf node
        return create_leaf(target_values)

    # Additional stopping condition (limit tree depth)
    if current_depth >= max_depth:  ## YOUR CODE HERE
        print "Reached maximum depth. Stopping for now."
        # If the max tree depth has been reached, make current node a leaf node
        return create_leaf(target_values)

    # Find the best splitting feature (recall the function best_splitting_feature implemented above)
    ## YOUR CODE HERE
    splitting_feature = best_splitting_feature(data, features, target)

    # Split on the best feature that we found.
    left_split = data[data[splitting_feature] == 0]
    right_split = data[data[splitting_feature] == 1]      ## YOUR CODE HERE
    remaining_features.remove(splitting_feature)
    print "Split on feature %s. (%s, %s)" % (\
                      splitting_feature, len(left_split), len(right_split))

    # Create a leaf node if the split is "perfect"
    if len(left_split) == len(data):
     ...


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
As before, we reassign the labels to have +1 for a safe loan, and -1 for a risky (bad) loan.
Can you write Python code for it?


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Build a tree!
Now that your code is working, we will train a tree model on the train_data with
* max_depth = 6
* min_node_size = 100,
* min_error_reduction = 0.0
Warning: This code block may take a minute to learn.
Can you write Python code for it?

my_decision_tree_new = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                                min_node_size = 100, min_error_reduction=0.0)


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Early stopping methods for decision trees
In this section, we will extend the binary tree implementation from the previous assignment in order to handle some early stopping conditions. Recall the 3 early stopping methods that were discussed in lecture:

Reached a maximum depth. (set by parameter max_depth).
Reached a minimum node size. (set by parameter min_node_size).
Don't split if the gain in error reduction is too small. (set by parameter min_error_reduction).

For the rest of this assignment, we will refer to these three as early stopping conditions 1, 2, and 3.
Early stopping condition 1: Maximum depth
Recall that we already implemented the maximum depth stopping condition in the previous assignment. In this assignment, we will experiment with this condition a bit more and also write code to implement the 2nd and 3rd early stopping conditions.
We will be reusing code from the previous assignment and then building upon this.  We will alert you when you reach a function that was part of the previous assignment so that you can simply copy and past your previous code.
Early stopping condition 2: Minimum node size
The function reached_minimum_node_size takes 2 arguments:

The data (from a node)
The minimum number of data points that a node is allowed to split on, min_node_size.

This function simply calculates whether the number of data points at a given node is less than or equal to the specified minimum node size. This function will be used to detect this early stopping condition in the decision_tree_create function.
Fill in the parts of the function below where you find ## YOUR CODE HERE.  There is one instance in the function below.
Can you write Python code for it?

def reached_minimum_node_size(data, min_node_size):
    # Return True if the number of data points is less than or equal to the minimum node size.
    ## YOUR CODE HERE
    return len(data) <= min_node_size


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Evaluating the models
Let us evaluate the models on the train and validation data. Let us start by evaluating the classification error on the training data:
Can you write Python code for it?

print "Training data, classification error (model 1):", evaluate_classification_error(model_1, train_data)
print "Training data, classification error (model 2):", evaluate_classification_error(model_2, train_data)
print "Training data, classification error (model 3):", evaluate_classification_error(model_3, train_data)


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Here is a function to count the nodes in your tree:
Can you write Python code for it?


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Let's add some annotations to our prediction to see what the prediction path was that lead to this predicted class:
Can you write Python code for it?

classify(my_decision_tree_new, validation_set[0], annotate = True)


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Let's now recall the prediction path for the decision tree learned in the previous assignment, which we recreated here as my_decision_tree_old.
Can you write Python code for it?

classify(my_decision_tree_old, validation_set[0], annotate = True)


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Let's now train a tree model ignoring early stopping conditions 2 and 3 so that we get the same tree as in the previous assignment.  To ignore these conditions, we set min_node_size=0 and min_error_reduction=-1 (a negative value).
Can you write Python code for it?

my_decision_tree_old = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                                min_node_size = 0, min_error_reduction=-1)


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Load LendingClub Dataset
This assignment will use the LendingClub dataset used in the previous two assignments.
Can you write Python code for it?


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Note: There are many approaches for dealing with imbalanced data, including some where we modify the learning algorithm. These approaches are beyond the scope of this course, but some of them are reviewed in this paper. For this assignment, we use the simplest possible approach, where we subsample the overly represented class to get a more balanced dataset. In general, and especially when the data is highly imbalanced, we recommend using more advanced methods.
Transform categorical data into binary features
Since we are implementing binary decision trees, we transform our categorical data into binary data using 1-hot encoding, just as in the previous assignment. Here is the summary of that discussion:
For instance, the home_ownership feature represents the home ownership status of the loanee, which is either own, mortgage or rent. For example, if a data point has the feature
{'home_ownership': 'RENT'}
we want to turn this into three features:
{
   'home_ownership = OWN'      : 0,
   'home_ownership = MORTGAGE' : 0,
   'home_ownership = RENT'     : 1
 }
Since this code requires a few Python and GraphLab tricks, feel free to use this block of code as is. Refer to the API documentation for a deeper understanding.
Can you write Python code for it?


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Now, evaluate the validation error using my_decision_tree_old.
Can you write Python code for it?


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Now, let us evaluate the models (model_7, model_8, or model_9) on the validation_set.
Can you write Python code for it?

print "Validation data, classification error (model 4):", evaluate_classification_error(model_7, validation_set)
print "Validation data, classification error (model 5):", evaluate_classification_error(model_8, validation_set)
print "Validation data, classification error (model 6):", evaluate_classification_error(model_9, validation_set)


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Quiz Question: Which tree has the smallest error on the validation data?
Quiz Question: Does the tree with the smallest error in the training data also have the smallest error in the validation data?
Quiz Question: Is it always true that the tree with the lowest classification error on the training set will result in the lowest classification error in the validation set?
Measuring the complexity of the tree
Recall in the lecture that we talked about deeper trees being more complex. We will measure the complexity of the tree as
complexity(T) = number of leaves in the tree T
Here, we provide a function count_leaves that counts the number of leaves in a tree. Using this implementation, compute the number of nodes in model_1, model_2, and model_3.
Can you write Python code for it?

def count_leaves(tree):
    if tree['is_leaf']:
        return 1
    return count_leaves(tree['left']) + count_leaves(tree['right'])


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Quiz question: Assume an intermediate node has 6 safe loans and 3 risky loans.  For each of 4 possible features to split on, the error reduction is 0.0, 0.05, 0.1, and 0.14, respectively. If the minimum gain in error reduction parameter is set to 0.2, what should the tree learning algorithm do next?
Grabbing binary decision tree helper functions from past assignment
Recall from the previous assignment that we wrote a function intermediate_node_num_mistakes that calculates the number of misclassified examples when predicting the majority class. This is used to help determine which feature is best to split on at a given node of the tree.
Please copy and paste your code for intermediate_node_num_mistakes here.
Can you write Python code for it?

def intermediate_node_num_mistakes(labels_in_node):
    # Corner case: If labels_in_node is empty, return 0
    if len(labels_in_node) == 0:
        return 0

    # Count the number of 1's (safe loans)
    ## YOUR CODE HERE
    safe = (labels_in_node == 1).sum()

    # Count the number of -1's (risky loans)
    ## YOUR CODE HERE
    risky = (labels_in_node == -1).sum()

    # Return the number of mistakes that the majority classifier makes.
    ## YOUR CODE HERE
    return min(safe, risky)


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Quiz question: Is the validation error of the new decision tree (using early stopping conditions 2 and 3) lower than, higher than, or the same as that of the old decision tree from the previous assignment?
Exploring the effect of max_depth
We will compare three models trained with different values of the stopping criterion. We intentionally picked models at the extreme ends (too small, just right, and too large).
Train three models with these parameters:

model_1: max_depth = 2 (too small)
model_2: max_depth = 6 (just right)
model_3: max_depth = 14 (may be too large)

For each of these three, we set min_node_size = 0 and min_error_reduction = -1.
 Note: Each tree can take up to a few minutes to train. In particular, model_3 will probably take the longest to train.
Can you write Python code for it?

model_1 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 2,
                                min_node_size = 0, min_error_reduction=-1)
model_2 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                                min_node_size = 0, min_error_reduction=-1)
model_3 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 14,
                                min_node_size = 0, min_error_reduction=-1)