I am working on the file "Classification/Week 2/Assignment 1/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Aside. How the link function works with matrix algebra
Since the word counts are stored as columns in feature_matrix, each $i$-th row of the matrix corresponds to the feature vector $h(\mathbf{x}_i)$:
$$
[\text{feature_matrix}] =
\left[
\begin{array}{c}
h(\mathbf{x}_1)^T \
h(\mathbf{x}_2)^T \
\vdots \
h(\mathbf{x}_N)^T
\end{array}
\right] =
\left[
\begin{array}{cccc}
h_0(\mathbf{x}_1) & h_1(\mathbf{x}_1) & \cdots & h_D(\mathbf{x}_1) \
h_0(\mathbf{x}_2) & h_1(\mathbf{x}_2) & \cdots & h_D(\mathbf{x}_2) \
\vdots & \vdots & \ddots & \vdots \
h_0(\mathbf{x}_N) & h_1(\mathbf{x}_N) & \cdots & h_D(\mathbf{x}_N)
\end{array}
\right]
$$
By the rules of matrix multiplication, the score vector containing elements $\mathbf{w}^T h(\mathbf{x}_i)$ is obtained by multiplying feature_matrix and the coefficient vector $\mathbf{w}$.
$$
[\text{score}] =
[\text{feature_matrix}]\mathbf{w} =
\left[
\begin{array}{c}
h(\mathbf{x}_1)^T \
h(\mathbf{x}_2)^T \
\vdots \
h(\mathbf{x}_N)^T
\end{array}
\right]
\mathbf{w}
= \left[
\begin{array}{c}
h(\mathbf{x}_1)^T\mathbf{w} \
h(\mathbf{x}_2)^T\mathbf{w} \
\vdots \
h(\mathbf{x}_N)^T\mathbf{w}
\end{array}
\right]
= \left[
\begin{array}{c}
\mathbf{w}^T h(\mathbf{x}_1) \
\mathbf{w}^T h(\mathbf{x}_2) \
\vdots \
\mathbf{w}^T h(\mathbf{x}_N)
\end{array}
\right]
$$
Checkpoint
Just to make sure you are on the right track, we have provided a few examples. If your predict_probability function is implemented correctly, then the outputs will match:
Can you write Python code for it?

dummy_feature_matrix = np.array([[1.,2.,3.], [1.,-1.,-1]])
dummy_coefficients = np.array([1., 3., -1.])

correct_scores      = np.array( [ 1.*1. + 2.*3. + 3.*(-1.),          1.*1. + (-1.)*3. + (-1.)*(-1.) ] )
correct_predictions = np.array( [ 1./(1+np.exp(-correct_scores[0])), 1./(1+np.exp(-correct_scores[1])) ] )

print 'The following outputs must match '
print '------------------------------------------------'
print 'correct_predictions           =', correct_predictions
print 'output of predict_probability =', predict_probability(dummy_feature_matrix, dummy_coefficients)


I am working on the file "Classification/Week 2/Assignment 1/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Estimating conditional probability with link function
Recall from lecture that the link function is given by:
$$
P(y_i = +1 | \mathbf{x}_i,\mathbf{w}) = \frac{1}{1 + \exp(-\mathbf{w}^T h(\mathbf{x}_i))},
$$
where the feature vector $h(\mathbf{x}_i)$ represents the word counts of important_words in the review  $\mathbf{x}_i$. Complete the following function that implements the link function:
Can you write Python code for it?

'''
produces probablistic estimate for P(y_i = +1 | x_i, w).
estimate ranges between 0 and 1.
'''
def predict_probability(feature_matrix, coefficients):
    # Take dot product of feature_matrix and coefficients
    product = feature_matrix.dot(coefficients)

    # Compute P(y_i = +1 | x_i, w) using the link function
    predictions = 1 / (1 + np.exp(-product))

    # return predictions
    return predictions


I am working on the file "Classification/Week 2/Assignment 1/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
In the main lecture, our focus was on the likelihood.  In the advanced optional video, however, we introduced a transformation of this likelihood---called the log likelihood---that simplifies the derivation of the gradient and is more numerically stable.  Due to its numerical stability, we will use the log likelihood instead of the likelihood to assess the algorithm.
The log likelihood is computed using the following formula (see the advanced optional video if you are curious about the derivation of this equation):
$$\ell\ell(\mathbf{w}) = \sum_{i=1}^N \Big( (\mathbf{1}[y_i = +1] - 1)\mathbf{w}^T h(\mathbf{x}_i) - \ln\left(1 + \exp(-\mathbf{w}^T h(\mathbf{x}_i))\right) \Big) $$
We provide a function to compute the log likelihood for the entire dataset.
Can you write Python code for it?

def compute_log_likelihood(feature_matrix, sentiment, coefficients):
    indicator = (sentiment==+1)
    scores = np.dot(feature_matrix, coefficients)
    logexp = np.log(1. + np.exp(-scores))

    # Simple check to prevent overflow
    mask = np.isinf(logexp)
    logexp[mask] = -scores[mask]

    lp = np.sum((indicator-1)*scores - logexp)
    return lp


I am working on the file "Classification/Week 2/Assignment 1/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Load review dataset
For this assignment, we will use a subset of the Amazon product review dataset. The subset was chosen to contain similar numbers of positive and negative reviews, as the original dataset consisted primarily of positive reviews.
Can you write Python code for it?


I am working on the file "Classification/Week 2/Assignment 1/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Measuring accuracy
We will now measure the classification accuracy of the model. Recall from the lecture that the classification accuracy can be computed as follows:
$$
\mbox{accuracy} = \frac{\mbox{# correctly classified data points}}{\mbox{# total data points}}
$$
Complete the following code block to compute the accuracy of the model.
Can you write Python code for it?

num_mistakes =  (sentiment != predictions).sum()
accuracy = 1. * (len(products) - num_mistakes) / len(products)
print "-----------------------------------------------------"
print '# Reviews   correctly classified =', len(products) - num_mistakes
print '# Reviews incorrectly classified =', num_mistakes
print '# Reviews total                  =', len(products)
print "-----------------------------------------------------"
print 'Accuracy = %.2f' % accuracy


I am working on the file "Classification/Week 2/Assignment 1/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Note: For this assignment, we eliminated class imbalance by choosing
a subset of the data with a similar number of positive and negative reviews.
Apply text cleaning on the review data
In this section, we will perform some simple feature cleaning using SFrames. The last assignment used all words in building bag-of-words features, but here we limit ourselves to 193 words (for simplicity). We compiled a list of 193 most frequent words into a JSON file.
Now, we will load these words from this JSON file:
Can you write Python code for it?

import json
with open('important_words.json', 'r') as f: # Reads the list of most frequent words
    important_words = json.load(f)
important_words = [str(s) for s in important_words]

print important_words


I am working on the file "Classification/Week 2/Assignment 1/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Now we proceed with Step 2. For each word in important_words, we compute a count for the number of times the word occurs in the review. We will store this count in a separate column (one for each word). The result of this feature processing is a single column for each word in important_words which keeps a count of the number of times the respective word occurs in the review text.
Note: There are several ways of doing this. In this assignment, we use the built-in count function for Python lists. Each review string is first split into individual words and the number of occurances of a given word is counted.
Can you write Python code for it?

for word in important_words:
    products[word] = products['review_clean'].apply(lambda s : s.split().count(word))


I am working on the file "Classification/Week 2/Assignment 1/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Now, we will perform 2 simple data transformations:

Remove punctuation using Python's built-in string functionality.
Compute word counts (only for important_words)

We start with Step 1 which can be done as follows:
Can you write Python code for it?

def remove_punctuation(text):
    import string
    return text.translate(None, string.punctuation)

products['review_clean'] = products['review'].apply(remove_punctuation)


I am working on the file "Classification/Week 2/Assignment 1/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Now, word_coefficient_tuples contains a sorted list of (word, coefficient_value) tuples. The first 10 elements in this list correspond to the words that are most positive.
Ten "most positive" words
Now, we compute the 10 words that have the most positive coefficient values. These words are associated with positive sentiment.
Can you write Python code for it?


I am working on the file "Classification/Week 2/Assignment 1/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Now, write some code to compute the number of product reviews that contain the word perfect.
Hint:
* First create a column called contains_perfect which is set to 1 if the count of the word perfect (stored in column perfect) is >= 1.
* Sum the number of 1s in the column contains_perfect.
Can you write Python code for it?

products['contains_perfect'] = products['perfect'].apply(lambda x: 1 if x >= 1 else 0)
products['contains_perfect'].sum()


I am working on the file "Classification/Week 2/Assignment 1/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Quiz Question. How many reviews contain the word perfect?
Convert SFrame to NumPy array
As you have seen previously, NumPy is a powerful library for doing matrix manipulation. Let us convert our data to matrices and then implement our algorithms with matrices.
First, make sure you can perform the following import.
Can you write Python code for it?


I am working on the file "Classification/Week 2/Assignment 1/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Quiz Question: How many features are there in the feature_matrix?
 Quiz Question: Assuming that the intercept is present, how does the number of features in feature_matrix relate to the number of features in the logistic regression model?
Now, let us see what the sentiment column looks like:
Can you write Python code for it?


I am working on the file "Classification/Week 2/Assignment 1/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Quiz question:  How many reviews were predicted to have positive sentiment?
Can you write Python code for it?


I am working on the file "Classification/Week 2/Assignment 1/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Quiz question: As each iteration of gradient ascent passes, does the log likelihood increase or decrease?
Predicting sentiments
Recall from lecture that class predictions for a data point $\mathbf{x}$ can be computed from the coefficients $\mathbf{w}$ using the following formula:
$$
\hat{y}_i =
\left{
\begin{array}{ll}
      +1 & \mathbf{x}_i^T\mathbf{w} > 0 \
      -1 & \mathbf{x}_i^T\mathbf{w} \leq 0 \
\end{array}
\right.
$$
Now, we will write some code to compute class predictions. We will do this in two steps:
* Step 1: First compute the scores using feature_matrix and coefficients using a dot product.
* Step 2: Using the formula above, compute the class predictions from the scores.
Step 1 can be implemented as follows:
Can you write Python code for it?

# Compute the scores as a dot product between feature_matrix and coefficients.
scores = np.dot(feature_matrix, coefficients)


I am working on the file "Classification/Week 2/Assignment 1/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Quiz question: What is the accuracy of the model on predictions made above? (round to 2 digits of accuracy)
Which words contribute most to positive & negative sentiments?
Recall that in Module 2 assignment, we were able to compute the "most positive words". These are words that correspond most strongly with positive reviews. In order to do this, we will first do the following:
* Treat each coefficient as a tuple, i.e. (word, coefficient_value).
* Sort all the (word, coefficient_value) tuples by coefficient_value in descending order.
Can you write Python code for it?

coefficients = list(coefficients[1:]) # exclude intercept
word_coefficient_tuples = [(word, coefficient) for word, coefficient in zip(important_words, coefficients)]
word_coefficient_tuples = sorted(word_coefficient_tuples, key=lambda x:x[1], reverse=True)


I am working on the file "Classification/Week 2/Assignment 1/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Taking gradient steps
Now we are ready to implement our own logistic regression. All we have to do is to write a gradient ascent function that takes gradient steps towards the optimum.
Complete the following function to solve the logistic regression model using gradient ascent:
Can you write Python code for it?

from math import sqrt

def logistic_regression(feature_matrix, sentiment, initial_coefficients, step_size, max_iter):
    coefficients = np.array(initial_coefficients) # make sure it's a numpy array
    for itr in xrange(max_iter):

        # Predict P(y_i = +1|x_i,w) using your predict_probability() function
        predictions = predict_probability(feature_matrix, coefficients)

        # Compute indicator value for (y_i = +1)
        indicator = (sentiment==+1)

        # Compute the errors as indicator - predictions
        errors = indicator - predictions
        for j in xrange(len(coefficients)): # loop over each coefficient

            # Recall that feature_matrix[:,j] is the feature column associated with coefficients[j].
            # Compute the derivative for coefficients[j]. Save it in a variable called derivative
            derivative = feature_derivative(errors, feature_matrix[:,j])

            # add the step size times the derivative to the current coefficient
            coefficients[j] = coefficients[j] + (step_size * derivative)

        # Checking whether log likelihood is increasing
        if itr <= 15 or (itr <= 100 and itr % 10 == 0) or (itr <= 1000 and itr % 100 == 0) \
        or (itr <= 10000 and itr % 1000 == 0) or itr % 10000 == 0:
            lp = compute_log_likelihood(feature_matrix, sentiment, coefficients)
            print 'iteration %*d: log likelihood of observed labels = %.8f' % \
                (int(np.ceil(np.log10(max_iter))), itr, lp)
    return coefficients


I am working on the file "Classification/Week 2/Assignment 1/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
The SFrame products now contains one column for each of the 193 important_words. As an example, the column perfect contains a count of the number of times the word perfect occurs in each of the reviews.
Can you write Python code for it?


I am working on the file "Classification/Week 2/Assignment 1/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
We now provide you with a function that extracts columns from an SFrame and converts them into a NumPy array. Two arrays are returned: one representing features and another representing class labels. Note that the feature matrix includes an additional column 'intercept' to take account of the intercept term.
Can you write Python code for it?

def get_numpy_data(data_sframe, features, label):
    data_sframe['intercept'] = 1
    features = ['intercept'] + features
    features_sframe = data_sframe[features]
    feature_matrix = features_sframe.to_numpy()
    label_sarray = data_sframe[label]
    label_array = label_sarray.to_numpy()
    return(feature_matrix, label_array)


I am working on the file "Classification/Week 2/Assignment 2/module-4-linear-classifier-regularization-assignment-blank.ipynb".
The first task is:
Adding  L2 penalty
Let us now work on extending logistic regression with L2 regularization. As discussed in the lectures, the L2 regularization is particularly useful in preventing overfitting. In this assignment, we will explore L2 regularization in detail.
Recall from lecture and the previous assignment that for logistic regression without an L2 penalty, the derivative of the log likelihood function is:
$$
\frac{\partial\ell}{\partial w_j} = \sum_{i=1}^N h_j(\mathbf{x}_i)\left(\mathbf{1}[y_i = +1] - P(y_i = +1 | \mathbf{x}_i, \mathbf{w})\right)
$$
 Adding L2 penalty to the derivative
It takes only a small modification to add a L2 penalty. All terms indicated in red refer to terms that were added due to an L2 penalty.

Recall from the lecture that the link function is still the sigmoid:
$$
P(y_i = +1 | \mathbf{x}_i,\mathbf{w}) = \frac{1}{1 + \exp(-\mathbf{w}^T h(\mathbf{x}_i))},
$$
We add the L2 penalty term to the per-coefficient derivative of log likelihood:
$$
\frac{\partial\ell}{\partial w_j} = \sum_{i=1}^N h_j(\mathbf{x}_i)\left(\mathbf{1}[y_i = +1] - P(y_i = +1 | \mathbf{x}_i, \mathbf{w})\right) \color{red}{-2\lambda w_j }
$$

The per-coefficient derivative for logistic regression with an L2 penalty is as follows:
$$
\frac{\partial\ell}{\partial w_j} = \sum_{i=1}^N h_j(\mathbf{x}i)\left(\mathbf{1}[y_i = +1] - P(y_i = +1 | \mathbf{x}_i, \mathbf{w})\right) \color{red}{-2\lambda w_j }
$$
and for the intercept term, we have
$$
\frac{\partial\ell}{\partial w_0} = \sum{i=1}^N h_0(\mathbf{x}_i)\left(\mathbf{1}[y_i = +1] - P(y_i = +1 | \mathbf{x}_i, \mathbf{w})\right)
$$
Note: As we did in the Regression course, we do not apply the L2 penalty on the intercept. A large intercept does not necessarily indicate overfitting because the intercept is not associated with any particular feature.
Write a function that computes the derivative of log likelihood with respect to a single coefficient $w_j$. Unlike its counterpart in the last assignment, the function accepts five arguments:
 * errors vector containing...
Can you write Python code for it?

def feature_derivative_with_L2(errors, feature, coefficient, l2_penalty, feature_is_constant):

    # Compute the dot product of errors and feature
    derivative = sum(feature * errors)

    # add L2 penalty term for any feature that isn't the intercept.
    if not feature_is_constant:
        derivative = derivative - (2 * l2_penalty * coefficient)

    return derivative


I am working on the file "Classification/Week 2/Assignment 2/module-4-linear-classifier-regularization-assignment-blank.ipynb".
The first task is:
Below, we compare the accuracy on the training data and validation data for all the models that were trained in this assignment.  We first calculate the accuracy values and then build a simple report summarizing the performance for the various models.
Can you write Python code for it?

train_accuracy = {}
train_accuracy[0]   = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_0_penalty)
train_accuracy[4]   = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_4_penalty)
train_accuracy[10]  = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_10_penalty)
train_accuracy[1e2] = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_1e2_penalty)
train_accuracy[1e3] = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_1e3_penalty)
train_accuracy[1e5] = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_1e5_penalty)

validation_accuracy = {}
validation_accuracy[0]   = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_0_penalty)
validation_accuracy[4]   = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_4_penalty)
validation_accuracy[10]  = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_10_penalty)
validation_accuracy[1e2] = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_1e2_penalty)
validation_accuracy[1e3] = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_1e3_penalty)
validation_accuracy[1e5] = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_1e5_penalty)

# Build a simple report
for key in sorted(validation_accuracy.keys()):
    print "L2 penalty = %g" % key
    print "train accuracy = %s, validation_accuracy = %s" % (train_accuracy[key], validation_accuracy[key])
    print "--------------------------------------------------------------------------------"


I am working on the file "Classification/Week 2/Assignment 2/module-4-linear-classifier-regularization-assignment-blank.ipynb".
The first task is:
Just like we did previously, we will work with a hand-curated list of important words extracted from the review data. We will also perform 2 simple data transformations:

Remove punctuation using Python's built-in string functionality.
Compute word counts (only for the important_words)

Refer to Module 3 assignment for more details.
Can you write Python code for it?

# The same feature processing (same as the previous assignments)
# ---------------------------------------------------------------
import json
with open('important_words.json', 'r') as f: # Reads the list of most frequent words
    important_words = json.load(f)
important_words = [str(s) for s in important_words]

def remove_punctuation(text):
    import string
    return text.translate(None, string.punctuation)

# Remove punctuation.
products['review_clean'] = products['review'].apply(remove_punctuation)

# Split out the words into individual columns
for word in important_words:
    products[word] = products['review_clean'].apply(lambda s : s.split().count(word))


I am working on the file "Classification/Week 2/Assignment 2/module-4-linear-classifier-regularization-assignment-blank.ipynb".
The first task is:
Load and process review dataset
For this assignment, we will use the same subset of the Amazon product review dataset that we used in Module 3 assignment. The subset was chosen to contain similar numbers of positive and negative reviews, as the original dataset consisted of mostly positive reviews.
Can you write Python code for it?


I am working on the file "Classification/Week 2/Assignment 2/module-4-linear-classifier-regularization-assignment-blank.ipynb".
The first task is:
Now, let us take a look at what the dataset looks like (Note: This may take a few minutes).
Can you write Python code for it?


I am working on the file "Classification/Week 2/Assignment 2/module-4-linear-classifier-regularization-assignment-blank.ipynb".
The first task is:
Now, let's run the function add_coefficients_to_table for each of the L2 penalty strengths.
Can you write Python code for it?

add_coefficients_to_table(coefficients_0_penalty, 'coefficients [L2=0]')
add_coefficients_to_table(coefficients_4_penalty, 'coefficients [L2=4]')
add_coefficients_to_table(coefficients_10_penalty, 'coefficients [L2=10]')
add_coefficients_to_table(coefficients_1e2_penalty, 'coefficients [L2=1e2]')
add_coefficients_to_table(coefficients_1e3_penalty, 'coefficients [L2=1e3]')
add_coefficients_to_table(coefficients_1e5_penalty, 'coefficients [L2=1e5]')


I am working on the file "Classification/Week 2/Assignment 2/module-4-linear-classifier-regularization-assignment-blank.ipynb".
The first task is:
Quiz question: In the code above, was the intercept term regularized?
To verify the correctness of the gradient ascent algorithm, we provide a function for computing log likelihood (which we recall from the last assignment was a topic detailed in an advanced optional video, and used here for its numerical stability).
$$\ell\ell(\mathbf{w}) = \sum_{i=1}^N \Big( (\mathbf{1}[y_i = +1] - 1)\mathbf{w}^T h(\mathbf{x}_i) - \ln\left(1 + \exp(-\mathbf{w}^T h(\mathbf{x}_i))\right) \Big) \color{red}{-\lambda\|\mathbf{w}\|_2^2} $$
Can you write Python code for it?

def compute_log_likelihood_with_L2(feature_matrix, sentiment, coefficients, l2_penalty):
    indicator = (sentiment==+1)
    scores = np.dot(feature_matrix, coefficients)

    lp = np.sum((indicator-1)*scores - np.log(1. + np.exp(-scores))) - l2_penalty*np.sum(coefficients[1:]**2)

    return lp


I am working on the file "Classification/Week 3/Assignment 1/module-5-decision-tree-assignment-1-blank.ipynb".
The first task is:
Evaluating accuracy of the decision tree model
Recall that the accuracy is defined as follows:
$$
\mbox{accuracy} = \frac{\mbox{# correctly classified examples}}{\mbox{# total examples}}
$$
Let us start by evaluating the accuracy of the small_model and decision_tree_model on the training data
Can you write Python code for it?

print small_model.evaluate(train_data)['accuracy']
print decision_tree_model.evaluate(train_data)['accuracy']


I am working on the file "Classification/Week 3/Assignment 1/module-5-decision-tree-assignment-1-blank.ipynb".
The first task is:
Exploring some features
Let's quickly explore what the dataset looks like. First, let's print out the column names to see what features we have in this dataset.
Can you write Python code for it?


I am working on the file "Classification/Week 3/Assignment 1/module-5-decision-tree-assignment-1-blank.ipynb".
The first task is:
False negatives are predictions where the model predicts -1 but the true label is +1. Complete the following code block for the number of false negatives:
Quiz Question: Let us assume that each mistake costs money:
* Assume a cost of \$10,000 per false negative.
* Assume a cost of \$20,000 per false positive.
What is the total cost of mistakes made by decision_tree_model on validation_data?
Can you write Python code for it?

10000 * false_negatives

20000 * false_positives

(10000 * false_negatives) + (20000 * false_positives)


I am working on the file "Classification/Week 3/Assignment 1/module-5-decision-tree-assignment-1-blank.ipynb".
The first task is:
False positives are predictions where the model predicts +1 but the true label is -1. Complete the following code block for the number of false positives:
Can you write Python code for it?

false_positives = 0
false_negatives = 0
for item in xrange(len(validation_data)):
    if predictions[item] != validation_data['safe_loans'][item]:
        if predictions[item] == 1:
            false_positives += 1
        else:
            false_negatives += 1

print false_positives
print false_negatives


I am working on the file "Classification/Week 3/Assignment 1/module-5-decision-tree-assignment-1-blank.ipynb".
The first task is:
Let's visualize the small tree here to do the traversing for this data point.
Can you write Python code for it?


I am working on the file "Classification/Week 3/Assignment 1/module-5-decision-tree-assignment-1-blank.ipynb".
The first task is:
Load LendingClub dataset
We will be using a dataset from the LendingClub. A parsed and cleaned form of the dataset is availiable here. Make sure you download the dataset before running the following command.
Can you write Python code for it?


I am working on the file "Classification/Week 3/Assignment 1/module-5-decision-tree-assignment-1-blank.ipynb".
The first task is:
Now, let us explore the distribution of the column safe_loans. This gives us a sense of how many safe and risky loans are present in the dataset.
Can you write Python code for it?


I am working on the file "Classification/Week 3/Assignment 1/module-5-decision-tree-assignment-1-blank.ipynb".
The first task is:
One way to combat class imbalance is to undersample the larger class until the class distribution is approximately half and half. Here, we will undersample the larger class (safe loans) in order to balance out our dataset. This means we are throwing away many data points. We used seed=1 so everyone gets the same results.
Can you write Python code for it?

# Since there are fewer risky loans than safe loans, find the ratio of the sizes
# and use that percentage to undersample the safe loans.
percentage = len(risky_loans_raw)/float(len(safe_loans_raw))

risky_loans = risky_loans_raw
safe_loans = safe_loans_raw.sample(percentage, seed=1)

# Append the risky_loans with the downsampled version of safe_loans
loans_data = risky_loans.append(safe_loans)


I am working on the file "Classification/Week 3/Assignment 1/module-5-decision-tree-assignment-1-blank.ipynb".
The first task is:
Quiz Question: Notice that the probability preditions are the exact same for the 2nd and 3rd loans. Why would this happen?
Answer: The same leaf node
Visualize the prediction on a tree
Note that you should be able to look at the small tree, traverse it yourself, and visualize the prediction being made. Consider the following point in the sample_validation_data
Can you write Python code for it?


I am working on the file "Classification/Week 3/Assignment 1/module-5-decision-tree-assignment-1-blank.ipynb".
The first task is:
Quiz Question: What is the accuracy of decision_tree_model on the validation set, rounded to the nearest .01?
Evaluating accuracy of a complex decision tree model
Here, we will train a large decision tree with max_depth=10. This will allow the learned tree to become very deep, and result in a very complex model. Recall that in lecture, we prefer simpler models with similar predictive power. This will be an example of a more complicated model which has similar predictive power, i.e. something we don't want.
Can you write Python code for it?

big_model = graphlab.decision_tree_classifier.create(train_data, validation_set=None,
                   target = target, features = features, max_depth = 10)


I am working on the file "Classification/Week 3/Assignment 1/module-5-decision-tree-assignment-1-blank.ipynb".
The first task is:
Quiz Question: What percentage of the predictions on sample_validation_data did decision_tree_model get correct?
Explore probability predictions
For each row in the sample_validation_data, what is the probability (according decision_tree_model) of a loan being classified as safe?
Hint: Set output_type='probability' to make probability predictions using decision_tree_model on sample_validation_data:
Can you write Python code for it?


I am working on the file "Classification/Week 3/Assignment 1/module-5-decision-tree-assignment-1-blank.ipynb".
The first task is:
Quiz Question: Which loan has the highest probability of being classified as a safe loan?
Checkpoint: Can you verify that for all the predictions with probability >= 0.5, the model predicted the label +1?
Tricky predictions!
Now, we will explore something pretty interesting. For each row in the sample_validation_data, what is the probability (according to small_model) of a loan being classified as safe?
Hint: Set output_type='probability' to make probability predictions using small_model on sample_validation_data:
Can you write Python code for it?


I am working on the file "Classification/Week 3/Assignment 1/module-5-decision-tree-assignment-1-blank.ipynb".
The first task is:
This feature describes whether the loanee is mortaging, renting, or owns a home. We can see that a small percentage of the loanees own a home.
Exploring the target column
The target column (label column) of the dataset that we are interested in is called bad_loans. In this column 1 means a risky (bad) loan 0 means a safe  loan.
In order to make this more intuitive and consistent with the lectures, we reassign the target to be:
* +1 as a safe  loan,
* -1 as a risky (bad) loan.
We put this in a new column called safe_loans.
Can you write Python code for it?

# safe_loans =  1 => safe
# safe_loans = -1 => risky
loans['safe_loans'] = loans['bad_loans'].apply(lambda x : +1 if x==0 else -1)
loans = loans.remove_column('bad_loans')


I am working on the file "Classification/Week 3/Assignment 1/module-5-decision-tree-assignment-1-blank.ipynb".
The first task is:
Use decision tree to build a classifier
Now, let's use the built-in GraphLab Create decision tree learner to create a loan prediction model on the training data. (In the next assignment, you will implement your own decision tree learning algorithm.)  Our feature columns and target column have already been decided above. Use validation_set=None to get the same results as everyone else.
Can you write Python code for it?

decision_tree_model = graphlab.decision_tree_classifier.create(train_data, validation_set=None,
                                target = target, features = features)


I am working on the file "Classification/Week 3/Assignment 1/module-5-decision-tree-assignment-1-blank.ipynb".
The first task is:
You should have:
* Around 81% safe loans
* Around 19% risky loans
It looks like most of these loans are safe loans (thankfully). But this does make our problem of identifying risky loans challenging.
Features for the classification algorithm
In this assignment, we will be using a subset of features (categorical and numeric). The features we will be using are described in the code comments below. If you are a finance geek, the LendingClub website has a lot more details about these features.
Can you write Python code for it?

features = ['grade',                     # grade of the loan
            'sub_grade',                 # sub-grade of the loan
            'short_emp',                 # one year or less of employment
            'emp_length_num',            # number of years of employment
            'home_ownership',            # home_ownership status: own, mortgage or rent
            'dti',                       # debt to income ratio
            'purpose',                   # the purpose of the loan
            'term',                      # the term of the loan
            'last_delinq_none',          # has borrower had a delinquincy
            'last_major_derog_none',     # has borrower had 90 day or worse rating
            'revol_util',                # percent of available credit being used
            'total_rec_late_fee',        # total late fees received to day
           ]

target = 'safe_loans'                   # prediction target (y) (+1 means safe, -1 is risky)

# Extract the feature columns and target column
loans = loans[features + [target]]


I am working on the file "Classification/Week 3/Assignment 2/module-5-decision-tree-assignment-2-blank.ipynb".
The first task is:
Because there are several steps in this assignment, we have introduced some stopping points where you can check your code and make sure it is correct before proceeding. To test your intermediate_node_num_mistakes function, run the following code until you get a Test passed!, then you should proceed. Otherwise, you should spend some time figuring out where things went wrong.
Can you write Python code for it?

# Test case 1
example_labels = graphlab.SArray([-1, -1, 1, 1, 1])
if intermediate_node_num_mistakes(example_labels) == 2:
    print 'Test passed!'
else:
    print 'Test 1 failed... try again!'

# Test case 2
example_labels = graphlab.SArray([-1, -1, 1, 1, 1, 1, 1])
if intermediate_node_num_mistakes(example_labels) == 2:
    print 'Test passed!'
else:
    print 'Test 2 failed... try again!'

# Test case 3
example_labels = graphlab.SArray([-1, -1, -1, -1, -1, 1, 1])
if intermediate_node_num_mistakes(example_labels) == 2:
    print 'Test passed!'
else:
    print 'Test 3 failed... try again!'


I am working on the file "Classification/Week 3/Assignment 2/module-5-decision-tree-assignment-2-blank.ipynb".
The first task is:
Build the tree!
Now that all the tests are passing, we will train a tree model on the train_data. Limit the depth to 6 (max_depth = 6) to make sure the algorithm doesn't run for too long. Call this tree my_decision_tree.
Warning: This code block may take 1-2 minutes to learn.
Can you write Python code for it?

# Make sure to cap the depth at 6 by using max_depth = 6
my_decision_tree = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6)


I am working on the file "Classification/Week 3/Assignment 2/module-5-decision-tree-assignment-2-blank.ipynb".
The first task is:
Building the tree
With the above functions implemented correctly, we are now ready to build our decision tree. Each node in the decision tree is represented as a dictionary which contains the following keys and possible values:
{
   'is_leaf'            : True/False.
   'prediction'         : Prediction at the leaf node.
   'left'               : (dictionary corresponding to the left tree).
   'right'              : (dictionary corresponding to the right tree).
   'splitting_feature'  : The feature that this node splits on.
}

First, we will write a function that creates a leaf node given a set of target values. Fill in the places where you find ## YOUR CODE HERE. There are three places in this function for you to fill in.
Can you write Python code for it?

def create_leaf(target_values):

    # Create a leaf node
    leaf = {'splitting_feature' : None,
            'left' : None,
            'right' : None,
            'is_leaf': True,
            'prediction': None}

    # Count the number of data points that are +1 and -1 in this node.
    num_ones = len(target_values[target_values == +1])
    num_minus_ones = len(target_values[target_values == -1])

    # For the leaf node, set the prediction to be the majority class.
    # Store the predicted class (1 or -1) in leaf['prediction']
    if num_ones > num_minus_ones:
        leaf['prediction'] = 1
    else:
        leaf['prediction'] = -1

    # Return the leaf node
    return leaf


I am working on the file "Classification/Week 3/Assignment 2/module-5-decision-tree-assignment-2-blank.ipynb".
The first task is:
Decision tree implementation
In this section, we will implement binary decision trees from scratch. There are several steps involved in building a decision tree. For that reason, we have split the entire assignment into several sections.
Function to count number of mistakes while predicting majority class
Recall from the lecture that prediction at an intermediate node works by predicting the majority class for all data points that belong to this node.
Now, we will write a function that calculates the number of missclassified examples when predicting the majority class. This will be used to help determine which feature is the best to split on at a given node of the tree.
Note: Keep in mind that in order to compute the number of mistakes for a majority classifier, we only need the label (y values) of the data points in the node.
 Steps to follow :
*  Step 1: Calculate the number of safe loans and risky loans.
*  Step 2: Since we are assuming majority class prediction, all the data points that are not in the majority class are considered mistakes.
*  Step 3: Return the number of mistakes.
Now, let us write the function intermediate_node_num_mistakes which computes the number of misclassified examples of an intermediate node given the set of labels (y values) of the data points contained in the node. Fill in the places where you find ## YOUR CODE HERE. There are three places in this function for you to fill in.
Can you write Python code for it?

def intermediate_node_num_mistakes(labels_in_node):
    # Corner case: If labels_in_node is empty, return 0
    if len(labels_in_node) == 0:
        return 0

    # Count the number of 1's (safe loans)
    safe_loans = len(labels_in_node[labels_in_node == +1])

    # Count the number of -1's (risky loans)
    risky_loans = len(labels_in_node[labels_in_node == -1])

    # Return the number of mistakes that the majority classifier makes.
    if safe_loans > risky_loans:
        return risky_loans
    else:
        return safe_loans


I am working on the file "Classification/Week 3/Assignment 2/module-5-decision-tree-assignment-2-blank.ipynb".
The first task is:
Exploring the left subtree of the left subtree
Can you write Python code for it?


I am working on the file "Classification/Week 3/Assignment 2/module-5-decision-tree-assignment-2-blank.ipynb".
The first task is:
Here is a recursive function to count the nodes in your tree:
Can you write Python code for it?

def count_nodes(tree):
    if tree['is_leaf']:
        return 1
    return 1 + count_nodes(tree['left']) + count_nodes(tree['right'])


I am working on the file "Classification/Week 3/Assignment 2/module-5-decision-tree-assignment-2-blank.ipynb".
The first task is:
Let's add some annotations to our prediction to see what the prediction path was that lead to this predicted class:
Can you write Python code for it?


I am working on the file "Classification/Week 3/Assignment 2/module-5-decision-tree-assignment-2-blank.ipynb".
The first task is:
Let's explore what one of these columns looks like:
Can you write Python code for it?


I am working on the file "Classification/Week 3/Assignment 2/module-5-decision-tree-assignment-2-blank.ipynb".
The first task is:
Let's explore what the dataset looks like.
Can you write Python code for it?


I am working on the file "Classification/Week 3/Assignment 2/module-5-decision-tree-assignment-2-blank.ipynb".
The first task is:
Let's see what the feature columns look like now:
Can you write Python code for it?

features = loans_data.column_names()
features.remove('safe_loans')  # Remove the response variable
features

print "Number of features (after binarizing categorical variables) = %s" % len(features)


I am working on the file "Classification/Week 3/Assignment 2/module-5-decision-tree-assignment-2-blank.ipynb".
The first task is:
Load the lending club dataset
We will be using the same LendingClub dataset as in the previous assignment.
Can you write Python code for it?


I am working on the file "Classification/Week 3/Assignment 2/module-5-decision-tree-assignment-2-blank.ipynb".
The first task is:
Note: There are many approaches for dealing with imbalanced data, including some where we modify the learning algorithm. These approaches are beyond the scope of this course, but some of them are reviewed in this paper. For this assignment, we use the simplest possible approach, where we subsample the overly represented class to get a more balanced dataset. In general, and especially when the data is highly imbalanced, we recommend using more advanced methods.
Transform categorical data into binary features
In this assignment, we will implement binary decision trees (decision trees for binary features, a specific case of categorical variables taking on two values, e.g., true/false). Since all of our features are currently categorical features, we want to turn them into binary features.
For instance, the home_ownership feature represents the home ownership status of the loanee, which is either own, mortgage or rent. For example, if a data point has the feature
{'home_ownership': 'RENT'}
we want to turn this into three features:
{
   'home_ownership = OWN'      : 0,
   'home_ownership = MORTGAGE' : 0,
   'home_ownership = RENT'     : 1
 }
Since this code requires a few Python and GraphLab tricks, feel free to use this block of code as is. Refer to the API documentation for a deeper understanding.
Can you write Python code for it?

loans_data = risky_loans.append(safe_loans)
for feature in features:
    loans_data_one_hot_encoded = loans_data[feature].apply(lambda x: {x: 1})
    loans_data_unpacked = loans_data_one_hot_encoded.unpack(column_name_prefix=feature)

    # Change None's to 0's
    for column in loans_data_unpacked.column_names():
        loans_data_unpacked[column] = loans_data_unpacked[column].fillna(0)

    loans_data.remove_column(feature)
    loans_data.add_columns(loans_data_unpacked)


I am working on the file "Classification/Week 3/Assignment 2/module-5-decision-tree-assignment-2-blank.ipynb".
The first task is:
Quiz question: What is the path of the first 3 feature splits considered along the left-most branch of my_decision_tree?
Quiz question: What is the path of the first 3 feature splits considered along the right-most branch of my_decision_tree?
Can you write Python code for it?


I am working on the file "Classification/Week 3/Assignment 2/module-5-decision-tree-assignment-2-blank.ipynb".
The first task is:
Quiz question: What was the feature that my_decision_tree first split on while making the prediction for test_data[0]?
 Quiz question: What was the first feature that lead to a right split of test_data[0]?
 Quiz question: What was the last feature split on before reaching a leaf node for test_data[0]?
Evaluating your decision tree
Now, we will write a function to evaluate a decision tree by computing the classification error of the tree on the given dataset.
Again, recall that the classification error is defined as follows:
$$
\mbox{classification error} = \frac{\mbox{# mistakes}}{\mbox{# total examples}}
$$
Now, write a function called evaluate_classification_error that takes in as input:
1. tree (as described above)
2. data (an SFrame)
This function should return a prediction (class label) for each row in data using the decision tree. Fill in the places where you find ## YOUR CODE HERE. There is one place in this function for you to fill in.
Can you write Python code for it?

def evaluate_classification_error(tree, data):
    # Apply the classify(tree, x) to each row in your data
    prediction = data.apply(lambda x: classify(tree, x))

    # Once you've made the predictions, calculate the classification error and return it
    ## YOUR CODE HERE
    num_errors = 0
    for item in xrange(len(data)):
        if data['safe_loans'][item] != prediction[item]:
            num_errors += 1

    return num_errors / float(len(data))
    #return len(data[(data['safe_loans'] == +1) & (prediction == +1)]) / float(len(data))


I am working on the file "Classification/Week 3/Assignment 2/module-5-decision-tree-assignment-2-blank.ipynb".
The first task is:
Subsample dataset to make sure classes are balanced
Just as we did in the previous assignment, we will undersample the larger class (safe loans) in order to balance out our dataset. This means we are throwing away many data points. We use seed=1 so everyone gets the same results.
Can you write Python code for it?

safe_loans_raw = loans[loans[target] == 1]
risky_loans_raw = loans[loans[target] == -1]

# Since there are less risky loans than safe loans, find the ratio of the sizes
# and use that percentage to undersample the safe loans.
percentage = len(risky_loans_raw)/float(len(safe_loans_raw))
safe_loans = safe_loans_raw.sample(percentage, seed = 1)
risky_loans = risky_loans_raw
loans_data = risky_loans.append(safe_loans)

print "Percentage of safe loans                 :", len(safe_loans) / float(len(loans_data))
print "Percentage of risky loans                :", len(risky_loans) / float(len(loans_data))
print "Total number of loans in our new dataset :", len(loans_data)


I am working on the file "Classification/Week 3/Assignment 2/module-5-decision-tree-assignment-2-blank.ipynb".
The first task is:
This column is set to 1 if the loan grade is A and 0 otherwise.
Checkpoint: Make sure the following answers match up.
Can you write Python code for it?

print "Total number of grade.A loans : %s" % loans_data['grade.A'].sum()
print "Expexted answer               : 6422"


I am working on the file "Classification/Week 3/Assignment 2/module-5-decision-tree-assignment-2-blank.ipynb".
The first task is:
Train-test split
We split the data into a train test split with 80% of the data in the training set and 20% of the data in the test set. We use seed=1 so that everyone gets the same result.
Can you write Python code for it?

train_data, test_data = loans_data.random_split(.8, seed=1)


I am working on the file "Classification/Week 3/Assignment 2/module-5-decision-tree-assignment-2-blank.ipynb".
The first task is:
We have provided a function that learns the decision tree recursively and implements 3 stopping conditions:
1. Stopping condition 1: All data points in a node are from the same class.
2. Stopping condition 2: No more features to split on.
3. Additional stopping condition: In addition to the above two stopping conditions covered in lecture, in this assignment we will also consider a stopping condition based on the max_depth of the tree. By not letting the tree grow too deep, we will save computational effort in the learning process.
Now, we will write down the skeleton of the learning algorithm. Fill in the places where you find ## YOUR CODE HERE. There are seven places in this function for you to fill in.
Can you write Python code for it?

def decision_tree_create(data, features, target, current_depth = 0, max_depth = 10):
    remaining_features = features[:] # Make a copy of the features.

    target_values = data[target]
    print "--------------------------------------------------------------------"
    print "Subtree, depth = %s (%s data points)." % (current_depth, len(target_values))

    # Stopping condition 1
    # (Check if there are mistakes at current node.
    # Recall you wrote a function intermediate_node_num_mistakes to compute this.)
    if intermediate_node_num_mistakes(target_values) == 0:  ## YOUR CODE HERE
        print "Stopping condition 1 reached."
        # If not mistakes at current node, make current node a leaf node
        return create_leaf(target_values)

    # Stopping condition 2 (check if there are remaining features to consider splitting on)
    if len(remaining_features) == 0:   ## YOUR CODE HERE
        print "Stopping condition 2 reached."
        # If there are no remaining features to consider, make current node a leaf node
        return create_leaf(target_values)

    # Additional stopping condition (limit tree depth)
    if current_depth >= max_depth:  ## YOUR CODE HERE
        print "Reached maximum depth. Stopping for now."
        # If the max tree depth has been reached, make current node a leaf node
        return create_leaf(target_values)

    # Find the best splitting feature (recall the function best_splitting_feature implemented above)
    ## YOUR CODE HERE
    splitting_feature = best_splitting_feature(data, remaining_features, target)

    # Split on the best feature that we found.
    left_split = data[data[splitting_feature] == 0]
    right_split = data[data[splitting_feature] == 1]      ## YOUR CODE HERE
    remaining_features.remove(splitting_feature)
    print "Split on feature %s. (%s, %s)" % (\
                      splitting_feature, len(left_split), len(right_split))

    # Create a leaf node if the split is "perfect"
    if len(left_split) ==...


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
As before, we reassign the labels to have +1 for a safe loan, and -1 for a risky (bad) loan.
Can you write Python code for it?

loans['safe_loans'] = loans['bad_loans'].apply(lambda x : +1 if x==0 else -1)
loans = loans.remove_column('bad_loans')


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Calculate the accuracy of each model (model_4, model_5, or model_6) on the validation set.
Can you write Python code for it?

print "Validation data, classification error (model 4):", evaluate_classification_error(model_4, validation_set)
print "Validation data, classification error (model 5):", evaluate_classification_error(model_5, validation_set)
print "Validation data, classification error (model 6):", evaluate_classification_error(model_6, validation_set)


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Compute the number of nodes in model_1, model_2, and model_3.
Can you write Python code for it?

print "Complexity (model 1):", count_leaves(model_1)
print "Complexity (model 2):", count_leaves(model_2)
print "Complexity (model 3):", count_leaves(model_3)


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Early stopping methods for decision trees
In this section, we will extend the binary tree implementation from the previous assignment in order to handle some early stopping conditions. Recall the 3 early stopping methods that were discussed in lecture:

Reached a maximum depth. (set by parameter max_depth).
Reached a minimum node size. (set by parameter min_node_size).
Don't split if the gain in error reduction is too small. (set by parameter min_error_reduction).

For the rest of this assignment, we will refer to these three as early stopping conditions 1, 2, and 3.
Early stopping condition 1: Maximum depth
Recall that we already implemented the maximum depth stopping condition in the previous assignment. In this assignment, we will experiment with this condition a bit more and also write code to implement the 2nd and 3rd early stopping conditions.
We will be reusing code from the previous assignment and then building upon this.  We will alert you when you reach a function that was part of the previous assignment so that you can simply copy and past your previous code.
Early stopping condition 2: Minimum node size
The function reached_minimum_node_size takes 2 arguments:

The data (from a node)
The minimum number of data points that a node is allowed to split on, min_node_size.

This function simply calculates whether the number of data points at a given node is less than or equal to the specified minimum node size. This function will be used to detect this early stopping condition in the decision_tree_create function.
Fill in the parts of the function below where you find ## YOUR CODE HERE.  There is one instance in the function below.
Can you write Python code for it?

def reached_minimum_node_size(data, min_node_size):
    # Return True if the number of data points is less than or equal to the minimum node size.
    ## YOUR CODE HERE
    return (len(data) <= min_node_size)


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Evaluating the models
Let us evaluate the models on the train and validation data. Let us start by evaluating the classification error on the training data:
Can you write Python code for it?

print "Training data, classification error (model 1):", evaluate_classification_error(model_1, train_data)
print "Training data, classification error (model 2):", evaluate_classification_error(model_2, train_data)
print "Training data, classification error (model 3):", evaluate_classification_error(model_3, train_data)


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Finally, recall the function create_leaf from the previous assignment, which creates a leaf node given a set of target values.
Please copy and paste your create_leaf code here.
Can you write Python code for it?


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Here is a function to count the nodes in your tree:
Can you write Python code for it?


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Incorporating new early stopping conditions in binary decision tree implementation
Now, you will implement a function that builds a decision tree handling the three early stopping conditions described in this assignment.  In particular, you will write code to detect early stopping conditions 2 and 3.  You implemented above the functions needed to detect these conditions.  The 1st early stopping condition, max_depth, was implemented in the previous assigment and you will not need to reimplement this.  In addition to these early stopping conditions, the typical stopping conditions of having no mistakes or no more features to split on (which we denote by "stopping conditions" 1 and 2) are also included as in the previous assignment.
Implementing early stopping condition 2: minimum node size:

Step 1: Use the function reached_minimum_node_size that you implemented earlier to write an if condition to detect whether we have hit the base case, i.e., the node does not have enough data points and should be turned into a leaf. Don't forget to use the min_node_size argument.
Step 2: Return a leaf. This line of code should be the same as the other (pre-implemented) stopping conditions.

Implementing early stopping condition 3: minimum error reduction:
Note: This has to come after finding the best splitting feature so we can calculate the error after splitting in order to calculate the error reduction.

Step 1: Calculate the classification error before splitting.  Recall that classification error is defined as:

$$
\text{classification error} = \frac{\text{# mistakes}}{\text{# total examples}}
$$
* Step 2: Calculate the classification error after splitting. This requires calculating the number of mistakes in the left and right splits, and then dividing by the total number of examples.
* Step 3: Use the function error_reduction to that you implemented earlier to write an if condition to detect whether  the reduction in error is less than the constant provided (min_error_reduction). Don't forget to use that argument.
* Step...
Can you write Python code for it?

def decision_tree_create(data, features, target, current_depth = 0,
                         max_depth = 10, min_node_size=1,
                         min_error_reduction=0.0):

    remaining_features = features[:] # Make a copy of the features.

    target_values = data[target]
    print "--------------------------------------------------------------------"
    print "Subtree, depth = %s (%s data points)." % (current_depth, len(target_values))

    # Stopping condition 1: All nodes are of the same type.
    if intermediate_node_num_mistakes(target_values) == 0:
        print "Stopping condition 1 reached. All data points have the same target value."
        return create_leaf(target_values)

    # Stopping condition 2: No more features to split on.
    if remaining_features == []:
        print "Stopping condition 2 reached. No remaining features."
        return create_leaf(target_values)

    # Early stopping condition 1: Reached max depth limit.
    if current_depth >= max_depth:
        print "Early stopping condition 1 reached. Reached maximum depth."
        return create_leaf(target_values)

    # Early stopping condition 2: Reached the minimum node size.
    # If the number of data points is less than or equal to the minimum size, return a leaf.
    if reached_minimum_node_size(data, min_node_size) == True:
        print "Early stopping condition 2 reached. Reached minimum node size."
        return create_leaf(target_values) ## YOUR CODE HERE

    # Find the best splitting feature
    splitting_feature = best_splitting_feature(data, features, target)

    # Split on the best feature that we found.
    left_split = data[data[splitting_feature] == 0]
    right_split = data[data[splitting_feature] == 1]

    # Early stopping condition 3: Minimum error reduction
    # Calculate the error before splitting (number of misclassified examples
    # divided by the total number of examples)
    error_before_split =...


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Let's now recall the prediction path for the decision tree learned in the previous assignment, which we recreated here as my_decision_tree_old.
Can you write Python code for it?

classify(my_decision_tree_old, validation_set[0], annotate = True)


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Let's now train a tree model ignoring early stopping conditions 2 and 3 so that we get the same tree as in the previous assignment.  To ignore these conditions, we set min_node_size=0 and min_error_reduction=-1 (a negative value).
Can you write Python code for it?

my_decision_tree_old = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                                min_node_size = 0, min_error_reduction=-1)


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Making predictions
Recall that in the previous assignment you implemented a function classify to classify a new point x using a given tree.
Please copy and paste your classify code here.
Can you write Python code for it?

def classify(tree, x, annotate = False):
    # if the node is a leaf node.
    if tree['is_leaf']:
        if annotate:
            print "At leaf, predicting %s" % tree['prediction']
        return tree['prediction']
    else:
        # split on feature.
        split_feature_value = x[tree['splitting_feature']]
        if annotate:
            print "Split on %s = %s" % (tree['splitting_feature'], split_feature_value)
        if split_feature_value == 0:
            return classify(tree['left'], x, annotate)
        else:
               ### YOUR CODE HERE
            return classify(tree['right'], x, annotate)


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Note: There are many approaches for dealing with imbalanced data, including some where we modify the learning algorithm. These approaches are beyond the scope of this course, but some of them are reviewed in this paper. For this assignment, we use the simplest possible approach, where we subsample the overly represented class to get a more balanced dataset. In general, and especially when the data is highly imbalanced, we recommend using more advanced methods.
Transform categorical data into binary features
Since we are implementing binary decision trees, we transform our categorical data into binary data using 1-hot encoding, just as in the previous assignment. Here is the summary of that discussion:
For instance, the home_ownership feature represents the home ownership status of the loanee, which is either own, mortgage or rent. For example, if a data point has the feature
{'home_ownership': 'RENT'}
we want to turn this into three features:
{
   'home_ownership = OWN'      : 0,
   'home_ownership = MORTGAGE' : 0,
   'home_ownership = RENT'     : 1
 }
Since this code requires a few Python and GraphLab tricks, feel free to use this block of code as is. Refer to the API documentation for a deeper understanding.
Can you write Python code for it?


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Now evaluate the classification error on the validation data.
Can you write Python code for it?

print "Validation data, classification error (model 1):", evaluate_classification_error(model_1, validation_set)
print "Validation data, classification error (model 2):", evaluate_classification_error(model_2, validation_set)
print "Validation data, classification error (model 3):", evaluate_classification_error(model_3, validation_set)


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Now, evaluate the validation error using my_decision_tree_old.
Can you write Python code for it?


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Now, let us evaluate the models (model_7, model_8, or model_9) on the validation_set.
Can you write Python code for it?

print "Validation data, classification error (model 7):", evaluate_classification_error(model_7, validation_set)
print "Validation data, classification error (model 8):", evaluate_classification_error(model_8, validation_set)
print "Validation data, classification error (model 9):", evaluate_classification_error(model_9, validation_set)


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Now, let's consider the first example of the validation set and see what the my_decision_tree_new model predicts for this data point.
Can you write Python code for it?

validation_set[0]

print 'Predicted class: %s ' % classify(my_decision_tree_new, validation_set[0])


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Now, let's use this function to evaluate the classification error of my_decision_tree_new on the validation_set.
Can you write Python code for it?


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Quiz question: Assume an intermediate node has 6 safe loans and 3 risky loans.  For each of 4 possible features to split on, the error reduction is 0.0, 0.05, 0.1, and 0.14, respectively. If the minimum gain in error reduction parameter is set to 0.2, what should the tree learning algorithm do next?
Grabbing binary decision tree helper functions from past assignment
Recall from the previous assignment that we wrote a function intermediate_node_num_mistakes that calculates the number of misclassified examples when predicting the majority class. This is used to help determine which feature is best to split on at a given node of the tree.
Please copy and paste your code for intermediate_node_num_mistakes here.
Can you write Python code for it?

def intermediate_node_num_mistakes(labels_in_node):
    # Corner case: If labels_in_node is empty, return 0
    if len(labels_in_node) == 0:
        return 0

    # Count the number of 1's (safe loans)
    safe_loans = len(labels_in_node[labels_in_node == +1])

    # Count the number of -1's (risky loans)
    risky_loans = len(labels_in_node[labels_in_node == -1])

    # Return the number of mistakes that the majority classifier makes.
    if safe_loans > risky_loans:
        return risky_loans
    else:
        return safe_loans


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Quiz question: Is the validation error of the new decision tree (using early stopping conditions 2 and 3) lower than, higher than, or the same as that of the old decision tree from the previous assignment?
Exploring the effect of max_depth
We will compare three models trained with different values of the stopping criterion. We intentionally picked models at the extreme ends (too small, just right, and too large).
Train three models with these parameters:

model_1: max_depth = 2 (too small)
model_2: max_depth = 6 (just right)
model_3: max_depth = 14 (may be too large)

For each of these three, we set min_node_size = 0 and min_error_reduction = -1.
 Note: Each tree can take up to a few minutes to train. In particular, model_3 will probably take the longest to train.
Can you write Python code for it?

model_1 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 2,
                                min_node_size = 0, min_error_reduction=-1)
model_2 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                                min_node_size = 0, min_error_reduction=-1)
model_3 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 14,
                                min_node_size = 0, min_error_reduction=-1)


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Quiz question: Which tree has the largest complexity?
Quiz question: Is it always true that the most complex tree will result in the lowest classification error in the validation_set?
Exploring the effect of min_error
We will compare three models trained with different values of the stopping criterion. We intentionally picked models at the extreme ends (negative, just right, and too positive).
Train three models with these parameters:
1. model_4: min_error_reduction = -1 (ignoring this early stopping condition)
2. model_5: min_error_reduction = 0 (just right)
3. model_6: min_error_reduction = 5 (too positive)
For each of these three, we set max_depth = 6, and min_node_size = 0.
 Note: Each tree can take up to 30 seconds to train.
Can you write Python code for it?

model_4 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                                min_node_size = 0, min_error_reduction=-1)
model_5 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                                min_node_size = 0, min_error_reduction=0)
model_6 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                                min_node_size = 0, min_error_reduction=5)


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Run the following test code to check your implementation. Make sure you get 'Test passed' before proceeding.
Can you write Python code for it?

small_decision_tree = decision_tree_create(train_data, features, 'safe_loans', max_depth = 2,
                                        min_node_size = 10, min_error_reduction=0.0)
if count_nodes(small_decision_tree) == 7:
    print 'Test passed!'
else:
    print 'Test failed... try again!'
    print 'Number of nodes found                :', count_nodes(small_decision_tree)
    print 'Number of nodes that should be there : 5'


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Subsample dataset to make sure classes are balanced
Just as we did in the previous assignment, we will undersample the larger class (safe loans) in order to balance out our dataset. This means we are throwing away many data points. We used seed = 1 so everyone gets the same results.
Can you write Python code for it?

safe_loans_raw = loans[loans[target] == 1]
risky_loans_raw = loans[loans[target] == -1]

# Since there are less risky loans than safe loans, find the ratio of the sizes
# and use that percentage to undersample the safe loans.
percentage = len(risky_loans_raw)/float(len(safe_loans_raw))
safe_loans = safe_loans_raw.sample(percentage, seed = 1)
risky_loans = risky_loans_raw
loans_data = risky_loans.append(safe_loans)

print "Percentage of safe loans                 :", len(safe_loans) / float(len(loans_data))
print "Percentage of risky loans                :", len(risky_loans) / float(len(loans_data))
print "Total number of loans in our new dataset :", len(loans_data)


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Train-Validation split
We split the data into a train-validation split with 80% of the data in the training set and 20% of the data in the validation set. We use seed=1 so that everyone gets the same result.
Can you write Python code for it?

train_data, validation_set = loans_data.random_split(.8, seed=1)


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
Using the count_leaves function, compute the number of leaves in each of each models in (model_4, model_5, and model_6).
Can you write Python code for it?

print "Complexity (model 4):", count_leaves(model_4)
print "Complexity (model 5):", count_leaves(model_5)
print "Complexity (model 6):", count_leaves(model_6)


I am working on the file "Classification/Week 4/module-6-decision-tree-practical-assignment-blank.ipynb".
The first task is:
We will be using the same 4 categorical features as in the previous assignment:
1. grade of the loan
2. the length of the loan term
3. the home ownership status: own, mortgage, rent
4. number of years of employment.
In the dataset, each of these features is a categorical feature. Since we are building a binary decision tree, we will have to convert this to binary data in a subsequent section using 1-hot encoding.
Can you write Python code for it?

features = ['grade',              # grade of the loan
            'term',               # the term of the loan
            'home_ownership',     # home_ownership status: own, mortgage or rent
            'emp_length',         # number of years of employment
           ]
target = 'safe_loans'
loans = loans[features + [target]]


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Calculate the number of false positives made by the model.
Can you write Python code for it?

predictions = model_5.predict(validation_data)

fp = 0
for i in xrange(len(validation_data)):
    if validation_data["safe_loans"][i] != predictions[i]:
        if predictions[i] == 1 and validation_data["safe_loans"][i] == -1:
            fp = fp + 1

print fp


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Checkpoint: For each row, the probabilities should be a number in the range [0, 1]. We have provided a simple check here to make sure your answers are correct.
Can you write Python code for it?


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Checkpoint: You should expect to see 5 loans with the grade ['D', 'C', 'C', 'C', 'B'].
Effect of adding more trees
In this assignment, we will train 5 different ensemble classifiers in the form of gradient boosted trees. We will train models with 10, 50, 100, 200, and 500 trees.  We use the max_iterations parameter in the boosted tree module.
Let's get sarted with a model with max_iterations = 10:
Can you write Python code for it?

model_10 = graphlab.boosted_trees_classifier.create(train_data, validation_set=None,
        target = target, features = features, max_iterations = 10, verbose=False)


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Checkpoint: You should now see that the dataset is balanced (approximately 50-50 safe vs risky loans).
Note: There are many approaches for dealing with imbalanced data, including some where we modify the learning algorithm. These approaches are beyond the scope of this course, but some of them are reviewed in this paper. For this assignment, we use the simplest possible approach, where we subsample the overly represented class to get a more balanced dataset. In general, and especially when the data is highly imbalanced, we recommend using more advanced methods.
Split data into training and validation sets
We split the data into training data and validation data. We used seed=1 to make sure everyone gets the same results. We will use the validation data to help us select model parameters.
Can you write Python code for it?

train_data, validation_data = loans_data.random_split(.8, seed=1)


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Compare accuracy on entire validation set
Now we will compare the predicitve accuracy of our models on the validation set. Evaluate the accuracy of the 10, 50, 100, 200, and 500 tree models on the validation_data. Use the .evaluate method.
Can you write Python code for it?

print model_10.evaluate(validation_data)['accuracy']
print model_50.evaluate(validation_data)['accuracy']
print model_100.evaluate(validation_data)['accuracy']
print model_200.evaluate(validation_data)['accuracy']
print model_500.evaluate(validation_data)['accuracy']


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Gradient boosted tree classifier
Gradient boosted trees are a powerful variant of boosting methods; they have been used to win many Kaggle competitions, and have been widely used in industry.  We will explore the predictive power of multiple decision trees as opposed to a single decision tree.
Additional reading: If you are interested in gradient boosted trees, here is some additional reading material:
* GraphLab Create user guide
* Advanced material on boosted trees
We will now train models to predict safe_loans using the features above. In this section, we will experiment with training an ensemble of 5 trees. To cap the ensemble classifier at 5 trees, we call the function with max_iterations=5 (recall that each iterations corresponds to adding a tree). We set validation_set=None to make sure everyone gets the same results.
Can you write Python code for it?

model_5 = graphlab.boosted_trees_classifier.create(train_data, validation_set=None,
        target = target, features = features, max_iterations = 5)


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
In order to plot the classification errors (on the train_data and validation_data) versus the number of trees, we will need lists of these accuracies, which we get by applying the method .evaluate.
Steps to follow:

Step 1: Calculate the classification error for model on the training data (train_data).
Step 2: Store the training errors into a list (called training_errors) that looks like this:
[train_err_10, train_err_50, ..., train_err_500]
Step 3: Calculate the classification error of each model on the validation data (validation_data).
Step 4: Store the validation classification error into a list (called validation_errors) that looks like this:
[validation_err_10, validation_err_50, ..., validation_err_500]
Once that has been completed, the rest of the code should be able to evaluate correctly and generate the plot.

Let us start with Step 1. Write code to compute the classification error on the train_data for models model_10, model_50, model_100, model_200, and model_500.
Can you write Python code for it?

train_err_10 = 1 - model_10.evaluate(train_data)['accuracy']
train_err_50 = 1 - model_50.evaluate(train_data)['accuracy']
train_err_100 = 1 - model_100.evaluate(train_data)['accuracy']
train_err_200 = 1 - model_200.evaluate(train_data)['accuracy']
train_err_500 = 1 - model_500.evaluate(train_data)['accuracy']


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Let's quickly explore what the dataset looks like. First, let's print out the column names to see what features we have in this dataset. We have done this in previous assignments, so we won't belabor this here.
Can you write Python code for it?


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Load LendingClub dataset
We will be using the LendingClub data. As discussed earlier, the LendingClub is a peer-to-peer leading company that directly connects borrowers and potential lenders/investors.
Just like we did in previous assignments, we will build a classification model to predict whether or not a loan provided by lending club is likely to default.
Let us start by loading the data.
Can you write Python code for it?


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Making predictions
Just like we did in previous sections, let us consider a few positive and negative examples from the validation set. We will do the following:
* Predict whether or not a loan is likely to default.
* Predict the probability with which the loan is likely to default.
Can you write Python code for it?

# Select all positive and negative examples.
validation_safe_loans = validation_data[validation_data[target] == 1]
validation_risky_loans = validation_data[validation_data[target] == -1]

# Select 2 examples from the validation set for positive & negative loans
sample_validation_data_risky = validation_risky_loans[0:2]
sample_validation_data_safe = validation_safe_loans[0:2]

# Append the 4 examples into a single dataset
sample_validation_data = sample_validation_data_safe.append(sample_validation_data_risky)
sample_validation_data


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Modifying the target column
The target column (label column) of the dataset that we are interested in is called bad_loans. In this column 1 means a risky (bad) loan 0 means a safe  loan.
As in past assignments, in order to make this more intuitive and consistent with the lectures, we reassign the target to be:
* +1 as a safe  loan,
* -1 as a risky (bad) loan.
We put this in a new column called safe_loans.
Can you write Python code for it?

loans['safe_loans'] = loans['bad_loans'].apply(lambda x : +1 if x==0 else -1)
loans = loans.remove_column('bad_loans')


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Now, let us run Step 2. Save the training errors into a list called training_errors
Can you write Python code for it?

training_errors = [train_err_10, train_err_50, train_err_100,
                   train_err_200, train_err_500]


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Now, onto Step 3. Write code to compute the classification error on the validation_data for models model_10, model_50, model_100, model_200, and model_500.
Can you write Python code for it?

validation_err_10 = 1 - model_10.evaluate(validation_data)['accuracy']
validation_err_50 = 1 - model_50.evaluate(validation_data)['accuracy']
validation_err_100 = 1 - model_100.evaluate(validation_data)['accuracy']
validation_err_200 = 1 - model_200.evaluate(validation_data)['accuracy']
validation_err_500 = 1 - model_500.evaluate(validation_data)['accuracy']


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Now, train 4 models with max_iterations to be:
* max_iterations = 50,
* max_iterations = 100
* max_iterations = 200
* max_iterations = 500.
Let us call these models model_50, model_100, model_200, and model_500. You can pass in verbose=False in order to suppress the printed output.
Warning: This could take a couple of minutes to run.
Can you write Python code for it?

model_50 = graphlab.boosted_trees_classifier.create(train_data, validation_set=None,
        target = target, features = features, max_iterations = 50, verbose=False)
model_100 = graphlab.boosted_trees_classifier.create(train_data, validation_set=None,
        target = target, features = features, max_iterations = 100, verbose=False)
model_200 = graphlab.boosted_trees_classifier.create(train_data, validation_set=None,
        target = target, features = features, max_iterations = 200, verbose=False)
model_500 = graphlab.boosted_trees_classifier.create(train_data, validation_set=None,
        target = target, features = features, max_iterations = 500, verbose=False)


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Now, we are ready to go to Step 3. You can now use the prediction column to sort the loans in validation_data (in descending order) by prediction probability. Find the top 5 loans with the highest probability of being predicted as a safe loan.
Can you write Python code for it?


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Now, we will plot the training_errors and validation_errors versus the number of trees. We will compare the 10, 50, 100, 200, and 500 tree models. We provide some plotting code to visualize the plots within this notebook.
Run the following code to visualize the plots.
Can you write Python code for it?

plt.plot([10, 50, 100, 200, 500], training_errors, linewidth=4.0, label='Training error')
plt.plot([10, 50, 100, 200, 500], validation_errors, linewidth=4.0, label='Validation error')

make_figure(dim=(10,5), title='Error vs number of trees',
            xlabel='Number of trees',
            ylabel='Classification error',
            legend='best')


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Predicting on sample validation data
For each row in the sample_validation_data, write code to make model_5 predict whether or not the loan is classified as a safe loan.
Hint: Use the predict method in model_5 for this.
Can you write Python code for it?

predictions = model_5.predict(sample_validation_data)
#len(sample_validation_data[sample_validation_data["safe_loans"] != predictions])
count = 0
for i in xrange(len(sample_validation_data)):
    if sample_validation_data["safe_loans"][i] != predictions[i]:
        count = count + 1

print 1 - (count/float(len(predictions)))


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Quiz Question: According to model_5, which loan is the least likely to be a safe loan?
Checkpoint: Can you verify that for all the predictions with probability >= 0.5, the model predicted the label +1?
Evaluating the model on the validation data
Recall that the accuracy is defined as follows:
$$
\mbox{accuracy} = \frac{\mbox{# correctly classified examples}}{\mbox{# total examples}}
$$
Evaluate the accuracy of the model_5 on the validation_data.
Hint: Use the .evaluate() method in the model.
Can you write Python code for it?


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Quiz question: What grades are the top 5 loans?
Let us repeat this excercise to find the top 5 loans (in the validation_data) with the lowest probability of being predicted as a safe loan:
Can you write Python code for it?


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Quiz question: What is the number of false positives on the validation_data?
Calculate the number of false negatives made by the model.
Can you write Python code for it?

fn = 0
for i in xrange(len(validation_data)):
    if validation_data["safe_loans"][i] != predictions[i]:
        if predictions[i] == -1 and validation_data["safe_loans"][i] == 1:
            fn = fn + 1

print fn


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Quiz question: What percentage of the predictions on sample_validation_data did model_5 get correct?
Prediction probabilities
For each row in the sample_validation_data, what is the probability (according model_5) of a loan being classified as safe?
Hint: Set output_type='probability' to make probability predictions using model_5 on sample_validation_data:
Can you write Python code for it?


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Selecting features
In this assignment, we will be using a subset of features (categorical and numeric). The features we will be using are described in the code comments below. If you are a finance geek, the LendingClub website has a lot more details about these features.
The features we will be using are described in the code comments below:
Can you write Python code for it?

target = 'safe_loans'
features = ['grade',                     # grade of the loan (categorical)
            'sub_grade_num',             # sub-grade of the loan as a number from 0 to 1
            'short_emp',                 # one year or less of employment
            'emp_length_num',            # number of years of employment
            'home_ownership',            # home_ownership status: own, mortgage or rent
            'dti',                       # debt to income ratio
            'purpose',                   # the purpose of the loan
            'payment_inc_ratio',         # ratio of the monthly payment to income
            'delinq_2yrs',               # number of delinquincies
            'delinq_2yrs_zero',          # no delinquincies in last 2 years
            'inq_last_6mths',            # number of creditor inquiries in last 6 months
            'last_delinq_none',          # has borrower had a delinquincy
            'last_major_derog_none',     # has borrower had 90 day or worse rating
            'open_acc',                  # number of open credit accounts
            'pub_rec',                   # number of derogatory public records
            'pub_rec_zero',              # no derogatory public records
            'revol_util',                # percent of available credit being used
            'total_rec_late_fee',        # total late fees received to day
            'int_rate',                  # interest rate of the loan
            'total_rec_int',             # interest received to date
            'annual_inc',                # annual income of borrower
            'funded_amnt',               # amount committed to the loan
            'funded_amnt_inv',           # amount committed by investors for the loan
            'installment',               # monthly payment owed by the borrower
           ]


I am working on the file "Classification/Week 5/Assignment 1/module-8-boosting-assignment-1-blank.ipynb".
The first task is:
Skipping observations with missing values
Recall from the lectures that one common approach to coping with missing values is to skip observations that contain missing values.
We run the following code to do so:
Can you write Python code for it?

loans, loans_with_na = loans[[target] + features].dropna_split()

# Count the number of rows with missing data
num_rows_with_na = loans_with_na.num_rows()
num_rows = loans.num_rows()
print 'Dropping %s observations; keeping %s ' % (num_rows_with_na, num_rows)


I am working on the file "Classification/Week 5/Assignment 2/module-8-boosting-assignment-2-blank.ipynb".
The first task is:
Checking your Adaboost code
Train an ensemble of two tree stumps and see which features those stumps split on. We will run the algorithm with the following parameters:
* train_data
* features
* target
* num_tree_stumps = 2
Can you write Python code for it?

stump_weights, tree_stumps = adaboost_with_tree_stumps(train_data, features, target, num_tree_stumps=2)

def print_stump(tree):
    split_name = tree['splitting_feature'] # split_name is something like 'term. 36 months'
    if split_name is None:
        print "(leaf, label: %s)" % tree['prediction']
        return None
    split_feature, split_value = split_name.split('.')
    print '                       root'
    print '         |---------------|----------------|'
    print '         |                                |'
    print '         |                                |'
    print '         |                                |'
    print '  [{0} == 0]{1}[{0} == 1]    '.format(split_name, ' '*(27-len(split_name)))
    print '         |                                |'
    print '         |                                |'
    print '         |                                |'
    print '    (%s)                 (%s)' \
        % (('leaf, label: ' + str(tree['left']['prediction']) if tree['left']['is_leaf'] else 'subtree'),
           ('leaf, label: ' + str(tree['right']['prediction']) if tree['right']['is_leaf'] else 'subtree'))


I am working on the file "Classification/Week 5/Assignment 2/module-8-boosting-assignment-2-blank.ipynb".
The first task is:
Checkpoint: Now, we have another checkpoint to make sure you are on the right track.
Can you write Python code for it?

example_data_weights = graphlab.SArray(len(train_data)* [1.5])
if best_splitting_feature(train_data, features, target, example_data_weights) == 'term. 36 months':
    print 'Test passed!'
else:
    print 'Test failed... try again!'


I am working on the file "Classification/Week 5/Assignment 2/module-8-boosting-assignment-2-blank.ipynb".
The first task is:
Computing training error at the end of each iteration
Now, we will compute the classification error on the train_data and see how it is reduced as trees are added.
Can you write Python code for it?

error_all = []
for n in xrange(1, 31):
    predictions = predict_adaboost(stump_weights[:n], tree_stumps[:n], train_data)
    error = 1.0 - graphlab.evaluation.accuracy(train_data[target], predictions)
    error_all.append(error)
    print "Iteration %s, training error = %s" % (n, error_all[n-1])


I am working on the file "Classification/Week 5/Assignment 2/module-8-boosting-assignment-2-blank.ipynb".
The first task is:
Example: Training a weighted decision tree
To build intuition on how weighted data points affect the tree being built, consider the following:
Suppose we only care about making good predictions for the first 10 and last 10 items in train_data, we assign weights:
* 1 to the last 10 items
* 1 to the first 10 items
* and 0 to the rest.
Let us fit a weighted decision tree with max_depth = 2.
Can you write Python code for it?

# Assign weights
example_data_weights = graphlab.SArray([1.] * 10 + [0.]*(len(train_data) - 20) + [1.] * 10)

# Train a weighted decision tree model.
small_data_decision_tree_subset_20 = weighted_decision_tree_create(train_data, features, target,
                         example_data_weights, max_depth=2)


I am working on the file "Classification/Week 5/Assignment 2/module-8-boosting-assignment-2-blank.ipynb".
The first task is:
Extracting the target and the feature columns
We will now repeat some of the feature processing steps that we saw in the previous assignment:
First, we re-assign the target to have +1 as a safe (good) loan, and -1 as a risky (bad) loan.
Next, we select four categorical features:
1. grade of the loan
2. the length of the loan term
3. the home ownership status: own, mortgage, rent
4. number of years of employment.
Can you write Python code for it?

features = ['grade',              # grade of the loan
            'term',               # the term of the loan
            'home_ownership',     # home ownership status: own, mortgage or rent
            'emp_length',         # number of years of employment
           ]
loans['safe_loans'] = loans['bad_loans'].apply(lambda x : +1 if x==0 else -1)
loans.remove_column('bad_loans')
target = 'safe_loans'
loans = loans[features + [target]]


I am working on the file "Classification/Week 5/Assignment 2/module-8-boosting-assignment-2-blank.ipynb".
The first task is:
Getting the data ready
We will be using the same LendingClub dataset as in the previous assignment.
Can you write Python code for it?

loans = graphlab.SFrame('lending-club-data.gl/')

len(loans)


I am working on the file "Classification/Week 5/Assignment 2/module-8-boosting-assignment-2-blank.ipynb".
The first task is:
Here is what the first stump looks like:
Can you write Python code for it?


I am working on the file "Classification/Week 5/Assignment 2/module-8-boosting-assignment-2-blank.ipynb".
The first task is:
Here is what the next stump looks like:
Can you write Python code for it?

print_stump(tree_stumps[1])

print stump_weights


I am working on the file "Classification/Week 5/Assignment 2/module-8-boosting-assignment-2-blank.ipynb".
The first task is:
If your Adaboost is correctly implemented, the following things should be true:

tree_stumps[0] should split on term. 36 months with the prediction -1 on the left and +1 on the right.
tree_stumps[1] should split on grade.A with the prediction -1 on the left and +1 on the right.
Weights should be approximately [0.158, 0.177]

Reminders
- Stump weights ($\mathbf{\hat{w}}$) and data point weights ($\mathbf{\alpha}$) are two different concepts.
- Stump weights ($\mathbf{\hat{w}}$) tell you how important each stump is while making predictions with the entire boosted ensemble.
- Data point weights ($\mathbf{\alpha}$) tell you how important each data point is while training a decision stump.
Training a boosted ensemble of 10 stumps
Let us train an ensemble of 10 decision tree stumps with Adaboost. We run the adaboost_with_tree_stumps function with the following parameters:
* train_data
* features
* target
* num_tree_stumps = 10
Can you write Python code for it?

stump_weights, tree_stumps = adaboost_with_tree_stumps(train_data, features,
                                target, num_tree_stumps=10)


I am working on the file "Classification/Week 5/Assignment 2/module-8-boosting-assignment-2-blank.ipynb".
The first task is:
Let us take a quick look at what the trained tree is like. You should get something that looks like the following
{'is_leaf': False,
    'left': {'is_leaf': False,
        'left': {'is_leaf': True, 'prediction': -1, 'splitting_feature': None},
        'prediction': None,
        'right': {'is_leaf': True, 'prediction': 1, 'splitting_feature': None},
        'splitting_feature': 'grade.A'
     },
    'prediction': None,
    'right': {'is_leaf': False,
        'left': {'is_leaf': True, 'prediction': 1, 'splitting_feature': None},
        'prediction': None,
        'right': {'is_leaf': True, 'prediction': -1, 'splitting_feature': None},
        'splitting_feature': 'grade.D'
     },
     'splitting_feature': 'term. 36 months'
}
Can you write Python code for it?


I am working on the file "Classification/Week 5/Assignment 2/module-8-boosting-assignment-2-blank.ipynb".
The first task is:
Let's see what the feature columns look like now:
Can you write Python code for it?

features = loans_data.column_names()
features.remove('safe_loans')  # Remove the response variable
features


I am working on the file "Classification/Week 5/Assignment 2/module-8-boosting-assignment-2-blank.ipynb".
The first task is:
Making predictions
Recall from the lecture that in order to make predictions, we use the following formula:
$$
\hat{y} = sign\left(\sum_{t=1}^T \hat{w}_t f_t(x)\right)
$$
We need to do the following things:
- Compute the predictions $f_t(x)$ using the $t$-th decision tree
- Compute $\hat{w}_t f_t(x)$ by multiplying the stump_weights with the predictions $f_t(x)$ from the decision trees
- Sum the weighted predictions over each stump in the ensemble.
Complete the following skeleton for making predictions:
Can you write Python code for it?

def predict_adaboost(stump_weights, tree_stumps, data):
    scores = graphlab.SArray([0.]*len(data))

    for i, tree_stump in enumerate(tree_stumps):
        predictions = data.apply(lambda x: classify(tree_stump, x))

        # Accumulate predictions on scores array
        # YOUR CODE HERE
        scores = scores + (predictions * stump_weights[i])

    return scores.apply(lambda score : +1 if score > 0 else -1)

predictions = predict_adaboost(stump_weights, tree_stumps, test_data)
accuracy = graphlab.evaluation.accuracy(test_data[target], predictions)
print 'Accuracy of 10-component ensemble = %s' % accuracy


I am working on the file "Classification/Week 5/Assignment 2/module-8-boosting-assignment-2-blank.ipynb".
The first task is:
Making predictions with a weighted decision tree
We give you a function that classifies one data point. It can also return the probability if you want to play around with that as well.
Can you write Python code for it?

def classify(tree, x, annotate = False):
    # If the node is a leaf node.
    if tree['is_leaf']:
        if annotate:
            print "At leaf, predicting %s" % tree['prediction']
        return tree['prediction']
    else:
        # Split on feature.
        split_feature_value = x[tree['splitting_feature']]
        if annotate:
            print "Split on %s = %s" % (tree['splitting_feature'], split_feature_value)
        if split_feature_value == 0:
            return classify(tree['left'], x, annotate)
        else:
            return classify(tree['right'], x, annotate)