I am working on the file "Programming Assignment 3/module-4-linear-classifier-regularization-assignment-blank.ipynb".
The first task is:
Load and process review dataset
For this assignment, we will use the same subset of the Amazon product review dataset that we used in Module 3 assignment. The subset was chosen to contain similar numbers of positive and negative reviews, as the original dataset consisted of mostly positive reviews.
Can you write Python code for it?


I am working on the file "Programming Assignment 3/module-4-linear-classifier-regularization-assignment-blank.ipynb".
The first task is:
Now, let us take a look at what the dataset looks like (Note: This may take a few minutes).
Can you write Python code for it?


I am working on the file "Programming Assignment 3/module-4-linear-classifier-regularization-assignment-blank.ipynb".
The first task is:
Now, let's run the function add_coefficients_to_table for each of the L2 penalty strengths.
Can you write Python code for it?

add_coefficients_to_table(coefficients_0_penalty, 'coefficients [L2=0]')
add_coefficients_to_table(coefficients_4_penalty, 'coefficients [L2=4]')
add_coefficients_to_table(coefficients_10_penalty, 'coefficients [L2=10]')
add_coefficients_to_table(coefficients_1e2_penalty, 'coefficients [L2=1e2]')
add_coefficients_to_table(coefficients_1e3_penalty, 'coefficients [L2=1e3]')
add_coefficients_to_table(coefficients_1e5_penalty, 'coefficients [L2=1e5]')


I am working on the file "Programming Assignment 3/module-4-linear-classifier-regularization-assignment-blank.ipynb".
The first task is:
Quiz Question: Does the term with L2 regularization increase or decrease $\ell\ell(\mathbf{w})$?
The logistic regression function looks almost like the one in the last assignment, with a minor modification to account for the L2 penalty.  Fill in the code below to complete this modification.
Can you write Python code for it?

def logistic_regression_with_L2(feature_matrix, sentiment, initial_coefficients, step_size, l2_penalty, max_iter):
    coefficients = np.array(initial_coefficients) # make sure it's a numpy array
    for itr in xrange(max_iter):
        # Predict P(y_i = +1|x_i,w) using your predict_probability() function
        ## YOUR CODE HERE
        predictions = ...

        # Compute indicator value for (y_i = +1)
        indicator = (sentiment==+1)

        # Compute the errors as indicator - predictions
        errors = indicator - predictions
        for j in xrange(len(coefficients)): # loop over each coefficient
            is_intercept = (j == 0)
            # Recall that feature_matrix[:,j] is the feature column associated with coefficients[j].
            # Compute the derivative for coefficients[j]. Save it in a variable called derivative
            ## YOUR CODE HERE
            derivative = ...

            # add the step size times the derivative to the current coefficient
            ## YOUR CODE HERE
            ...

        # Checking whether log likelihood is increasing
        if itr <= 15 or (itr <= 100 and itr % 10 == 0) or (itr <= 1000 and itr % 100 == 0) \
        or (itr <= 10000 and itr % 1000 == 0) or itr % 10000 == 0:
            lp = compute_log_likelihood_with_L2(feature_matrix, sentiment, coefficients, l2_penalty)
            print 'iteration %*d: log likelihood of observed labels = %.8f' % \
                (int(np.ceil(np.log10(max_iter))), itr, lp)
    return coefficients


I am working on the file "Programming Assignment 3/module-4-linear-classifier-regularization-assignment-blank.ipynb".
The first task is:
Quiz Question: In the code above, was the intercept term regularized?
To verify the correctness of the gradient ascent algorithm, we provide a function for computing log likelihood (which we recall from the last assignment was a topic detailed in an advanced optional video, and used here for its numerical stability).
$$\ell\ell(\mathbf{w}) = \sum_{i=1}^N \Big( (\mathbf{1}[y_i = +1] - 1)\mathbf{w}^T h(\mathbf{x}_i) - \ln\left(1 + \exp(-\mathbf{w}^T h(\mathbf{x}_i))\right) \Big) \color{red}{-\lambda\|\mathbf{w}\|_2^2} $$
Can you write Python code for it?

def compute_log_likelihood_with_L2(feature_matrix, sentiment, coefficients, l2_penalty):
    indicator = (sentiment==+1)
    scores = np.dot(feature_matrix, coefficients)

    lp = np.sum((indicator-1)*scores - np.log(1. + np.exp(-scores))) - l2_penalty*np.sum(coefficients[1:]**2)

    return lp


I am working on the file "Programming Assignment 3/module-4-linear-classifier-regularization-assignment-blank.ipynb".
The first task is:
Train-Validation split
We split the data into a train-validation split with 80% of the data in the training set and 20% of the data in the validation set. We use seed=2 so that everyone gets the same result.
Note: In previous assignments, we have called this a train-test split. However, the portion of data that we don't train on will be used to help select model parameters. Thus, this portion of data should be called a validation set. Recall that examining performance of various potential models (i.e. models with different parameters) should be on a validation set, while evaluation of selected model should always be on a test set.
Can you write Python code for it?

train_data, validation_data = products.random_split(.8, seed=2)

print 'Training set   : %d data points' % len(train_data)
print 'Validation set : %d data points' % len(validation_data)


I am working on the file "Programming Assignment 3/module-4-linear-classifier-regularization-assignment-blank.ipynb".
The first task is:
Using the coefficients trained with L2 penalty 0, find the 5 most positive words (with largest positive coefficients). Save them to positive_words. Similarly, find the 5 most negative words (with largest negative coefficients) and save them to negative_words.
Quiz Question. Which of the following is not listed in either positive_words or negative_words?
Let us observe the effect of increasing L2 penalty on the 10 words just selected. We provide you with a utility function to  plot the coefficient path.
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = 10, 6

def make_coefficient_plot(table, positive_words, negative_words, l2_penalty_list):
    cmap_positive = plt.get_cmap('Reds')
    cmap_negative = plt.get_cmap('Blues')

    xx = l2_penalty_list
    plt.plot(xx, [0.]*len(xx), '--', lw=1, color='k')

    table_positive_words = table.filter_by(column_name='word', values=positive_words)
    table_negative_words = table.filter_by(column_name='word', values=negative_words)
    del table_positive_words['word']
    del table_negative_words['word']

    for i in xrange(len(positive_words)):
        color = cmap_positive(0.8*((i+1)/(len(positive_words)*1.2)+0.15))
        plt.plot(xx, table_positive_words[i:i+1].to_numpy().flatten(),
                 '-', label=positive_words[i], linewidth=4.0, color=color)

    for i in xrange(len(negative_words)):
        color = cmap_negative(0.8*((i+1)/(len(negative_words)*1.2)+0.15))
        plt.plot(xx, table_negative_words[i:i+1].to_numpy().flatten(),
                 '-', label=negative_words[i], linewidth=4.0, color=color)

    plt.legend(loc='best', ncol=3, prop={'size':16}, columnspacing=0.5)
    plt.axis([1, 1e5, -1, 2])
    plt.title('Coefficient path')
    plt.xlabel('L2 penalty ($\lambda$)')
    plt.ylabel('Coefficient value')
    plt.xscale('log')
    plt.rcParams.update({'font.size': 18})
    plt.tight_layout()


I am working on the file "Programming Assignment 3/module-4-linear-classifier-regularization-assignment-blank.ipynb".
The first task is:
We convert both the training and validation sets into NumPy arrays.
Warning: This may take a few minutes.
Can you write Python code for it?

feature_matrix_train, sentiment_train = get_numpy_data(train_data, important_words, 'sentiment')
feature_matrix_valid, sentiment_valid = get_numpy_data(validation_data, important_words, 'sentiment')


I am working on the file "Programming Assignment 4/Identifying safe loans with decision trees.ipynb".
The first task is:
12. Now, we will use our model to predict whether or not a loan is likely to default.
Can you write Python code for it?

sample_validation_data_Y = sample_validation_data['safe_loans'].as_matrix()
sample_validation_data_X = sample_validation_data.drop('safe_loans', axis=1).as_matrix()
print sample_validation_data_Y.shape
print sample_validation_data_X.shape

decision_tree_model.predict(sample_validation_data_X)


I am working on the file "Programming Assignment 4/Identifying safe loans with decision trees.ipynb".
The first task is:
17. Now, evaluate the accuracy of the small_model and decision_tree_model on the entire validation_data, not just the subsample considered above.
Can you write Python code for it?

validation_Y = validation_data['safe_loans'].as_matrix()
validation_X = validation_data.drop('safe_loans', axis=1).as_matrix()
print validation_Y.shape
print validation_X.shape

small_model.score(validation_X, validation_Y)

decision_tree_model.score(validation_X, validation_Y)


I am working on the file "Programming Assignment 4/Identifying safe loans with decision trees.ipynb".
The first task is:
4. Now, let us explore the distribution of the column safe_loans.
Can you write Python code for it?

loans['safe_loans'].value_counts()

print len(loans[loans['safe_loans']==+1]) *1.0/ len(loans['safe_loans'])
print len(loans[loans['safe_loans']==-1]) *1.0/ len(loans['safe_loans'])


I am working on the file "Programming Assignment 4/Identifying safe loans with decision trees.ipynb".
The first task is:
Build a decision tree classifier
Can you write Python code for it?

train_Y = train_data['safe_loans'].as_matrix()
train_X = train_data.drop('safe_loans', axis=1).as_matrix()
print train_Y.shape
print train_X.shape

from sklearn.tree import DecisionTreeClassifier
decision_tree_model = DecisionTreeClassifier(max_depth=6)
decision_tree_model = decision_tree_model.fit(train_X, train_Y)

small_model = DecisionTreeClassifier(max_depth=2)
small_model = small_model.fit(train_X, train_Y)

import sklearn
print('The scikit-learn version is {}.'.format(sklearn.__version__))


I am working on the file "Programming Assignment 4/Identifying safe loans with decision trees.ipynb".
The first task is:
Exploring some features
2. print out the column names
Can you write Python code for it?


I am working on the file "Programming Assignment 4/Identifying safe loans with decision trees.ipynb".
The first task is:
Features for the classification algorithm
5. using a subset of features (categorical and numeric)
Can you write Python code for it?

features = ['grade',                     # grade of the loan
            'sub_grade',                 # sub-grade of the loan
            'short_emp',                 # one year or less of employment
            'emp_length_num',            # number of years of employment
            'home_ownership',            # home_ownership status: own, mortgage or rent
            'dti',                       # debt to income ratio
            'purpose',                   # the purpose of the loan
            'term',                      # the term of the loan
            'last_delinq_none',          # has borrower had a delinquincy
            'last_major_derog_none',     # has borrower had 90 day or worse rating
            'revol_util',                # percent of available credit being used
            'total_rec_late_fee',        # total late fees received to day
           ]

target = 'safe_loans'                    # prediction target (y) (+1 means safe, -1 is risky)

# Extract the feature columns and target column
loans = loans[features + [target]]


I am working on the file "Programming Assignment 4/Identifying safe loans with decision trees.ipynb".
The first task is:
Load the Lending Club dataset
1. Load the dataset into a data frame named loans
Can you write Python code for it?

loans = pd.read_csv('lending-club-data.csv')
loans.head(2)


I am working on the file "Programming Assignment 4/Identifying safe loans with decision trees.ipynb".
The first task is:
One-hot encoding
Can you write Python code for it?

categorical_variables = []
for feat_name, feat_type in zip(loans.columns, loans.dtypes):
    if feat_type == object:
        categorical_variables.append(feat_name)

for feature in categorical_variables:

    loans_one_hot_encoded = pd.get_dummies(loans[feature],prefix=feature)
    #print loans_one_hot_encoded

    loans = loans.drop(feature, axis=1)
    for col in loans_one_hot_encoded.columns:
        loans[col] = loans_one_hot_encoded[col]

print loans.head(2)
print loans.columns
#pd.get_dummies(loans, prefix=['col1', 'col2'])

with open('module-5-assignment-1-train-idx.json') as train_data_file:
    train_idx  = json.load(train_data_file)
with open('module-5-assignment-1-validation-idx.json') as validation_data_file:
    validation_idx = json.load(validation_data_file)

print train_idx[:3]
print validation_idx[:3]

print len(train_idx)
print len(validation_idx)

train_data = loans.iloc[train_idx]
validation_data = loans.iloc[validation_idx]

print len(loans.dtypes )


I am working on the file "Programming Assignment 4/Identifying safe loans with decision trees.ipynb".
The first task is:
Quiz Question 3:
Notice that the probability preditions are the exact same for the 2nd and 3rd loans. Why would this happen?
Quiz Question 4:
Based on the visualized tree, what prediction would you make for this data point (according to small_model)? (If you don't have Graphviz, you can answer this quiz question by executing the next part.)
15. Now, verify your prediction by examining the prediction made using small_model.
Evaluating accuracy of the decision tree model
16. Evaluate the accuracy of small_model and decision_tree_model on the training data. (Hint: if you are using scikit-learn, you can use the .score() method)
Can you write Python code for it?

small_model.score(train_X, train_Y)

decision_tree_model.score(train_X, train_Y)


I am working on the file "Programming Assignment 4/Identifying safe loans with decision trees.ipynb".
The first task is:
Quiz Question 5:
What is the accuracy of decision_tree_model on the validation set, rounded to the nearest .01?
Answer:
0.64
Evaluating accuracy of a complex decision tree model
18. Using sklearn.tree.DecisionTreeClassifier, train a decision tree with maximum depth = 10. Call this model big_model.
Can you write Python code for it?

big_model = DecisionTreeClassifier(max_depth=10)
big_model = big_model.fit(train_X, train_Y)


I am working on the file "Programming Assignment 4/Identifying safe loans with decision trees.ipynb".
The first task is:
Visualizing a learned model (Optional)
Can you write Python code for it?

from sklearn import tree
import graphviz
from os import system

dot_data = tree.export_graphviz(small_model, out_file='simple_tree.dot',
                               feature_names=train_data.columns,
                         class_names=['+1','-1'],
                         filled=True, rounded=True,
                         special_characters=True)
system("dot -Tpng simple_tree.dot -o simple_tree.png")

from IPython.display import Image
Image(filename='simple_tree.png')


I am working on the file "Programming Assignment 5/Implementing binary decision trees.ipynb".
The first task is:
1. Load the dataset into a data frame named loans
Can you write Python code for it?

loans = pd.read_csv('lending-club-data.csv')
loans.head(2)

# safe_loans =  1 => safe
# safe_loans = -1 => risky
loans['safe_loans'] = loans['bad_loans'].apply(lambda x : +1 if x==0 else -1)

#loans = loans.remove_column('bad_loans')
loans = loans.drop('bad_loans', axis=1)

features = ['grade',              # grade of the loan
            'term',               # the term of the loan
            'home_ownership',     # home_ownership status: own, mortgage or rent
            'emp_length',         # number of years of employment
           ]
target = 'safe_loans'

loans = loans[features + [target]]

loans.iloc[122602]


I am working on the file "Programming Assignment 5/Implementing binary decision trees.ipynb".
The first task is:
11. Now, we will provide a Python skeleton of the learning algorithm. Note that this code is not complete; it needs to be completed by you if you are using Python. Otherwise, your code should be analogous to

Stopping condition 1: All data points in a node are from the same class.
Stopping condition 2: No more features to split on.
Additional stopping condition: In addition to the above two stopping conditions covered in lecture, in this assignment we will also consider a stopping condition based on the max_depth of the tree. By not letting the tree grow too deep, we will save computational effort in the learning process.
Can you write Python code for it?

def decision_tree_create(data, features, target, current_depth = 0, max_depth = 10):
    remaining_features = features[:] # Make a copy of the features.

    target_values = data[target]
    print "--------------------------------------------------------------------"
    print "Subtree, depth = %s (%s data points)." % (current_depth, len(target_values))

    # Stopping condition 1
    # (Check if there are mistakes at current node.
    # Recall you wrote a function intermediate_node_num_mistakes to compute this.)
    if intermediate_node_num_mistakes(target_values) == 0:  ## YOUR CODE HERE
        print "Stopping condition 1 reached."
        # If not mistakes at current node, make current node a leaf node
        return create_leaf(target_values)

    # Stopping condition 2 (check if there are remaining features to consider splitting on)
    if remaining_features == []:   ## YOUR CODE HERE
        print "Stopping condition 2 reached."
        # If there are no remaining features to consider, make current node a leaf node
        return create_leaf(target_values)

    # Additional stopping condition (limit tree depth)
    if current_depth >= max_depth:  ## YOUR CODE HERE
        print "Reached maximum depth. Stopping for now."
        # If the max tree depth has been reached, make current node a leaf node
        return create_leaf(target_values)

    # Find the best splitting feature (recall the function best_splitting_feature implemented above)
    ## YOUR CODE HERE
    splitting_feature = best_splitting_feature(data, remaining_features, target)

    # Split on the best feature that we found.
    left_split = data[data[splitting_feature] == 0]
    right_split = data[data[splitting_feature] == 1]      ## YOUR CODE HERE
    remaining_features.remove(splitting_feature)
    print "Split on feature %s. (%s, %s)" % (\
                      splitting_feature, len(left_split), len(right_split))

    # Create a leaf node if the split is "perfect"
    if len(left_split) ==...


I am working on the file "Programming Assignment 5/Implementing binary decision trees.ipynb".
The first task is:
19. Using this function, we can print out the root of our decision tree:
Can you write Python code for it?


I am working on the file "Programming Assignment 5/Implementing binary decision trees.ipynb".
The first task is:
Building the tree
With the above functions implemented correctly, we are now ready to build our decision tree. Each node in the decision tree is represented as a dictionary which contains the following keys and possible values:
10. First, we will write a function that creates a leaf node given a set of target values.
Your code should be analogous to
Can you write Python code for it?

def create_leaf(target_values):
    # Create a leaf node
    leaf = {'splitting_feature' : None,
            'left' : None,
            'right' : None,
            'is_leaf': True }   ## YOUR CODE HERE

    # Count the number of data points that are +1 and -1 in this node.
    num_ones = len(target_values[target_values == +1])
    num_minus_ones = len(target_values[target_values == -1])

    # For the leaf node, set the prediction to be the majority class.
    # Store the predicted class (1 or -1) in leaf['prediction']
    if num_ones > num_minus_ones:
        leaf['prediction'] = 1         ## YOUR CODE HERE
    else:
        leaf['prediction'] = -1         ## YOUR CODE HERE

    # Return the leaf node
    return leaf


I am working on the file "Programming Assignment 5/Implementing binary decision trees.ipynb".
The first task is:
Decision tree implementation
Function to count number of mistakes while predicting majority class
Recall from the lecture that prediction at an intermediate node works by predicting the majority class for all data points that belong to this node. Now, we will write a function that calculates the number of misclassified examples when predicting the majority class. This will be used to help determine which feature is the best to split on at a given node of the tree.
Note: Keep in mind that in order to compute the number of mistakes for a majority classifier, we only need the label (y values) of the data points in the node.
Steps to follow:

Step 1: Calculate the number of safe loans and risky loans.
Step 2: Since we are assuming majority class prediction, all the data points that are not in the majority class are considered mistakes.
Step 3: Return the number of mistakes.

7. Now, let us write the function intermediate_node_num_mistakes which computes the number of misclassified examples of an intermediate node given the set of labels (y values) of the data points contained in the node. Your code should be analogous to
Can you write Python code for it?

def intermediate_node_num_mistakes(labels_in_node):
    # Corner case: If labels_in_node is empty, return 0
    if len(labels_in_node) == 0:
        return 0
    # Count the number of 1's (safe loans)
    ## YOUR CODE HERE
    safe_loan = (labels_in_node==1).sum()
    # Count the number of -1's (risky loans)
    ## YOUR CODE HERE
    risky_loan = (labels_in_node==-1).sum()
    # Return the number of mistakes that the majority classifier makes.
    ## YOUR CODE HERE
    return min(safe_loan, risky_loan)


I am working on the file "Programming Assignment 5/Implementing binary decision trees.ipynb".
The first task is:
Function to pick best feature to split on
The function best_splitting_feature takes 3 arguments:

The data
The features to consider for splits (a list of strings of column names to consider for splits)
The name of the target/label column (string)

The function will loop through the list of possible features, and consider splitting on each of them. It will calculate the classification error of each split and return the feature that had the smallest classification error when split on.
Recall that the classification error is defined as follows:
9. Follow these steps to implement best_splitting_feature:

Step 1: Loop over each feature in the feature list
Step 2: Within the loop, split the data into two groups: one group where all of the data has feature value 0 or False (we will call this the left split), and one group where all of the data has feature value 1 or True (we will call this the right split). Make sure the left split corresponds with 0 and the right split corresponds with 1 to ensure your implementation fits with our implementation of the tree building process.
Step 3: Calculate the number of misclassified examples in both groups of data and use the above formula to compute theclassification error.
Step 4: If the computed error is smaller than the best error found so far, store this feature and its error.

Note: Remember that since we are only dealing with binary features, we do not have to consider thresholds for real-valued features. This makes the implementation of this function much easier.
Your code should be analogous to
Can you write Python code for it?

def best_splitting_feature(data, features, target):

    target_values = data[target]
    best_feature = None # Keep track of the best feature
    best_error = 10     # Keep track of the best error so far
    # Note: Since error is always <= 1, we should intialize it with something larger than 1.

    # Convert to float to make sure error gets computed correctly.
    num_data_points = float(len(data))

    # Loop through each feature to consider splitting on that feature
    for feature in features:

        # The left split will have all data points where the feature value is 0
        left_split = data[data[feature] == 0]

        # The right split will have all data points where the feature value is 1
        ## YOUR CODE HERE
        right_split = data[data[feature] == 1]

        # Calculate the number of misclassified examples in the left split.
        # Remember that we implemented a function for this! (It was called intermediate_node_num_mistakes)
        # YOUR CODE HERE
        left_mistakes = intermediate_node_num_mistakes(left_split[target])

        # Calculate the number of misclassified examples in the right split.
        ## YOUR CODE HERE
        right_mistakes = intermediate_node_num_mistakes(right_split[target])

        # Compute the classification error of this split.
        # Error = (# of mistakes (left) + # of mistakes (right)) / (# of data points)
        ## YOUR CODE HERE
        error = (left_mistakes + right_mistakes) / num_data_points

        # If this is the best error we have found so far, store the feature as best_feature and the error as best_error
        ## YOUR CODE HERE
        if error < best_error:
            best_feature = feature
            best_error = error

    return best_feature # Return the best feature we found


I am working on the file "Programming Assignment 5/Implementing binary decision trees.ipynb".
The first task is:
Quiz Question:
What is the feature that is used for the split at the root node?
Answer:
term_ 36 months
Exploring the intermediate left subtree
The tree is a recursive dictionary, so we do have access to all the nodes! We can use

my_decision_tree['left'] to go left
my_decision_tree['right'] to go right

20. We can print out the left subtree by running the code
Can you write Python code for it?

print_stump(my_decision_tree['left'], my_decision_tree['splitting_feature'])

print_stump(my_decision_tree['left']['left'], my_decision_tree['left']['splitting_feature'])

print_stump(my_decision_tree['right'], my_decision_tree['splitting_feature'])

print_stump(my_decision_tree['right']['right'], my_decision_tree['right']['splitting_feature'])


I am working on the file "Programming Assignment 5/Implementing binary decision trees.ipynb".
The first task is:
Quiz question:
What was the feature that my_decision_tree first split on while making the prediction for test_data[0]?
Quiz question:
What was the first feature that lead to a right split of test_data[0]?
Quiz question:
What was the last feature split on before reaching a leaf node for test_data[0]?
Answer:
term_36 months
Answer:
grade_D
Answer:
grade_D
Evaluating your decision tree
16. Now, we will write a function to evaluate a decision tree by computing the classification error of the tree on the given dataset. Write a function called evaluate_classification_error that takes in as input:

tree (as described above)
data (a data frame of data points)

This function should return a prediction (class label) for each row in data using the decision tree. Your code should be analogous to
Can you write Python code for it?

def evaluate_classification_error(tree, data):
    # Apply the classify(tree, x) to each row in your data
    prediction = data.apply(lambda x: classify(tree, x), axis=1)

    # Once you've made the predictions, calculate the classification error and return it
    ## YOUR CODE HERE

    return (data['safe_loans'] != np.array(prediction)).values.sum() *1. / len(data)


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
As before, we reassign the labels to have +1 for a safe loan, and -1 for a risky (bad) loan.
Can you write Python code for it?

loans['safe_loans'] = loans['bad_loans'].apply(lambda x : +1 if x==0 else -1)
loans = loans.drop('bad_loans', axis=1)


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Build a tree!
Now that your code is working, we will train a tree model on the train_data with
* max_depth = 6
* min_node_size = 100,
* min_error_reduction = 0.0
Warning: This code block may take a minute to learn.
Can you write Python code for it?

my_decision_tree_new = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                                min_node_size = 100, min_error_reduction=0.0)


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Calculate the accuracy of each model (model_4, model_5, or model_6) on the validation set.
Can you write Python code for it?

print "Validation data, classification error (model 4):", evaluate_classification_error(model_4, validation_data, target)
print "Validation data, classification error (model 5):", evaluate_classification_error(model_5, validation_data, target)
print "Validation data, classification error (model 6):", evaluate_classification_error(model_6, validation_data, target)


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Compute the number of nodes in model_1, model_2, and model_3.
Can you write Python code for it?

print "number of leaves in model_1 is : {}".format(count_leaves(model_1))
print "number of leaves in model_2 is : {}".format(count_leaves(model_2))
print "number of leaves in model_3 is : {}".format(count_leaves(model_3))


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Evaluating the models
Let us evaluate the models on the train and validation data. Let us start by evaluating the classification error on the training data:
Can you write Python code for it?

print "Training data, classification error (model 1):", evaluate_classification_error(model_1, train_data, target)
print "Training data, classification error (model 2):", evaluate_classification_error(model_2, train_data, target)
print "Training data, classification error (model 3):", evaluate_classification_error(model_3, train_data, target)


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Incorporating new early stopping conditions in binary decision tree implementation
Now, you will implement a function that builds a decision tree handling the three early stopping conditions described in this assignment.  In particular, you will write code to detect early stopping conditions 2 and 3.  You implemented above the functions needed to detect these conditions.  The 1st early stopping condition, max_depth, was implemented in the previous assigment and you will not need to reimplement this.  In addition to these early stopping conditions, the typical stopping conditions of having no mistakes or no more features to split on (which we denote by "stopping conditions" 1 and 2) are also included as in the previous assignment.
Implementing early stopping condition 2: minimum node size:

Step 1: Use the function reached_minimum_node_size that you implemented earlier to write an if condition to detect whether we have hit the base case, i.e., the node does not have enough data points and should be turned into a leaf. Don't forget to use the min_node_size argument.
Step 2: Return a leaf. This line of code should be the same as the other (pre-implemented) stopping conditions.

Implementing early stopping condition 3: minimum error reduction:
Note: This has to come after finding the best splitting feature so we can calculate the error after splitting in order to calculate the error reduction.

Step 1: Calculate the classification error before splitting.  Recall that classification error is defined as:

$$
\text{classification error} = \frac{\text{# mistakes}}{\text{# total examples}}
$$
* Step 2: Calculate the classification error after splitting. This requires calculating the number of mistakes in the left and right splits, and then dividing by the total number of examples.
* Step 3: Use the function error_reduction to that you implemented earlier to write an if condition to detect whether  the reduction in error is less than the constant provided (min_error_reduction). Don't forget to use that argument.
* Step...
Can you write Python code for it?

def decision_tree_create(data, features, target, current_depth = 0,
                         max_depth = 10, min_node_size=1,
                         min_error_reduction=0.0):

    remaining_features = features[:] # Make a copy of the features.

    target_values = data[target]
    print "--------------------------------------------------------------------"
    print "Subtree, depth = %s (%s data points)." % (current_depth, len(target_values))

    # Stopping condition 1: All nodes are of the same type.
    if intermediate_node_num_mistakes(target_values) == 0:
        print "Stopping condition 1 reached. All data points have the same target value."
        return create_leaf(target_values)

    # Stopping condition 2: No more features to split on.
    if remaining_features == []:
        print "Stopping condition 2 reached. No remaining features."
        return create_leaf(target_values)

    # Early stopping condition 1: Reached max depth limit.
    if current_depth >= max_depth:
        print "Early stopping condition 1 reached. Reached maximum depth."
        return create_leaf(target_values)

    # Early stopping condition 2: Reached the minimum node size.
    # If the number of data points is less than or equal to the minimum size, return a leaf.
    if  reached_minimum_node_size(data, min_node_size):          ## YOUR CODE HERE
        print "Early stopping condition 2 reached. Reached minimum node size."
        return  create_leaf(target_values) ## YOUR CODE HERE

    # Find the best splitting feature
    splitting_feature = best_splitting_feature(data, features, target)

    # Split on the best feature that we found.
    left_split = data[data[splitting_feature] == 0]
    right_split = data[data[splitting_feature] == 1]

    # Early stopping condition 3: Minimum error reduction
    # Calculate the error before splitting (number of misclassified examples
    # divided by the total number of examples)
   ...


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Let's add some annotations to our prediction to see what the prediction path was that lead to this predicted class:
Can you write Python code for it?

classify(my_decision_tree_new, validation_data.iloc[0], annotate = True)


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Let's now recall the prediction path for the decision tree learned in the previous assignment, which we recreated here as my_decision_tree_old.
Can you write Python code for it?

classify(my_decision_tree_old, validation_data.iloc[0], annotate = True)


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Let's now train a tree model ignoring early stopping conditions 2 and 3 so that we get the same tree as in the previous assignment.  To ignore these conditions, we set min_node_size=0 and min_error_reduction=-1 (a negative value).
Can you write Python code for it?

my_decision_tree_old = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                                min_node_size = 0, min_error_reduction=-1)


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Load LendingClub Dataset
This assignment will use the LendingClub dataset used in the previous two assignments.
Can you write Python code for it?

loans = pd.read_csv('lending-club-data.csv')
loans.head(2)


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Now evaluate the classification error on the validation data.
Can you write Python code for it?

print "Validation data, classification error (model 1):", evaluate_classification_error(model_1, validation_data, target)
print "Validation data, classification error (model 2):", evaluate_classification_error(model_2, validation_data, target)
print "Validation data, classification error (model 3):", evaluate_classification_error(model_3, validation_data, target)


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Now, let us evaluate the models (model_7, model_8, or model_9) on the validation_set.
Can you write Python code for it?

print "Validation data, classification error (model 7):", evaluate_classification_error(model_7, validation_data, target)
print "Validation data, classification error (model 8):", evaluate_classification_error(model_8, validation_data, target)
print "Validation data, classification error (model 9):", evaluate_classification_error(model_9, validation_data, target)


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Quiz Question: Assume an intermediate node has 6 safe loans and 3 risky loans.  For each of 4 possible features to split on, the error reduction is 0.0, 0.05, 0.1, and 0.14, respectively. If the minimum gain in error reduction parameter is set to 0.2, what should the tree learning algorithm do next?
STOP
Grabbing binary decision tree helper functions from past assignment
Recall from the previous assignment that we wrote a function intermediate_node_num_mistakes that calculates the number of misclassified examples when predicting the majority class. This is used to help determine which feature is best to split on at a given node of the tree.
Please copy and paste your code for intermediate_node_num_mistakes here.
Can you write Python code for it?


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Quiz Question: For my_decision_tree_new trained with max_depth = 6, min_node_size = 100, min_error_reduction=0.0, is the prediction path for validation_set[0] shorter, longer, or the same as for my_decision_tree_old that ignored the early stopping conditions 2 and 3?
shorter
Quiz Question: For my_decision_tree_new trained with max_depth = 6, min_node_size = 100, min_error_reduction=0.0, is the prediction path for any point always shorter, always longer, always the same, shorter or the same, or longer or the same as for my_decision_tree_old that ignored the early stopping conditions 2 and 3?
shorter or the same
 Quiz Question: For a tree trained on any dataset using max_depth = 6, min_node_size = 100, min_error_reduction=0.0, what is the maximum number of splits encountered while making a single prediction?
6
Evaluating the model
Now let us evaluate the model that we have trained. You implemented this evaluation in the function evaluate_classification_error from the previous assignment.
Please copy and paste your evaluate_classification_error code here.
Can you write Python code for it?

def evaluate_classification_error(tree, data, target):
    # Apply the classify(tree, x) to each row in your data
    prediction = data.apply(lambda x: classify(tree, x), axis=1)

    # Once you've made the predictions, calculate the classification error and return it
    ## YOUR CODE HERE
    return (data[target] != np.array(prediction)).values.sum() / float(len(data))


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Quiz Question: Given an intermediate node with 6 safe loans and 3 risky loans, if the min_node_size parameter is 10, what should the tree learning algorithm do next?
STOP
Early stopping condition 3: Minimum gain in error reduction
The function error_reduction takes 2 arguments:

The error before a split, error_before_split.
The error after a split, error_after_split.

This function computes the gain in error reduction, i.e., the difference between the error before the split and that after the split. This function will be used to detect this early stopping condition in the decision_tree_create function.
Fill in the parts of the function below where you find ## YOUR CODE HERE.  There is one instance in the function below.
Can you write Python code for it?

def error_reduction(error_before_split, error_after_split):
    # Return the error before the split minus the error after the split.
    ## YOUR CODE HERE
    return error_before_split - error_after_split


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Quiz Question: Is the validation error of the new decision tree (using early stopping conditions 2 and 3) lower than, higher than, or the same as that of the old decision tree from the previous assignment?
lower
Exploring the effect of max_depth
We will compare three models trained with different values of the stopping criterion. We intentionally picked models at the extreme ends (too small, just right, and too large).
Train three models with these parameters:

model_1: max_depth = 2 (too small)
model_2: max_depth = 6 (just right)
model_3: max_depth = 14 (may be too large)

For each of these three, we set min_node_size = 0 and min_error_reduction = -1.
 Note: Each tree can take up to a few minutes to train. In particular, model_3 will probably take the longest to train.
Can you write Python code for it?

model_1 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 2,
                                min_node_size = 0, min_error_reduction=-1)
model_2 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                                min_node_size = 0, min_error_reduction=-1)
model_3 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 14,
                                min_node_size = 0, min_error_reduction=-1)


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Quiz Question: Using the complexity definition above, which model (model_4, model_5, or model_6) has the largest complexity?
Did this match your expectation?
model_4
Quiz Question: model_4 and model_5 have similar classification error on the validation set but model_5 has lower complexity. Should you pick model_5 over model_4?
model_5
Exploring the effect of min_node_size
We will compare three models trained with different values of the stopping criterion. Again, intentionally picked models at the extreme ends (too small, just right, and just right).
Train three models with these parameters:
1. model_7: min_node_size = 0 (too small)
2. model_8: min_node_size = 2000 (just right)
3. model_9: min_node_size = 50000 (too large)
For each of these three, we set max_depth = 6, and min_error_reduction = -1.
 Note: Each tree can take up to 30 seconds to train.
Can you write Python code for it?

model_7 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                                min_node_size = 0, min_error_reduction=-1)
model_8 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                                min_node_size = 2000, min_error_reduction=-1)
model_9 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                                min_node_size = 50000, min_error_reduction=-1)


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Train-Validation split
We split the data into a train-validation split with 80% of the data in the training set and 20% of the data in the validation set. We use seed=1 so that everyone gets the same result.
Can you write Python code for it?

with open('module-6-assignment-train-idx.json') as train_data_file:
    train_idx  = json.load(train_data_file)
with open('module-6-assignment-validation-idx.json') as validation_data_file:
    validation_idx = json.load(validation_data_file)

print train_idx[:3]
print validation_idx[:3]

print len(train_idx)
print len(validation_idx)

train_data = loans.iloc[train_idx]
validation_data = loans.iloc[validation_idx]

print len(loans.dtypes )


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Transform categorical data into binary features
Since we are implementing binary decision trees, we transform our categorical data into binary data using 1-hot encoding, just as in the previous assignment. Here is the summary of that discussion:
For instance, the home_ownership feature represents the home ownership status of the loanee, which is either own, mortgage or rent. For example, if a data point has the feature
{'home_ownership': 'RENT'}
we want to turn this into three features:
{
   'home_ownership = OWN'      : 0,
   'home_ownership = MORTGAGE' : 0,
   'home_ownership = RENT'     : 1
 }
Since this code requires a few Python and GraphLab tricks, feel free to use this block of code as is. Refer to the API documentation for a deeper understanding.
Can you write Python code for it?

categorical_variables = []
for feat_name, feat_type in zip(loans.columns, loans.dtypes):
    if feat_type == object:
        categorical_variables.append(feat_name)

for feature in categorical_variables:

    loans_one_hot_encoded = pd.get_dummies(loans[feature],prefix=feature)
    loans_one_hot_encoded.fillna(0)
    #print loans_one_hot_encoded

    loans = loans.drop(feature, axis=1)
    for col in loans_one_hot_encoded.columns:
        loans[col] = loans_one_hot_encoded[col]

print loans.head(2)
print loans.columns

loans.iloc[122602]


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Using the count_leaves function, compute the number of leaves in each of each models (model_7, model_8, and model_9).
Can you write Python code for it?

print "number of leaves in model_7 is : {}".format(count_leaves(model_7))
print "number of leaves in model_8 is : {}".format(count_leaves(model_8))
print "number of leaves in model_9 is : {}".format(count_leaves(model_9))


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
Using the count_leaves function, compute the number of leaves in each of each models in (model_4, model_5, and model_6).
Can you write Python code for it?

print "number of leaves in model_4 is : {}".format(count_leaves(model_4))
print "number of leaves in model_5 is : {}".format(count_leaves(model_5))
print "number of leaves in model_6 is : {}".format(count_leaves(model_6))


I am working on the file "Programming Assignment 6/Decision Trees in Practice.ipynb".
The first task is:
We then wrote a function best_splitting_feature that finds the best feature to split on given the data and a list of features to consider.
Please copy and paste your best_splitting_feature code here.
Can you write Python code for it?

def best_splitting_feature(data, features, target):

    target_values = data[target]
    best_feature = None # Keep track of the best feature
    best_error = 10     # Keep track of the best error so far
    # Note: Since error is always <= 1, we should intialize it with something larger than 1.

    # Convert to float to make sure error gets computed correctly.
    num_data_points = float(len(data))

    # Loop through each feature to consider splitting on that feature
    for feature in features:

        # The left split will have all data points where the feature value is 0
        left_split = data[data[feature] == 0]

        # The right split will have all data points where the feature value is 1
        ## YOUR CODE HERE
        right_split = data[data[feature] == 1]

        # Calculate the number of misclassified examples in the left split.
        # Remember that we implemented a function for this! (It was called intermediate_node_num_mistakes)
        # YOUR CODE HERE
        left_mistakes = intermediate_node_num_mistakes(left_split[target])

        # Calculate the number of misclassified examples in the right split.
        ## YOUR CODE HERE
        right_mistakes = intermediate_node_num_mistakes(right_split[target])

        # Compute the classification error of this split.
        # Error = (# of mistakes (left) + # of mistakes (right)) / (# of data points)
        ## YOUR CODE HERE
        error = (left_mistakes + right_mistakes) / num_data_points

        # If this is the best error we have found so far, store the feature as best_feature and the error as best_error
        ## YOUR CODE HERE
        if error < best_error:
            best_feature = feature
            best_error = error

    return best_feature # Return the best feature we found


I am working on the file "Programming Concepts.ipynb".
The first task is:
Above, Python has inferred the type of the variable pizza to be an integer. Since strings can only be concatenated with other strings, our print statement generates an error. There are two ways we can resolve the error:

Declare the pizzas variable as type string (str) on assignment or
Re-cast the pizzas variable as a string within the print statement.
Can you write Python code for it?

customer = 'Carol'
pizzas = str(2)
print(customer + ' ordered ' + pizzas + ' pizzas.')

customer = 'Carol'
pizzas = 2
print(customer + ' ordered ' + str(pizzas) + ' pizzas.')


I am working on the file "Programming Concepts.ipynb".
The first task is:
Info on additional list methods is available at https://docs.python.org/3/library/stdtypes.html?highlight=lists#mutable-sequence-types
Because lists are ordered, it is possible to access list items by referencing their positions. Note that the position of the first item in a list is 0 (zero), not 1!
Can you write Python code for it?

string_list = ['apples', 'oranges', 'pears', 'grapes', 'pineapples']

string_list[0]

# We can use positions to 'slice' or selection sections of a list:

string_list[3:]

string_list[:3]

string_list[1:4]

# If we don't know the position of a list item, we can use the 'index()' method to find out.
# Note that in the case of duplicate list items, this only returns the position of the first one:

string_list.index('pears')

string_list.append('oranges')

string_list

string_list.index('oranges')

# one more time with lists and dictionaries
list_ex1 = my_stuff[0] + my_stuff[1] + int(my_stuff[2])
print(list_ex1)

list_ex2 = (
    str(my_stuff[0])
    + str(my_stuff[1])
    + my_stuff[2]
    + my_stuff[3][0]
)
print(list_ex2)

dict_ex1 = (
    more_stuff['item1']
    + more_stuff['item2']
    + int(more_stuff['item3'])
)
print(dict_ex1)

dict_ex2 = (
    str(more_stuff['item1'])
    + str(more_stuff['item2'])
    + more_stuff['item3']
)
print(dict_ex2)

# Now try it yourself ...
# print out the phrase "The answer: 42" using the following
# variables and one or more of your own and the 'print()' function
# (remember spaces are characters as well)

start = "The"
answer = 42


I am working on the file "Programming Concepts.ipynb".
The first task is:
Lists
https://docs.python.org/3/library/stdtypes.html?highlight=lists#list
Lists are a type of collection in Python. Lists allow us to store sequences of items that are typically but not always similar. All of the following lists are legal in Python:
Can you write Python code for it?

# Separate list items with commas!

number_list = [1, 2, 3, 4, 5]
string_list = ['apples', 'oranges', 'pears', 'grapes', 'pineapples']
combined_list = [1, 2, 'oranges', 3.14, 'peaches', 'grapes', 99.19876]

# Nested lists - lists of lists - are allowed.

list_of_lists = [[1, 2, 3], ['oranges', 'grapes', 8], [['small list'], ['bigger', 'list', 55], ['url_1', 'url_2']]]


I am working on the file "Programming Concepts.ipynb".
The first task is:
Lists in Python are:

mutable - the list and list items can be changed
ordered - list items keep the same "place" in the list

Ordered here does not mean sorted. The list below is printed with the numbers in the order we added them to the list, not in numeric order:
Can you write Python code for it?

ordered = [3, 2, 7, 1, 19, 0]
ordered

# There is a 'sort' method for sorting list items as needed:

ordered.sort()
ordered


I am working on the file "Programming Concepts.ipynb".
The first task is:
Operators
If objects are the nouns, operators are the verbs of a programming language. We've already seen examples of some operators: assignment with the = operator, arithmetic addition and string concatenation with the + operator, arithmetic division with the / and - operators, and comparison with the > operator. Different object types have different operators that may be used with them. The Python Documentation provides detailed information about the operators and their functions as they relate to the standard object types described above.
Flow Control and Logical Tests
Flow control commands allow for the dynamic execution of parts of the program based upon logical conditions, or processing of objects within an iterable object (like a list or dictionary). Some key flow control commands in python include:

while-else loops that continue to run until the termination test is False or a break command is issued within the loop:
done = False
i = 0
while not done:
    i = i+1
    if i > 5: done = True

if-elif-else statements defined alternative blocks of code that are executed if a test condition is met:
do_something = "what?"
if do_something == "what?":
    print(do_something)
elif do_something == "where?":
    print("Where are we going?")
else:
    print("I guess nothing is going to happen")

for loops allow for repeated execution of a block of code for each item in a python sequence such as a list or dictionary. For example:
my_stuff = ['a', 'b', 'c']
for item in my_stuff:
    print(item)

a
b
c

Functions
Functions represent reusable blocks of code that you can reference by name and pass informatin into to customize the exectuion of the function, and receive a response representing the outcome of the defined code in the function. When would you want to define a function? You should consider defining a function when you find yourself entering very similar code to execute variations of the same process. The dataset used for the following example is part of the supplementary materials (Data S1 -...
Can you write Python code for it?

# read data into a list of dictionaries
import csv

# create an empty list that will be filled with the rows of data from the CSV as dictionaries
csv_content = []

# open and loop through each line of the csv file to populate our data file
with open('aaj1945_DataS1_Egg_shape_by_species_v2.csv') as csv_file:
    csv_reader = csv.DictReader(csv_file)
    for row in csv_reader:             # process each row of the csv file
        csv_content.append(row)

print(csv_content[0].keys())
#print()
#print(csv_content[0])

# extract content of each "column" individually

order = []
for item in csv_content:
    try:
        order.append(item['Order'])
    except:
        order.append(None)

family = []
for item in csv_content:
    try:
        family.append(item['Family'])
    except:
        family.append(None)

species = []
for item in csv_content:
    try:
        species.append(item['Species'])
    except:
        species.append(None)

asymmetry = []
for item in csv_content:
    try:
        asymmetry.append(item['Asymmetry'])
    except:
        asymmetry.append(None)

ellipticity = []
for item in csv_content:
    try:
        ellipticity.append(item['Ellipticity'])
    except:
        ellipticity.append(None)

avgLength = []
for item in csv_content:
    try:
        avgLength.append(item['AvgLength (cm)'])
    except:
        avgLength.append(None)

noImages = []
for item in csv_content:
    try:
        noImages.append(item['Number of images'])
    except:
        noImages.append(None)

noEggs = []
for item in csv_content:
    try:
        noEggs.append(item['Number of eggs'])
    except:
        noEggs.append(None)

print(order[0:3])
print(family[0:3])
print(species[0:3])
print(asymmetry[0:3])
print(ellipticity[0:3])
print(avgLength[0:3])
print(noImages[0:3])
print(noEggs[0:3])

# define a function that can extract a named column from a named list of dictionaries
def extract_column(source_list, source_column):
    new_list = []
    for item in source_list:
        try:
           ...


I am working on the file "Programming Concepts.ipynb".
The first task is:
The Basics
At the core of Python (and any programming language) there are some key characteristics of how a program is structured that enable the proper execution of that program. These characteristics include the structure of the code itself, the core data types from which others are built, and core operators that modify objects or create new ones. From these raw materials more complex commands, functions, and modules are built.
For guidance on recommended Python structure refer to the Python Style Guide.
Examples: Variables and Data Types
The Interpreter
Can you write Python code for it?

# The interpreter can be used as a calculator, and can also echo or concatenate strings.

3 + 3

3 * 3

3 ** 3

3 / 2 # classic division - output is a floating point number

# Use quotes around strings

'dogs'

# + operator can be used to concatenate strings

'dogs' + "cats"

print('Hello World!')


I am working on the file "Programming Concepts.ipynb".
The first task is:
The above output for typed_list may seem odd. Referring to the documentation, we see that the argument to the type constructor is an iterable, which according to the documentation is "An object capable of returning its members one at a time." In our construtor statement above
```
Using the type constructor
constructed_list = list('purple')
```
the word 'purple' is the object - in this case a word - that when used to construct a list returns its members (individual letters) one at a time.
Compare the outputs below:
Can you write Python code for it?

constructed_list_int = list(123)

constructed_list_str = list('123')
constructed_list_str


I am working on the file "Programming Concepts.ipynb".
The first task is:
There are multiple ways to create a list:
Can you write Python code for it?

# Create an empty list

empty_list = []

# As we did above, by using square brackets around a comma-separated sequence of items

new_list = [1, 2, 3]

# Using the type constructor

constructed_list = list('purple')

# Using a list comprehension

result_list = [i for i in range(1, 20)]


I am working on the file "Programming Concepts.ipynb".
The first task is:
Try It Yourself
Go to the section 4.4. Numeric Types in the Python 3 documentation at https://docs.python.org/3.4/library/stdtypes.html. The table in that section describes different operators - try some!
What is the difference between the different division operators (/, //, and %)?
Variables
Variables allow us to store values for later use.
Can you write Python code for it?


I am working on the file "Programming Concepts.ipynb".
The first task is:
Variable data types can be inferred, so Python does not require us to declare the data type of a variable on assignment.
Can you write Python code for it?


I am working on the file "Programming Concepts.ipynb".
The first task is:
Variables can be reassigned:
Can you write Python code for it?


I am working on the file "Programming Concepts.ipynb".
The first task is:
We can inspect our lists:
Can you write Python code for it?

empty_list

new_list

result_list

constructed_list


I am working on the file "Programming Concepts.ipynb".
The first task is:
is equivalent to
Can you write Python code for it?

a = int(5)
type(a)

c = 'dogs'
print(type(c))

c = str('dogs')
print(type(c))


I am working on the file "Programming Lesson Modules/Module 1- Import Local CSVs.ipynb".
The first task is:
Normally, you're file of interest is not located in the default working directory of Python. os.chdir() changes the working directory so we can open our file of interest.
Can you write Python code for it?

filepath = 'C:/Users/Radley/Downloads/' # store file location as a string
filename = 'lbl.csv'                   # store the file name as a string


I am working on the file "Programming Lesson Modules/Module 1- Import Local CSVs.ipynb".
The first task is:
The following module is an example of reading a CSV and importing the data as variables in Python.  In many circumstances, you will want to import the data so that it can be used in Python.  In addition, I use csv.reader with different, more efficient syntax:
Can you write Python code for it?

def importlocalCSV(filepath, filename):
    datetime = []
    cpm = []
        # Set up an empty lists so you can place the data into it.
    line = 0
        # We want to ignore the 1st row of headers, so we set a variable to
        # count the row we are iterating on.

    file = os.path.join(filepath,filename)
    with open(file) as csvfile:
        # It is good practice to use the with statement when dealing with files
        # because it automatically closes the file after the nested lines
        # This with command performs same function as csvfile = open(file)

        reader = csv.reader(csvfile)
        for row in reader:
            if line != 0:
                # Conditional statement to ignore first row of headers
                datetime.append(row[0])
                    # Append means ATTACH.  We are attaching data in the 1st
                    # column to our list, row by row
                cpm.append(float(row[6]))
                    # The command float records each element as a numerical
                    # value rather than a character string.
            line = line + 1
                    # Progress iteration forward by adding +1 to line
    return datetime, cpm


I am working on the file "Programming Lesson Modules/Module 1- Import Local CSVs.ipynb".
The first task is:
The following module will read the CSV with csv.reader and display the data in your console one row at a time. This display command inserts a comma and space between each data entry, effectively reprinting the .CSV in your Python console.
Can you write Python code for it?

def printlocalCSV(filepath, filename):
    file = os.path.join(filepath,filename)      # join filepath/name into one string
    csvfile = open(file)                 # open the file with Python
    reader = csv.reader(csvfile)         # read the CSV with csv.reader
    for row in reader:                   # for every row in the CSV...
        print(', '.join(row))
            # Display/print the CSV with a comma and space separating each element
    csvfile.close()                      # close the file


I am working on the file "Programming Lesson Modules/Module 2- Import Web CSVs.ipynb".
The first task is:
The io module allows Python to deal with objects formatted as bytes. Web sources are usually formatted in HTTP/bytes, rendering it incompatible with default Python modules.
The urllib module provides an interface to fetch data from the Internet.
The following lines will: access the DDL, makes the file compatible to Python, and prints the CSV.  Take note that only the first two lines are different from reading a .CSV from your local disk storage.
Can you write Python code for it?

def printwebCSV(url):
    response = urllib.request.urlopen(url)
        # This line will fail without internet access.
    csvfile = io.TextIOWrapper(response)
        # io.TextIOWrapper decodes HTTP data and encodes the data as string
        # objects that can be understood by Python
    reader = csv.reader(csvfile)
    for row in reader:
        print(', '.join(row))


I am working on the file "Programming Lesson Modules/Module 2- Import Web CSVs.ipynb".
The first task is:
The next commands are an example of importing a CSV from a DDL.  It also uses more compact syntax in order to reduce the number of lines:
Can you write Python code for it?

def importwebCSV(url):
    response = urllib.request.urlopen(url)
    reader = csv.reader(io.TextIOWrapper(response))
    datetime = []
    cpm = []
    line = 0
    for row in reader:
        if line != 0:
            datetime.append(row[0])
            cpm.append(float(row[6]))
        line += 1
        # Python syntax for line = line + 1 (+1 to current value for line)
    return (datetime,cpm)


I am working on the file "Programming Lesson Modules/Module 3- Simple Plots and Histograms.ipynb".
The first task is:
Alternatively, you can create a scatter plot by with a modifier in plt.plot's inputs. The following module creates a simple scatter plot with CPM data from our device in Lawrence Berkeley.
For more insight into line, axes, and other plot modifiers, matplotlib's official website contains detailed documentation on plotting and graphic design.
Can you write Python code for it?

def scatter(cpm, plot_title):
    plt.plot(cpm,'ro')
        # The 'ro' modifier after cpm_lbl does two things:
        # 'r' changes the color to red and 'o' creates points instead of lines
    plt.ylabel('Counts Per Minute')
    plt.title(plot_title)

scatter(cpm_lbl,'LBL DoseNet Measurements: Scatter Plot')


I am working on the file "Programming Lesson Modules/Module 3- Simple Plots and Histograms.ipynb".
The first task is:
First we want to import two sets of data from DoseNet:
You should recognize the following steps!  If you are not famiar with importing data from a DDL, then check previous modules on retrieving and importing CSVs.
Can you write Python code for it?

def importwebCSV(url):
    response = urllib.request.urlopen(url)
    reader = csv.reader(io.TextIOWrapper(response))
    datetime = []
    cpm = []
    line = 0
    for row in reader:
        if line != 0:
            datetime.append(row[0])
            cpm.append(float(row[6]))
        line += 1
        # Python syntax for line = line + 1 (+1 to current value for line)
    return (datetime,cpm)

url_etch = 'https://radwatch.berkeley.edu/sites/default/files/dosenet/etch.csv'
datetime_etch, cpm_etch = importwebCSV(url_etch)
    # run function and store return values as datetime_etch and cpm_etch

url_lbl = 'http://radwatch.berkeley.edu/sites/default/files/dosenet/lbl.csv'
datetime_lbl, cpm_lbl = importwebCSV(url_lbl)
    # run function and store return values as datetime_lbl and cpm_lbl


I am working on the file "Programming Lesson Modules/Module 3- Simple Plots and Histograms.ipynb".
The first task is:
Histograms are a graphics that depict distributions of data.  Histograms are incredibly useful in statistical analysis of raw data.  Key pieces of information conveyed by a histogram include: mean, standard deviation, and data stability.
The following commands demonstrate the plt.hist command.  The plt.hist command organizes a list of data into a histogram, enabling us to streamline a process that can be extremely tedious by hand.
Can you write Python code for it?

def histogram(cpm, plot_title):
    plt.hist(cpm,bins=31)
    plt.ylabel('Frequency')
    plt.xlabel('Counts Per Minute')
    plt.title(plot_title)

histogram(cpm_etch,'Etcheverry DoseNet Measurements: Histogram')


I am working on the file "Programming Lesson Modules/Module 3- Simple Plots and Histograms.ipynb".
The first task is:
The plot command can be used in conjunction with a variety of other commands to create complex graphics.  Two straightforward examples of this are subplots and overlaid plots.  Subplots allow multiple plots to be placed onto single canvas.  Similarly, there are also methods to place multiple sets of data on a single axis to create overlapping plots.
The following example combines everything that had been introduced in this module and incorporates subplots and overlapping plots.
Can you write Python code for it?

def subplot_overlay(cpm1, plot_title1, cpm2, plot_title2, cpm3, cpm4, plot_title3):
    plt.subplot(2,2,1)
    # This means you are making a subplot with 2 rows, 2 columns, and are
    # currently working on the 1st one (plot in top left)
    scatter(cpm1,plot_title1)

    plt.subplot(2,2,2)
    # subplot with 2 rows, 2 columns, 2nd plot (top right)
    histogram(cpm2,plot_title2)

    plt.subplot(2,1,2)
    # 2 rows, 1 column, 2nd plot.  This plot will be placed on the 2nd plot
    # (bottom-most plot) as if it were a 2-by-1 grid
    p1, = line(cpm3,''),
    p2, = line(cpm4,''),
    # Note: The comma after the statement is necessary or plt.legend will fail
    plt.title(plot_title3)

    legend_labels = ['cpm3', 'cpm4']
    plt.legend(legend_labels, loc='best')
    # Giving the plots labels enables us to create a legend

    plt.tight_layout()
    # Tightens title and axes labels and prevents overlapping

    plt.show()

subplot_overlay(cpm_etch, 'Etcheverry Scatter Plot',
                cpm_lbl, 'LBL Histogram',
                cpm_lbl, cpm_etch, 'LBL (cpm3) and Etcheverry (cpm4) Line Plot')


I am working on the file "Programming Lesson Modules/Module 4- Example Plot of Weather Data.ipynb".
The first task is:
There are a few problems with this current plot.  Notably, the x-ticks are overlapping and it is difficult to examine the Cesium-137 data because it is so small.  Regarding the x-ticks, the labels can be made more visible by rotating the tick-labels.  In addition, the Cesium data can be made more visible with a logarithmic plot:
Can you write Python code for it?

def weather_plot2(timedata, Bi214, Cs137):
    weather_plot1(timedata, Bi214, Cs137)
    # the following commands are simple adjustments to the axes:
    plt.xticks(rotation=30)
    plt.yscale('log')
    plt.title('AirMonitor Data: Bi-214 and Cs-137 CPS from {0}-{1} to {2}-{3}'
              .format(timedata[0].month, timedata[0].day,
                      timedata[-1].month, timedata[-1].day))
    plt.show()

weather_plot2(timedata, Bi214, Cs137)


I am working on the file "Programming Lesson Modules/Module 4- Example Plot of Weather Data.ipynb".
The first task is:
While these plots are fine, many professionally-made graphics are thoroughly controlled and every aspect of the plot is kept in mind.  The following example is a more comprehensive approach to plotting.  Also, this example will calculate error and include error bars in the final plot.  The final plot is similar to the plot in the AirMonitor website: radwatch.berkeley.edu/airsampling
Can you write Python code for it?

def weather_plot3(timedata, Bi214, Cs137):
    import numpy as np
        # the next module explains numpy in more detail, but numpy is used
        # here to perform a square-root operation for error.

    # 1st step: plot the data
    fig, ax = plt.subplots()
    ax.plot(timedata, Bi214, 'ro-', label='Bismuth-214', linestyle='none')
    ax.errorbar(timedata, Bi214, yerr=np.sqrt(Bi214)/60, fmt='ro', ecolor='r')
    ax.plot(timedata, Cs137, 'bs-', label='Cesium-137', linestyle='none')
    ax.errorbar(timedata, Cs137, yerr=np.sqrt(Cs137)/60, fmt='bs', ecolor='b')
        # error is given by np.sqrt(Bi214)/60 and is based on the conversion
        # from hourly countrate to counts-per-second
        # for ax.errorbar, fmt=format and is identical to corresponding plot
        # ecolor=line color and is identical to corresponding plot for same reason.

    # 2nd step: legend and axis manipulations:
    plt.legend(loc='best')
    plt.yscale('log')
        # you can decide if this plot is better/worse in log scale

    # 3rd step: format ticks along axis; we will use matplotlib's built-in
    # datetime commands to format the axis:
    ax.xaxis.set_major_locator(mdates.DayLocator())
        # ticks on x-axis day-by-day basis
    ax.xaxis.set_major_formatter(mdates.DateFormatter('%m-%d'))
        # tick labels only occur on days in the format: Month-Day
        # you can customize the format, i.e. '%m-%d-%Y %H:00' would be
        # Month-Day-Year Hour:00
    ax.xaxis.set_minor_locator(mdates.HourLocator())
        # minor ticks on x-axis occur on hour marks
    plt.xticks(rotation=30)

    # 4th step: titles and labels
    plt.title('AirMonitor Data: Bi-214 and Cs-137 CPS from {0}-{1} to {2}-{3}'
              .format(timedata[0].month, timedata[0].day, timedata[-1].month, timedata[-1].day))
    plt.xlabel('Time')
    plt.ylabel('counts per second')

weather_plot3(timedata, Bi214, Cs137)


I am working on the file "Programming Lesson Modules/Module 5- Other Forms of Visualization.ipynb".
The first task is:
The following commands will create a  table with 4 rows, each containing an isotope (Bi-214, K-40, Cs-134, and Cs-137).  There will be 5 columns with the following information: isotope name, mean concentration, median concentration, max concentration measured, and time of occurrence:
Can you write Python code for it?

def table(bi214, k40, cs134, cs137):
    RowLabel = ("Bi-214", "K-40", "Cs-134", "Cs-137")
    ColLabel = ("isotope", "mean CPS", "median CPS",
                "max CPS", "time of occurrence")

    # The statistical meaning of mean and a computational method to obtain
    # it are explored in a different module.
    # For this module, we will use the numpy function 'np.mean'
    mean_data = (np.mean(bi214), np.mean(k40), np.mean(cs134), np.mean(cs137))
    mean_data = np.around(mean_data, decimals=4)
        # np.around rounds elements in a list to the chosen decimal point.

    # For median, we will use the numpy function 'np.median'
    median_data = np.around((np.median(bi214), np.median(k40), np.median(cs134), np.median(cs137)), decimals=4)

    # Numpy also has a function to scan a list for the max value contained in that list!
    max_data = np.around((np.max(bi214), np.max(k40), np.max(cs134), np.max(cs137)), decimals=4)
    # Each max CPS has a corresponding datetime component with the same index.
    # I will use LIST.index(max(LIST)) to find these corresponding indices.
    # Note: this method's weakness is that  it only identifies the first
    # occurrence of a maximum; if there the max occurs multiple times it will
    # not acknowledge them.  Can you think/find a way to do this in a better way?
    time_data = (timedata[Bi214.index(np.max(bi214))], timedata[k40.index(np.max(k40))],
                 timedata[Cs134.index(np.max(cs134))], timedata[cs137.index(np.max(cs137))])
                # if you have trouble understanding the previous commands, make
                # sure you understand each line piece their purpose together!

    data_array = np.vstack((RowLabel,mean_data,median_data, max_data,time_data)).T
        # vstack places the lists atop eachother and '.T' transposes the
        # information so it'll appear as it should on the table.
    fig, ax = plt.subplots()
    ax.axis('off')
        # By default, matplotlib will always plot against axes.
        #...


I am working on the file "Programming Lesson Modules/Module 5- Other Forms of Visualization.ipynb".
The first task is:
The following module creates a pi chart that illustrates the breakdown of which isotope is measured to have the highest concentration during the given interval of our data.
In order to do this, we start with an for-loop script that tallies which isotope has the highest concentration measurement at each index until it reaches the end of the data:
Can you write Python code for it?

def pie_chart(activity1, activity2, activity3, activity4, labels):
    tally = [0,0,0,0]
    for i in range(0,len(bi214)-1):
        comparing_list = [bi214[i],k40[i], cs134[i], cs137[i]]
        if max(comparing_list) == bi214[i]:
            tally[0] += 1
        elif max(comparing_list) == k40[i]:
            tally[1] += 1
        elif max(comparing_list) == cs134[i]:
            tally[2] += 1
        else:
            tally[3] += 1

    # plt.pie places the items counter-clockwise
    total_counts = sum(tally)
    fracs = [tally[0]/total_counts, tally[1]/total_counts,
            tally[2]/total_counts, tally[3]/total_counts]
    explode = (0,0,0,0)   # explode refers to "pulling out" certain items for exaggerated effect
    plt.pie(fracs, explode=explode, labels=labels, autopct='%1.1f%%')
        # 'autopct='1.1f%%' displays percent value for each slice
    plt.axis('equal')
        # set the pie chart against a square axis, otherwise it won't be circular.
    plt.title('Air Monitor Data:\nBreakdown of Highest Measured Count Rate in
              Isotopes\nof Interest from %s to %s' %(timedata[0],timedata[-1]))
        # this title string demonstrates a line break (signified by \n after
        # isotopes) and string interpolation (represented by %s).
    plt.show()

    # a more interesting example pie chart would breakdown total activity in each isotope:
    activity = [sum(bi214), sum(k40), sum(cs134), sum(cs137)]
    total_activity = sum(activity)

    fracs = [activity[0]/total_activity, activity[1]/total_activity,
            activity[2]/total_activity, activity[3]/total_activity]
    explode = (0,0,0,0)
    plt.pie(fracs, explode=explode, labels=labels, autopct='%1.1f%%')
    plt.axis('equal')
    plt.title('Air Monitor Data:\nBreakdown of Total Activity by Isotope of
              Interest\nfrom %s to %s' %(timedata[0],timedata[-1]))
    plt.show()

labels = ['Bi214','K40','Cs134','Cs137']
pie_chart(Bi214, K40, Cs134, Cs137, labels)


I am working on the file "Programming Lesson Modules/Module 6- Data Binning.ipynb".
The first task is:
For this example, I will bin DoseNet data from our device on the Etcheverry Roof and average the data.  Afterwards, I will plot the data to show the consequences of data binning.  While I perform this technique with my own code, it is important to note that there are many built-in functions within Python and plenty of examples of data binning you can find online.
Can you write Python code for it?

def month_bin(timedata, cpm, error):
    # First, we initialize important values.
    Year = [timedata[-1].year]
    Month = [timedata[-1].month]
    sumCPM = [0]                          # because mean = sum/total, we can just add CPM values as we iterate along the month
    sumError = [0]
    DataCount = [0]                       # variable to count total number of points per month
    bin_num = 0                           # variable to track current month; also tracks number of bins
    for i in range(len(counts)-1,0,-1):   # iterate from last point to first point on CSV in steps of -1
        if Year[bin_num] == timedata[i].year:        # if current bin year is same as iterated year
            if Month[bin_num] == timedata[i].month:  # if current bin month is same as iterated month, they belong in same bin!
                sumCPM[bin_num] += counts[i]
                sumError[bin_num] += CPMerror[i]     # so we collect the data we need...
                DataCount[bin_num] += 1
            else:
                Year.append(timedata[i].year)        # if the months don't match:
                Month.append(timedata[i].month)
                sumCPM.append(0)                     # add another bin by appending 0 and increasing bin_num
                sumError.append(0)
                DataCount.append(0)
                bin_num += 1
        else:                                        # if current bin year doesn't match iterated year:
            Year.append(timedata[i].year)
            Month.append(timedata[i].month)          # add another bin by appending 0 and increasing bin_num
            sumCPM.append(0)
            sumError.append(0)
            DataCount.append(0)
            bin_num += 1

    binnedCPM = np.array(sumCPM) / np.array(DataCount)     # np.array allows us to perform element-by-elemtent division

    avgError = np.array(sumError) / np.array(DataCount)
    stdError = avgError / np.sqrt(DataCount)               # standard error is average...


I am working on the file "Programming Lesson Modules/Module 7- Data Sorting and Searching.ipynb".
The first task is:
Now that we've performed a basic sorting operation, let's make a function that searches in which rainfall is detected (including trace amounts of rainfall, 'T').  To do this, we can use a technique called list comprehension to compile indices that fulfill our requirements.  The list comprehension command I use is:
indices = [i for i, target in enumerate(rainfall) if target > 0]

which breaks down to:
indices = []                               # initialize with an empty list
    for i, target in enumerate(my_list):
        if target > 0
            indices.append(i)
Can you write Python code for it?

def search_func(data):
    rainfall = list(data[3:,].flatten())
    indices_trace = [i for i, target in enumerate(rainfall) if target == 'T']

    # Next, we replace indices where 'T' appears with 0 so it doesn't interfere with next search
    for index in indices_trace:
        rainfall[index] = 0

    # Lastly, we convert list to floating values so we can compare values them numerically
    rainfall = [float(j) for j in rainfall]
    indices_rain = [i for i, target in enumerate(rainfall) if target > 0]

    # When we combine the two indices, we place T before the numerical values
    search_index = indices_trace + indices_rain
    return search_index

print(data[1:,search_func(data)])


I am working on the file "Programming Lesson Modules/Module 7- Data Sorting and Searching.ipynb".
The first task is:
The command below prints the first 9 columns of sort_func('temp'), excluding the first row.  I ignore the first row of dates because it clutters the array with datetime information, but you can change to the index to [:,:9] to include it.  Finally, I utilize the repr function to print the data in a cleaner array format.
In order to make this information more concrete: notice how the data with the coldest days (1st row) generally correlate with days of rain.
Can you write Python code for it?

print(repr(sort_func('temp')[1:,:9]))

print(repr(sort_func('wind')[1:,:9]))

print(repr(sort_func('rain')[1:,:8]))


I am working on the file "Programming assignments.ipynb".
The first task is:
Q1. Given a binary tree, return true if it is a BST. Do it in a non-recursive way.
Do DFS and at each step check the condition for BST. If fail return False. If passes all conditions return True.
Can you write Python code for it?

def append_bst(graph, root, item):
    if root is None:
        graph[item] = [None, None]
        return graph
    curr = root
    while (item < curr) or (item > curr):
        if item < curr:
            if graph[curr][0] is not None:
                curr = graph[curr][0]
            else:
                graph[curr][0] = item
                graph[item] = [None, None]
        elif item > curr:
            if graph[curr][1] is not None:
                curr = graph[curr][1]
            else:
                graph[curr][1] = item
                graph[item] = [None, None]
    return graph

def gen_bst(arr):
    graph = append_bst({}, None, arr[0])
    root = arr[0]
    for item in arr[1:]:
        graph = append_bst(graph, root, item)
        #print item, graph
    return graph, root

print "Final graph:\n", gen_bst(range(10))

print "Final graph:\n", gen_bst([4,5,3,2,10,1,7])

def print_graph(graph, root):
    visited, queue = (set([None]), [root])
    level = 0
    while queue:
        curr = queue.pop(0)
        print curr
        if curr not in visited:
            visited.add(curr)
            queue.extend(set(graph[curr]) - visited)

print_graph(*gen_bst([4,5,3,2,10,1,7]))

min([1, 2, 3])

def is_bst(graph, root):
    if root is None:
        return True
    visited, stack = (set([None]), [root])
    while stack:
        curr = stack.pop()
        left, right = graph[curr]
        if left is not None and left > curr:
            # Left is always smaller than parent
            return False
        if right is not None and right < curr:
            # Right is always greater than parent
            return False
        if curr not in visited:
            visited.add(curr)
            print curr
            stack.extend(set(graph[curr]) - visited)
    return True

graph, root = gen_bst([4,5,3,2,10,1,7])
is_bst(graph, root)

graph = {0: [1,2],
         1: [None, None],
         2: [None, None]}
root = 0
is_bst(graph, root)


I am working on the file "Programming assignments.ipynb".
The first task is:
Q2. There is a 2D matrix of size 10 X 10, where you have to begin from the location (0,0) and move to the location (9,9). You can either move right on down. Find out the number of distinct paths in which you can reach (9,9) from (0,0).
paths(i,j) = #paths(i-1, j-1) + 2
Can you write Python code for it?

def npaths(N=10):
    mat = [[0]*N for k in range(N)]
    mat[0][0] = 1
    for i in range(N):
        for j in range(N):
            if mat[i][j] == 0:
                if i == 0:
                    mat[i][j] = mat[i][j-1] # Only one movement from left cell
                    #print "mat[%s][%s] = mat[%s][%s]" % (i,j,i,j-1)
                    #print_mat(mat)
                elif j == 0:
                    mat[i][j] = mat[i-1][j] # Only one movement from top cell
                    #print "mat[%s][%s] = mat[%s][%s]" % (i,j,i-1,j)
                else:
                    mat[i][j] = mat[i-1][j] + mat[i][j-1]
                    #print "mat[%s][%s] = mat[%s][%s]+1" % (i,j,i-1,j-1)
    return mat

def print_mat(mat):
    for row in mat:
        print row

print_mat(npaths(N=2))

print_mat(npaths(N=3))

print_mat(npaths(N=10))


I am working on the file "Programming/LANG_Python/2018-01-17_threading.ipynb".
The first task is:
Example 1: run multiple threads
Can you write Python code for it?

def worker1():
    print(threading.currentThread().getName(), '--begin')
    time.sleep(3)
    print(threading.currentThread().getName(), '--end')

def worker2():
    print(threading.currentThread().getName(), '--BEGIN')
    time.sleep(2)
    print(threading.currentThread().getName(), '--END')

w1 = threading.Thread(name='firstWorker', target=worker1)
w2 = threading.Thread(name='secondWorker', target=worker2)
w3 = threading.Thread(name='thirdWorker', target=worker1)
w4 = threading.Thread(target=worker2)

w1.start()
w2.start()
w3.start()
w4.start()

# threads run asynchronously and look messy in the results.
# So, logging module can help tracking threads.


I am working on the file "Programming/LANG_Python/2018-01-17_threading.ipynb".
The first task is:
Example 2: use logging for tracking threads
Can you write Python code for it?

logging.basicConfig(level=logging.DEBUG,
                   format='[%(levelname)s] (%(threadName)-10s) %(message)s')

def worker1():
    logging.debug('--begin') # <-- use logging instead of print function!
    time.sleep(3)
    logging.debug('--end')

def worker2():
    logging.debug('--BEGIN')
    time.sleep(2)
    logging.debug('--END')

w1 = threading.Thread(name='firstWorker', target=worker1)
w2 = threading.Thread(name='secondWorker', target=worker2)
w3 = threading.Thread(name='thirdWorker', target=worker1)
w4 = threading.Thread(target=worker2)

w1.start()
w2.start()
w3.start()
w4.start()


I am working on the file "Programming/LANG_Python/2018-01-17_threading.ipynb".
The first task is:
Example 4: threads with locking
Can you write Python code for it?

num_threads = 0
lock = allocate_lock() # make a lock object
def elephant(a):
    global num_threads
    lock.acquire() # start locking
    num_threads += 1
    time.sleep(0.1)
    num_threads -= 1
    lock.release() # releasing locking
    print(num_threads)

start_new_thread(elephant,(99,))
start_new_thread(elephant,(999,))
start_new_thread(elephant,(1733,))
start_new_thread(elephant,(17334,))

if num_threads == 0:
    print('num_threads is 0')
else:
    print('num_threads is not 0')


I am working on the file "Programming/LANG_Python/2018-01-17_threading.ipynb".
The first task is:
Example 5: daemon vs. non-daemon threads
Can you write Python code for it?

# Daemon threads die when the main program dies.
# Non-daemon threads do not die until the work is done.
# NB. Default is non-daemon mode.

logging.basicConfig(level=logging.DEBUG,
                    format='(%(threadName)-10s) %(message)s')

def daemon():
    logging.debug('Start')
    time.sleep(3)
    logging.debug('Done')

d = threading.Thread(name='Daemon', target=daemon)
d.setDaemon(True)

def non_daemon():
    logging.debug('Start')
    logging.debug('Done')

t = threading.Thread(name='Non-daemon', target=non_daemon)

d.start()
t.start()

# It's supposed not to show the last line "[DEBUG] (Daemon    ) Done"
# But, Jupyter Notebook seemingly runs each cell as a program.
# If you run in the command line as
#   $ python test.py
# it will throw logs:
#    [DEBUG] (Daemon    ) Start
#    [DEBUG] (Non-daemon) Start
#    [DEBUG] (Non-daemon) Done
#
# This program exits before the daemon thread finishes.
#
# Adding below lines will make the program waits until daemon threads finish.
#
# d.join()
# t.join()


I am working on the file "Programming/LANG_Python/2018-01-17_threading.ipynb".
The first task is:
Example 6: subclassing thread (simple)
Can you write Python code for it?

logging.basicConfig(level=logging.DEBUG,
                    format='(%(threadName)-10s) %(message)s')

class SimpleThread(threading.Thread):
    def run(self):
        logging.debug('Running now...')
        return

for i in range(4):
    t = SimpleThread()
    t.start()


I am working on the file "Programming/LANG_Python/2018-01-17_threading.ipynb".
The first task is:
Example 7: subclassing thread (complex)
Can you write Python code for it?

# If you want to customize threads,
# make instance attributes so that they can be seen.

logging.basicConfig(level=logging.DEBUG,
                    format='(%(threadName)-10s) %(message)s')

class AwesomeThreads(threading.Thread):
    def __init__(self, group=None, target=None, name=None,
                args=(), kwargs=None, daemon=None):
        # This redefining let you control threads and save values
        threading.Thread.__init__(self, group=group, target=target,
                                  name=name, daemon=daemon)
        self.args = args
        self.kwargs = kwargs
        return

    def run(self):
        logging.debug('My args={}, My kwargs={}'.format(self.args, self.kwargs))
        return

for i in range(6):
    t = AwesomeThreads(args=(i,), kwargs={'하나':1, '둘':2})
    t.start()


I am working on the file "Programming_Lab_I/Programming Lab Sheet 1.ipynb".
The first task is:
Exercise 2: Body-Mass-Index
Can you write Python code for it?

def get_category(bmi):
    '''
    A function which returns the category for a given bmi
    '''
    if bmi < 15.0:
        return 'Very severely underweight'
    elif bmi < 16.0:
        return 'Severely underweight'
    elif bmi < 18.5:
        return 'Underweight'
    elif bmi < 25.0:
        return 'Normal'
    elif bmi < 30.0:
        return 'Overweight'
    elif bmi < 35.0:
        return 'Moderately obese'
    elif bmi < 40.0:
        return 'Severely obese'
    else:
        return 'Very severely obese'

def get_bmi(weight, height):
    '''
    This function takes the weight in [kg] and the height in [m]
    It return the BMI.
    '''
    return weight / height**2

def bmi_calculator(weight=None, height=None):
    '''
    A calculator which takes weight and heigt.
    It prints the bmi and corresponding category.
    If no or only one value is given it asks for both.
    '''
    if weight == None or height == None:
        print('This is a BMI calculator.')

        weight = float(input('Enter your weight (kg): '))
        height = float(input('Enter your height (m): '))
    else:
        status = 'Your weight is: {}\nYour height is: {}'
        print(status.format(weight, height))

    bmi = get_bmi(weight, height)
    category = get_category(bmi)

    skeletton = 'Your BMI is {:.1f}\nYour BMI category is {}'
    print(skeletton.format(bmi, category))

bmi_calculator(75, 1.74)
print('\n')
bmi_calculator(49, 1.85)
print('\n')
bmi_calculator(128, 1.71)

bmi_calculator(80,1.73)


I am working on the file "Programming_Lab_I/Programming Lab Sheet 1.ipynb".
The first task is:
Exercise 3: Sieve of Erathostenes
Can you write Python code for it?

def primes(limit):
    '''
    A function implementing the sieve of erathostenes.
    It take an upper limit and returns all prime numbers
    under this limit.
    '''
    status = [True] * limit
    status[0] = status[1] = False
    primes = []

    for (i, isprime) in enumerate(status):
        if isprime:
            primes.append(i)
            for n in range(i*i, limit, i):
                status[n] = False

    return primes

primes(57140)[-2]

max_num = int(input('Calculate prime numbers under: '))
for n in primes(max_num):
    print(n, end = " ")


I am working on the file "Programming_Lab_I/Programming Lab Sheet 4.ipynb".
The first task is:
Exercise 2: Factorize an integer
Can you write Python code for it?

def seaveEra(limit):
    '''
    A function implementing the sieve of erathostenes.
    It takes an upper limit and yields all prime numbers
    under this limit.

    >>> list(seaveEra(7))
    [2, 3, 5]
    '''
    status = [True] * limit
    status[0] = status[1] = False

    for (i, isprime) in enumerate(status):
        if isprime:
            yield i
            for n in range(i*i, limit, i):
                status[n] = False

def factorize(num, primes=None):
    '''
    This function takes an integer and returns a dict
    of prime factors.

    >>> factorize(1)
    Traceback (most recent call last):
     ...
    AssertionError: Input is smaller than 2

    >>> factorize(7)
    {7: 1}
    '''
    import math
    assert type(num) == int, 'Input is not an integer'
    assert num >= 2, 'Input is smaller than 2'

    if primes == None:
        primes = list(seaveEra(int(math.sqrt(num))+2))
    result = {}
    for x in primes:
        if num % x == 0:
            result[x] = 1
            num = num // x
            while num % x == 0:
                result[x] += 1
                num = num // x
    #if result == {}:
    #    raise ValueError(' c')
    if num == 1:
        return result
    if (max(primes)+1)**2 > num:
        result = {num: 1}
        return result
    else:
        mes = ('At leat one prime factor '
               'is bigger than {}')
        return mes.format(max(primes))

if __name__ == '__main__':
    import doctest
    doctest.testmod()

# Test the factorize function. A precomputed list of primes is used.
factorize_tests = [1231, 123259, 12345577, 1234567811, 112233445589,
                   11223344556607, 4, 1001, 198473094, 13918452024,
                   32574985749857]

primes = list(seaveEra(10000000))
for num in sorted(factorize_tests):
    print('{:15}   {}'.format(num, factorize(num, primes)))


I am working on the file "Programming_Lab_I/Programming Lab Sheet 4.ipynb".
The first task is:
Exercise 3: Measure execution time
Can you write Python code for it?

import time
from functools import wraps

def timeit(func):
    '''
    Decorator to time the execution time of a function
    '''
    @wraps(func)
    def timed(*args):
        start = time.perf_counter()
        result = func(*args)
        total = time.perf_counter() - start
        return result, total
    return timed

# Time the factorization function.
factorize_tests2 = [4,1001,1231,123259,198473094]
timed_factorize = timeit(factorize)
for num in factorize_tests2:
    print('{:15}   {}'.format(num, timed_factorize(num)[1]))

def multiply_polynoms(polyA, polyB):
    '''
    This fuction takes two lists as parameters which represent polynoms.
    It returns the product of those two polynomes.
    '''
    result = [0]*(len(polyA)+len(polyB)-1)
    for i in range(len(polyA)):
        for j in range(len(polyB)):
            result[i+j] += polyA[i]*polyB[j]
    return result

#Time the multiply_polynoms function
timed_multiply_polynoms = timeit(multiply_polynoms)
x = []
y = []
for i in range(0,2001, 100):
    poly = i*[1]
    x.append(i)
    calls = []
    for i in range(5):
        calls.append(timed_multiply_polynoms(poly, poly)[1])
    y.append(sum(calls)/len(calls))

# Plot the mean execution time against the polynom length
%matplotlib inline
import matplotlib.pyplot as plt
plt.plot(x,y)
plt.plot(x, [0.0000002*x**2 for x in x],'--')


I am working on the file "Programming_a_seismic_program.ipynb".
The first task is:
<hr />
Dealing with the real world: moving stations
Can you write Python code for it?


I am working on the file "Programming_a_seismic_program.ipynb".
The first task is:
As well as knowing the (x,y) of the midpoints, we'd also like to record the distance from each s to each live r (each r in the live patch). This is easy enough to compute:
Point(x1, y1).distance(Point(x2, y2))

Then we can make a list of all the offsets when we count the midpoints into the bins.
Can you write Python code for it?

offsets = [r.distance(s)
           for r in rcvrs
           for s in srcs]

azimuths = [np.arctan((r.x - s.x)/(r.y - s.y))
            for r in rcvrs
            for s in srcs]


I am working on the file "Programming_a_seismic_program.ipynb".
The first task is:
Calculate the distance each station has moved. We have to use GeoSeries to do this, and everything was transformed to ordinary Series when we did the join.
Can you write Python code for it?

old = gpd.GeoSeries(complete.geometry)
new = gpd.GeoSeries(complete['new_geometry'])

complete['skid'] = old.distance(new)

complete[15:20]


I am working on the file "Programming_a_seismic_program.ipynb".
The first task is:
Define some survey parameters:
Can you write Python code for it?

xmi = 575000        # Easting of bottom-left corner of grid (m)
ymi = 4710000       # Northing of bottom-left corner (m)
SL = 600            # Source line interval (m)
RL = 600            # Receiver line interval (m)
si = 100            # Source point interval (m)
ri = 100            # Receiver point interval (m)
x = 3000            # x extent of survey (m)
y = 1800            # y extent of survey (m)

epsg = 26911  # A CRS to place it on the earth


I am working on the file "Programming_a_seismic_program.ipynb".
The first task is:
Make a Geoseries of the midpoints, offsets and azimths:
Can you write Python code for it?

midpoints = gpd.GeoDataFrame({'geometry': midpoint_list,
                              'offset': offsets,
                              'azimuth': np.degrees(azimuths),
                              })

midpoints[:5]

ax = midpoints.plot(color='b')


I am working on the file "Programming_a_seismic_program.ipynb".
The first task is:
Make a Station ID row, so we can recognize the stations again.
Can you write Python code for it?

sid = np.arange(len(survey))
survey['SID'] = sid

survey.to_file('data/survey_orig.shp')


I am working on the file "Programming_a_seismic_program.ipynb".
The first task is:
Now that we have the new geometry, we can recompute everything, this time from the dataframe rather than from lists.
First the midpoints.
Can you write Python code for it?

rcvrs = complete[complete['station']=='r'][complete['new_geometry'].notnull()]['new_geometry']
srcs = complete[complete['station']!='r'][complete['new_geometry'].notnull()]['new_geometry']

midpoint_list = [LineString([r, s]).interpolate(0.5, normalized=True)
                 for r in rcvrs
                 for s in srcs]

len(srcs)

offsets = [r.distance(s)
           for r in rcvrs
           for s in srcs]

azimuths = [np.arctan((r.x - s.x)/(r.y - s.y))
            for r in rcvrs
            for s in srcs]

new_midpoints = gpd.GeoDataFrame({'geometry': midpoint_list,
                                  'offset': offsets,
                                  'azimuth': np.degrees(azimuths),
                                  })