I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Quiz question: Which tree has the largest complexity?
Quiz question: Is it always true that the most complex tree will result in the lowest classification error in the validation_set?
Exploring the effect of min_error
We will compare three models trained with different values of the stopping criterion. We intentionally picked models at the extreme ends (negative, just right, and too positive).
Train three models with these parameters:
1. model_4: min_error_reduction = -1 (ignoring this early stopping condition)
2. model_5: min_error_reduction = 0 (just right)
3. model_6: min_error_reduction = 5 (too positive)
For each of these three, we set max_depth = 6, and min_node_size = 0.
 Note: Each tree can take up to 30 seconds to train.
Can you write Python code for it?

model_4 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                               min_node_size = 0, min_error_reduction=-1)
model_5 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                               min_node_size = 0, min_error_reduction=0)
model_6 = decision_tree_create(train_data, features, 'safe_loans', max_depth = 6,
                               min_node_size = 0, min_error_reduction=5)


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Run the following test code to check your implementation. Make sure you get 'Test passed' before proceeding.
Can you write Python code for it?

small_decision_tree = decision_tree_create(train_data, features, 'safe_loans', max_depth = 2,
                                        min_node_size = 10, min_error_reduction=0.0)
if count_nodes(small_decision_tree) == 7:
    print 'Test passed!'
else:
    print 'Test failed... try again!'
    print 'Number of nodes found                :', count_nodes(small_decision_tree)
    print 'Number of nodes that should be there : 7'


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Subsample dataset to make sure classes are balanced
Just as we did in the previous assignment, we will undersample the larger class (safe loans) in order to balance out our dataset. This means we are throwing away many data points. We used seed = 1 so everyone gets the same results.
Can you write Python code for it?

safe_loans_raw = loans[loans[target] == 1]
risky_loans_raw = loans[loans[target] == -1]

# Since there are less risky loans than safe loans, find the ratio of the sizes
# and use that percentage to undersample the safe loans.
percentage = len(risky_loans_raw)/float(len(safe_loans_raw))
safe_loans = safe_loans_raw.sample(percentage, seed = 1)
risky_loans = risky_loans_raw
loans_data = risky_loans.append(safe_loans)

print "Percentage of safe loans                 :", len(safe_loans) / float(len(loans_data))
print "Percentage of risky loans                :", len(risky_loans) / float(len(loans_data))
print "Total number of loans in our new dataset :", len(loans_data)


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
The feature columns now look like this:
Can you write Python code for it?

features = loans_data.column_names()
features.remove('safe_loans')  # Remove the response variable
features


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Train-Validation split
We split the data into a train-validation split with 80% of the data in the training set and 20% of the data in the validation set. We use seed=1 so that everyone gets the same result.
Can you write Python code for it?

train_data, validation_set = loans_data.random_split(.8, seed=1)


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Using the count_leaves function, compute the number of leaves in each of each models (model_7, model_8, and model_9).
Can you write Python code for it?

print count_leaves(model_7)
print count_leaves(model_8)
print count_leaves(model_9)


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
Using the count_leaves function, compute the number of leaves in each of each models in (model_4, model_5, and model_6).
Can you write Python code for it?

print count_leaves(model_4)
print count_leaves(model_5)
print count_leaves(model_6)


I am working on the file "Classifier_4_decision_tree_combat_overfitting_assignment.ipynb".
The first task is:
We then wrote a function best_splitting_feature that finds the best feature to split on given the data and a list of features to consider.
Please copy and paste your best_splitting_feature code here.
Can you write Python code for it?

def best_splitting_feature(data, features, target):

    best_feature = None # Keep track of the best feature
    best_error = 10     # Keep track of the best error so far
    # Note: Since error is always <= 1, we should intialize it with something larger than 1.

    # Convert to float to make sure error gets computed correctly.
    num_data_points = float(len(data))

    # Loop through each feature to consider splitting on that feature
    for feature in features:

        # The left split will have all data points where the feature value is 0
        left_split = data[data[feature] == 0]

        # The right split will have all data points where the feature value is 1
        ## YOUR CODE HERE
        right_split = data[data[feature] == 1]

        # Calculate the number of misclassified examples in the left split.
        # Remember that we implemented a function for this! (It was called intermediate_node_num_mistakes)
        # YOUR CODE HERE
        left_mistakes = intermediate_node_num_mistakes(left_split[target])

        # Calculate the number of misclassified examples in the right split.
        ## YOUR CODE HERE
        right_mistakes = intermediate_node_num_mistakes(right_split[target])

        # Compute the classification error of this split.
        # Error = (# of mistakes (left) + # of mistakes (right)) / (# of data points)
        ## YOUR CODE HERE
        error = (left_mistakes + right_mistakes) / num_data_points

        # If this is the best error we have found so far, store the feature as best_feature and the error as best_error
        ## YOUR CODE HERE
        if error < best_error:
            best_feature = feature
            best_error = error

    return best_feature # Return the best feature we found


I am working on the file "Classifier_5_boosting_assignment_1.ipynb".
The first task is:
Calculate the number of false positives made by the model.
Can you write Python code for it?


I am working on the file "Classifier_5_boosting_assignment_1.ipynb".
The first task is:
Checkpoint: For each row, the probabilities should be a number in the range [0, 1]. We have provided a simple check here to make sure your answers are correct.
Can you write Python code for it?


I am working on the file "Classifier_5_boosting_assignment_1.ipynb".
The first task is:
Checkpoint: You should expect to see 5 loans with the grade ['D', 'C', 'C', 'C', 'B'].
Effect of adding more trees
In this assignment, we will train 5 different ensemble classifiers in the form of gradient boosted trees. We will train models with 10, 50, 100, 200, and 500 trees.  We use the max_iterations parameter in the boosted tree module.
Let's get sarted with a model with max_iterations = 10:
Can you write Python code for it?

model_10 = graphlab.boosted_trees_classifier.create(train_data, validation_set=None,
        target = target, features = features, max_iterations = 10, verbose=False)


I am working on the file "Classifier_5_boosting_assignment_1.ipynb".
The first task is:
Checkpoint: You should now see that the dataset is balanced (approximately 50-50 safe vs risky loans).
Note: There are many approaches for dealing with imbalanced data, including some where we modify the learning algorithm. These approaches are beyond the scope of this course, but some of them are reviewed in this paper. For this assignment, we use the simplest possible approach, where we subsample the overly represented class to get a more balanced dataset. In general, and especially when the data is highly imbalanced, we recommend using more advanced methods.
Split data into training and validation sets
We split the data into training data and validation data. We used seed=1 to make sure everyone gets the same results. We will use the validation data to help us select model parameters.
Can you write Python code for it?

train_data, validation_data = loans_data.random_split(.8, seed=1)


I am working on the file "Classifier_5_boosting_assignment_1.ipynb".
The first task is:
Fortunately, there are not too many missing values. We are retaining most of the data.
Make sure the classes are balanced
We saw in an earlier assignment that this dataset is also imbalanced. We will undersample the larger class (safe loans) in order to balance out our dataset. We used seed=1 to make sure everyone gets the same results.
Can you write Python code for it?

safe_loans_raw = loans[loans[target] == 1]
risky_loans_raw = loans[loans[target] == -1]

# Undersample the safe loans.
percentage = len(risky_loans_raw)/float(len(safe_loans_raw))
print "Ratio of risky loans to safe loans: ", percentage
safe_loans = safe_loans_raw.sample(percentage, seed = 1)
risky_loans = risky_loans_raw
loans_data = risky_loans.append(safe_loans)

print "Percentage of safe loans                 :", len(safe_loans) / float(len(loans_data))
print "Percentage of risky loans                :", len(risky_loans) / float(len(loans_data))
print "Total number of loans in our new dataset :", len(loans_data)


I am working on the file "Classifier_5_boosting_assignment_1.ipynb".
The first task is:
Let's quickly explore what the dataset looks like. First, let's print out the column names to see what features we have in this dataset. We have done this in previous assignments, so we won't belabor this here.
Can you write Python code for it?


I am working on the file "Classifier_5_boosting_assignment_1.ipynb".
The first task is:
Making predictions
Just like we did in previous sections, let us consider a few positive and negative examples from the validation set. We will do the following:
* Predict whether or not a loan is likely to default.
* Predict the probability with which the loan is likely to default.
Can you write Python code for it?

# Select all positive and negative examples.
validation_safe_loans = validation_data[validation_data[target] == 1]
validation_risky_loans = validation_data[validation_data[target] == -1]

# Select 2 examples from the validation set for positive & negative loans
sample_validation_data_risky = validation_risky_loans[0:2]
sample_validation_data_safe = validation_safe_loans[0:2]

# Append the 4 examples into a single dataset
sample_validation_data = sample_validation_data_safe.append(sample_validation_data_risky)
sample_validation_data


I am working on the file "Classifier_5_boosting_assignment_1.ipynb".
The first task is:
Now, let us run Step 2. Save the training errors into a list called training_errors
Can you write Python code for it?

training_errors = [1 - model.evaluate(train_data)['accuracy'] for model in models]


I am working on the file "Classifier_5_boosting_assignment_1.ipynb".
The first task is:
Now, train 4 models with max_iterations to be:
* max_iterations = 50,
* max_iterations = 100
* max_iterations = 200
* max_iterations = 500.
Let us call these models model_50, model_100, model_200, and model_500. You can pass in verbose=False in order to suppress the printed output.
Warning: This could take a couple of minutes to run.
Can you write Python code for it?

model_50 = graphlab.boosted_trees_classifier.create(train_data, validation_set=None,
                                                    target = target, features = features, max_iterations = 50, verbose=False)
model_100 = graphlab.boosted_trees_classifier.create(train_data, validation_set=None,
                                                     target = target, features = features, max_iterations = 100, verbose=False)
model_200 = graphlab.boosted_trees_classifier.create(train_data, validation_set=None,
                                                     target = target, features = features, max_iterations = 200, verbose=False)
model_500 = graphlab.boosted_trees_classifier.create(train_data, validation_set=None,
                                                     target = target, features = features, max_iterations = 500, verbose=False)


I am working on the file "Classifier_5_boosting_assignment_1.ipynb".
The first task is:
Now, we are ready to go to Step 3. You can now use the prediction column to sort the loans in validation_data (in descending order) by prediction probability. Find the top 5 loans with the highest probability of being predicted as a safe loan.
Can you write Python code for it?


I am working on the file "Classifier_5_boosting_assignment_1.ipynb".
The first task is:
Now, we will plot the training_errors and validation_errors versus the number of trees. We will compare the 10, 50, 100, 200, and 500 tree models. We provide some plotting code to visualize the plots within this notebook.
Run the following code to visualize the plots.
Can you write Python code for it?

plt.plot([10, 50, 100, 200, 500], training_errors, linewidth=4.0, label='Training error')
plt.plot([10, 50, 100, 200, 500], validation_errors, linewidth=4.0, label='Validation error')

make_figure(dim=(10,5), title='Error vs number of trees',
            xlabel='Number of trees',
            ylabel='Classification error',
            legend='best')


I am working on the file "Classifier_5_boosting_assignment_1.ipynb".
The first task is:
Predicting on sample validation data
For each row in the sample_validation_data, write code to make model_5 predict whether or not the loan is classified as a safe loan.
Hint: Use the predict method in model_5 for this.
Can you write Python code for it?


I am working on the file "Classifier_5_boosting_assignment_1.ipynb".
The first task is:
Quiz Question: According to model_5, which loan is the least likely to be a safe loan?
Checkpoint: Can you verify that for all the predictions with probability >= 0.5, the model predicted the label +1?
Evaluating the model on the validation data
Recall that the accuracy is defined as follows:
$$
\mbox{accuracy} = \frac{\mbox{# correctly classified examples}}{\mbox{# total examples}}
$$
Evaluate the accuracy of the model_5 on the validation_data.
Hint: Use the .evaluate() method in the model.
Can you write Python code for it?


I am working on the file "Classifier_5_boosting_assignment_1.ipynb".
The first task is:
Quiz question: What grades are the top 5 loans?
Let us repeat this excercise to find the top 5 loans (in the validation_data) with the lowest probability of being predicted as a safe loan:
Can you write Python code for it?


I am working on the file "Classifier_5_boosting_assignment_1.ipynb".
The first task is:
Quiz question: What is the number of false positives on the validation_data?
Calculate the number of false negatives made by the model.
Can you write Python code for it?

false_positives = 1618

false_negatives = 1463


I am working on the file "Classifier_5_boosting_assignment_1.ipynb".
The first task is:
Reminder: Compare the cost of the mistakes made by the boosted trees model with the decision tree model. The extra 3% improvement in prediction accuracy can translate to several million dollars!  And, it was so easy to get by simply boosting our decision trees.
Most positive & negative loans.
In this section, we will find the loans that are most likely to be predicted safe. We can do this in a few steps:

Step 1: Use the model_5 (the model with 5 trees) and make probability predictions for all the loans in the validation_data.
Step 2: Similar to what we did in the very first assignment, add the probability predictions as a column called predictions into the validation_data.
Step 3: Sort the data (in descreasing order) by the probability predictions.

Start here with Step 1 & Step 2. Make predictions using model_5 for examples in the validation_data. Use output_type = probability.
Can you write Python code for it?

predictions = model_5.predict(validation_data, output_type='probability')

validation_data['predictions'] = predictions


I am working on the file "Classifier_5_boosting_assignment_1.ipynb".
The first task is:
Selecting features
In this assignment, we will be using a subset of features (categorical and numeric). The features we will be using are described in the code comments below. If you are a finance geek, the LendingClub website has a lot more details about these features.
The features we will be using are described in the code comments below:
Can you write Python code for it?

target = 'safe_loans'
features = ['grade',                     # grade of the loan (categorical)
            'sub_grade_num',             # sub-grade of the loan as a number from 0 to 1
            'short_emp',                 # one year or less of employment
            'emp_length_num',            # number of years of employment
            'home_ownership',            # home_ownership status: own, mortgage or rent
            'dti',                       # debt to income ratio
            'purpose',                   # the purpose of the loan
            'payment_inc_ratio',         # ratio of the monthly payment to income
            'delinq_2yrs',               # number of delinquincies
            'delinq_2yrs_zero',          # no delinquincies in last 2 years
            'inq_last_6mths',            # number of creditor inquiries in last 6 months
            'last_delinq_none',          # has borrower had a delinquincy
            'last_major_derog_none',     # has borrower had 90 day or worse rating
            'open_acc',                  # number of open credit accounts
            'pub_rec',                   # number of derogatory public records
            'pub_rec_zero',              # no derogatory public records
            'revol_util',                # percent of available credit being used
            'total_rec_late_fee',        # total late fees received to day
            'int_rate',                  # interest rate of the loan
            'total_rec_int',             # interest received to date
            'annual_inc',                # annual income of borrower
            'funded_amnt',               # amount committed to the loan
            'funded_amnt_inv',           # amount committed by investors for the loan
            'installment',               # monthly payment owed by the borrower
           ]


I am working on the file "Classifier_5_boosting_assignment_1.ipynb".
The first task is:
Skipping observations with missing values
Recall from the lectures that one common approach to coping with missing values is to skip observations that contain missing values.
We run the following code to do so:
Can you write Python code for it?

loans, loans_with_na = loans[[target] + features].dropna_split()

# Count the number of rows with missing data
num_rows_with_na = loans_with_na.num_rows()
num_rows = loans.num_rows()
print 'Dropping %s observations; keeping %s ' % (num_rows_with_na, num_rows)


I am working on the file "Classifier_5_boosting_assignment_2.ipynb".
The first task is:
Checking your Adaboost code
Train an ensemble of two tree stumps and see which features those stumps split on. We will run the algorithm with the following parameters:
* train_data
* features
* target
* num_tree_stumps = 2
Can you write Python code for it?

stump_weights, tree_stumps = adaboost_with_tree_stumps(train_data, features, target, num_tree_stumps=2)

def print_stump(tree):
    split_name = tree['splitting_feature'] # split_name is something like 'term. 36 months'
    if split_name is None:
        print "(leaf, label: %s)" % tree['prediction']
        return None
    split_feature, split_value = split_name.split('.')
    print '                       root'
    print '         |---------------|----------------|'
    print '         |                                |'
    print '         |                                |'
    print '         |                                |'
    print '  [{0} == 0]{1}[{0} == 1]    '.format(split_name, ' '*(27-len(split_name)))
    print '         |                                |'
    print '         |                                |'
    print '         |                                |'
    print '    (%s)                 (%s)' \
        % (('leaf, label: ' + str(tree['left']['prediction']) if tree['left']['is_leaf'] else 'subtree'),
           ('leaf, label: ' + str(tree['right']['prediction']) if tree['right']['is_leaf'] else 'subtree'))


I am working on the file "Classifier_5_boosting_assignment_2.ipynb".
The first task is:
Checkpoint: Test your intermediate_node_weighted_mistakes function, run the following cell:
Can you write Python code for it?

example_labels = graphlab.SArray([-1, -1, 1, 1, 1])
example_data_weights = graphlab.SArray([1., 2., .5, 1., 1.])
if intermediate_node_weighted_mistakes(example_labels, example_data_weights) == (2.5, -1):
    print 'Test passed!'
else:
    print 'Test failed... try again!'


I am working on the file "Classifier_5_boosting_assignment_2.ipynb".
The first task is:
Computing training error at the end of each iteration
Now, we will compute the classification error on the train_data and see how it is reduced as trees are added.
Can you write Python code for it?

error_all = []
for n in xrange(1, 31):
    predictions = predict_adaboost(stump_weights[:n], tree_stumps[:n], train_data)
    error = 1.0 - graphlab.evaluation.accuracy(train_data[target], predictions)
    error_all.append(error)
    print "Iteration %s, training error = %s" % (n, error_all[n-1])


I am working on the file "Classifier_5_boosting_assignment_2.ipynb".
The first task is:
Evaluating the tree
Now, we will write a function to evaluate a decision tree by computing the classification error of the tree on the given dataset.
Again, recall that the classification error is defined as follows:
$$
\mbox{classification error} = \frac{\mbox{# mistakes}}{\mbox{# all data points}}
$$
The function called evaluate_classification_error takes in as input:
1. tree (as described above)
2. data (an SFrame)
The function does not change because of adding data point weights.
Can you write Python code for it?

def evaluate_classification_error(tree, data):
    # Apply the classify(tree, x) to each row in your data
    prediction = data.apply(lambda x: classify(tree, x))

    # Once you've made the predictions, calculate the classification error
    return (prediction != data[target]).sum() / float(len(data))

evaluate_classification_error(small_data_decision_tree, test_data)


I am working on the file "Classifier_5_boosting_assignment_2.ipynb".
The first task is:
Example: Training a weighted decision tree
To build intuition on how weighted data points affect the tree being built, consider the following:
Suppose we only care about making good predictions for the first 10 and last 10 items in train_data, we assign weights:
* 1 to the last 10 items
* 1 to the first 10 items
* and 0 to the rest.
Let us fit a weighted decision tree with max_depth = 2.
Can you write Python code for it?

# Assign weights
example_data_weights = graphlab.SArray([1.] * 10 + [0.]*(len(train_data) - 20) + [1.] * 10)

# Train a weighted decision tree model.
small_data_decision_tree_subset_20 = weighted_decision_tree_create(train_data, features, target,
                         example_data_weights, max_depth=2)


I am working on the file "Classifier_5_boosting_assignment_2.ipynb".
The first task is:
Extracting the target and the feature columns
We will now repeat some of the feature processing steps that we saw in the previous assignment:
First, we re-assign the target to have +1 as a safe (good) loan, and -1 as a risky (bad) loan.
Next, we select four categorical features:
1. grade of the loan
2. the length of the loan term
3. the home ownership status: own, mortgage, rent
4. number of years of employment.
Can you write Python code for it?

features = ['grade',              # grade of the loan
            'term',               # the term of the loan
            'home_ownership',     # home ownership status: own, mortgage or rent
            'emp_length',         # number of years of employment
           ]
loans['safe_loans'] = loans['bad_loans'].apply(lambda x : +1 if x==0 else -1)
loans.remove_column('bad_loans')
target = 'safe_loans'
loans = loans[features + [target]]


I am working on the file "Classifier_5_boosting_assignment_2.ipynb".
The first task is:
Here is a recursive function to count the nodes in your tree:
Can you write Python code for it?

def count_nodes(tree):
    if tree['is_leaf']:
        return 1
    return 1 + count_nodes(tree['left']) + count_nodes(tree['right'])


I am working on the file "Classifier_5_boosting_assignment_2.ipynb".
The first task is:
Here is what the first stump looks like:
Can you write Python code for it?


I am working on the file "Classifier_5_boosting_assignment_2.ipynb".
The first task is:
Let's see what the feature columns look like now:
Can you write Python code for it?

features = loans_data.column_names()
features.remove('safe_loans')  # Remove the response variable
features


I am working on the file "Classifier_5_boosting_assignment_2.ipynb".
The first task is:
Making predictions
Recall from the lecture that in order to make predictions, we use the following formula:
$$
\hat{y} = sign\left(\sum_{t=1}^T \hat{w}_t f_t(x)\right)
$$
We need to do the following things:
- Compute the predictions $f_t(x)$ using the $t$-th decision tree
- Compute $\hat{w}_t f_t(x)$ by multiplying the stump_weights with the predictions $f_t(x)$ from the decision trees
- Sum the weighted predictions over each stump in the ensemble.
Complete the following skeleton for making predictions:
Can you write Python code for it?

def predict_adaboost(stump_weights, tree_stumps, data):
    scores = graphlab.SArray([0.]*len(data))

    for i, tree_stump in enumerate(tree_stumps):
        predictions = data.apply(lambda x: classify(tree_stump, x))

        # Accumulate predictions on scores array
        # YOUR CODE HERE
        scores = scores + predictions * stump_weights[i]

    return scores.apply(lambda score : +1 if score > 0 else -1)

predictions = predict_adaboost(stump_weights, tree_stumps, test_data)
accuracy = graphlab.evaluation.accuracy(test_data[target], predictions)
print 'Accuracy of 10-component ensemble = %s' % accuracy


I am working on the file "Classifier_5_boosting_assignment_2.ipynb".
The first task is:
Note: There are many approaches for dealing with imbalanced data, including some where we modify the learning algorithm. These approaches are beyond the scope of this course, but some of them are reviewed in this paper. For this assignment, we use the simplest possible approach, where we subsample the overly represented class to get a more balanced dataset. In general, and especially when the data is highly imbalanced, we recommend using more advanced methods.
Transform categorical data into binary features
In this assignment, we will work with binary decision trees. Since all of our features are currently categorical features, we want to turn them into binary features using 1-hot encoding.
We can do so with the following code block (see the first assignments for more details):
Can you write Python code for it?

loans_data = risky_loans.append(safe_loans)
for feature in features:
    loans_data_one_hot_encoded = loans_data[feature].apply(lambda x: {x: 1})
    loans_data_unpacked = loans_data_one_hot_encoded.unpack(column_name_prefix=feature)

    # Change None's to 0's
    for column in loans_data_unpacked.column_names():
        loans_data_unpacked[column] = loans_data_unpacked[column].fillna(0)

    loans_data.remove_column(feature)
    loans_data.add_columns(loans_data_unpacked)


I am working on the file "Classifier_5_boosting_assignment_2.ipynb".
The first task is:
Now, let us compare the classification error of the model small_data_decision_tree_subset_20 on the entire test set train_data:
Can you write Python code for it?


I am working on the file "Classifier_5_boosting_assignment_2.ipynb".
The first task is:
Now, let us take a quick look what the stump_weights look like at the end of each iteration of the 10-stump ensemble:
Can you write Python code for it?


I am working on the file "Classifier_5_boosting_assignment_2.ipynb".
The first task is:
Quiz Question: Are the weights monotonically decreasing, monotonically increasing, or neither?
Reminder: Stump weights ($\mathbf{\hat{w}}$) tell you how important each stump is while making predictions with the entire boosted ensemble.
Performance plots
In this section, we will try to reproduce some of the performance plots dicussed in the lecture.
How does accuracy change with adding stumps to the ensemble?
We will now train an ensemble with:
* train_data
* features
* target
* num_tree_stumps = 30
Once we are done with this, we will then do the following:
* Compute the classification error at the end of each iteration.
* Plot a curve of classification error vs iteration.
First, lets train the model.
Can you write Python code for it?

# this may take a while...
stump_weights, tree_stumps = adaboost_with_tree_stumps(train_data,
                                 features, target, num_tree_stumps=30)


I am working on the file "Classifier_5_boosting_assignment_2.ipynb".
The first task is:
Quiz Question: Which of the following best describes a general trend in accuracy as we add more and more components? Answer based on the 30 components learned so far.

Training error goes down monotonically, i.e. the training error reduces with each iteration but never increases.
Training error goes down in general, with some ups and downs in the middle.
Training error goes up in general, with some ups and downs in the middle.
Training error goes down in the beginning, achieves the best error, and then goes up sharply.
None of the above

Evaluation on the test data
Performing well on the training data is cheating, so lets make sure it works on the test_data as well. Here, we will compute the classification error on the test_data at the end of each iteration.
Can you write Python code for it?

test_error_all = []
for n in xrange(1, 31):
    predictions = predict_adaboost(stump_weights[:n], tree_stumps[:n], test_data)
    error = 1.0 - graphlab.evaluation.accuracy(test_data[target], predictions)
    test_error_all.append(error)
    print "Iteration %s, test error = %s" % (n, test_error_all[n-1])


I am working on the file "Classifier_5_boosting_assignment_2.ipynb".
The first task is:
Recall that the classification error is defined as follows:
$$
\mbox{classification error} = \frac{\mbox{# mistakes}}{\mbox{# all data points}}
$$
Quiz Question: If we set the weights $\mathbf{\alpha} = 1$ for all data points, how is the weight of mistakes $\mbox{WM}(\mathbf{\alpha}, \mathbf{\hat{y}})$ related to the classification error?
Function to pick best feature to split on
We continue modifying our decision tree code from the earlier assignment to incorporate weighting of individual data points. The next step is to pick the best feature to split on.
The best_splitting_feature function is similar to the one from the earlier assignment with two minor modifications:
  1. The function best_splitting_feature should now accept an extra parameter data_weights to take account of weights of data points.
  2. Instead of computing the number of mistakes in the left and right side of the split, we compute the weight of mistakes for both sides, add up the two weights, and divide it by the total weight of the data.
Complete the following function. Comments starting with DIFFERENT HERE mark the sections where the weighted version differs from the original implementation.
Can you write Python code for it?

def best_splitting_feature(data, features, target, data_weights):

    # These variables will keep track of the best feature and the corresponding error
    best_feature = None
    best_error = float('+inf')
    num_points = float(len(data))
    total_weights = sum(data_weights)

    # Loop through each feature to consider splitting on that feature
    for feature in features:

        # The left split will have all data points where the feature value is 0
        # The right split will have all data points where the feature value is 1
        left_split = data[data[feature] == 0]
        right_split = data[data[feature] == 1]

        # Apply the same filtering to data_weights to create left_data_weights, right_data_weights
        ## YOUR CODE HERE
        left_data_weights = data_weights[data[feature] == 0]
        right_data_weights = data_weights[data[feature] == 1]

        # DIFFERENT HERE
        # Calculate the weight of mistakes for left and right sides
        ## YOUR CODE HERE
        left_weighted_mistakes, left_class = intermediate_node_weighted_mistakes(left_split[target], left_data_weights)
        right_weighted_mistakes, right_class = intermediate_node_weighted_mistakes(right_split[target], right_data_weights)

        # DIFFERENT HERE
        # Compute weighted error by computing
        #  ( [weight of mistakes (left)] + [weight of mistakes (right)] ) / [total weight of all data points]
        ## YOUR CODE HERE
        error = (left_weighted_mistakes + right_weighted_mistakes) / total_weights

        # If this is the best error we have found so far, store the feature and the error
        if error < best_error:
            best_feature = feature
            best_error = error

    # Return the best feature we found
    return best_feature


I am working on the file "Classifier_5_boosting_assignment_2.ipynb".
The first task is:
Run the following test code to check your implementation. Make sure you get 'Test passed' before proceeding.
Can you write Python code for it?

example_data_weights = graphlab.SArray([1.0 for i in range(len(train_data))])
small_data_decision_tree = weighted_decision_tree_create(train_data, features, target,
                                        example_data_weights, max_depth=2)
if count_nodes(small_data_decision_tree) == 7:
    print 'Test passed!'
else:
    print 'Test failed... try again!'
    print 'Number of nodes found:', count_nodes(small_data_decision_tree)
    print 'Number of nodes that should be there: 7'


I am working on the file "Classifier_5_boosting_assignment_2.ipynb".
The first task is:
Visualize both the training and test errors
Now, let us plot the training & test error with the number of iterations.
Can you write Python code for it?

plt.rcParams['figure.figsize'] = 7, 5
plt.plot(range(1,31), error_all, '-', linewidth=4.0, label='Training error')
plt.plot(range(1,31), test_error_all, '-', linewidth=4.0, label='Test error')

plt.title('Performance of Adaboost ensemble')
plt.xlabel('# of iterations')
plt.ylabel('Classification error')
plt.rcParams.update({'font.size': 16})
plt.legend(loc='best', prop={'size':15})
plt.tight_layout()


I am working on the file "Classifying challenge data.ipynb".
The first task is:
Using APPLY to classify observation types according to keywords or strings in certain fields
Can you write Python code for it?

df = df.replace(np.nan,' ', regex=True)#MUST REPLACE NAN AS NAN IS NOT ITERABLE
#analysis_terms = ['analysis', 'density', 'average','variance','pH']
def mymunge(row): # a VARIETY OF CRITERIA
    if any( ['along' in row['note'], 'transect' in row['type'], 'grade' in row['type'], 'line' in row['type'], 'map' in row['type'], 'plot' in row['type'], 'remonitor' in row['type'], 'plants' in row['type']] ):
        return 'transect'
    if any( ['vert' in row['type'], 'ang' in row['type'], 'step back' in row['note'], 'hoop' in row['note'], 'Hoop' in row['note'], 'bare' in row['label1']] ):
        return 'cover'
    if any( ['infiltracion' in row['type'], 'infiltration' in row['type']] ):
        return 'infiltration'
    if any( ['analysis' in row['type'], 'composited' in row['type'], 'misc' in row['type'], 'density' in row['type'],'variance' in row['type'], 'average' in row['type'], 'pH' in row['type']] ):
        return 'analysis'
    if any( ['photo' in row['type']] ):
        return 'photo'
    if any( ['change' in row['type']] ):
        return 'change'
    if any( ['brix' in row['type'], 'biomass' in row['type'], 'clip' in row['type']] ):
        return 'food_analysis'

df['obs_type'] = df.apply(mymunge, axis=1)
#SEE HOW WE DID
print(len(df))
print(df.obs_type.value_counts(normalize=False, sort=True, ascending=False, bins=None, dropna=False))

df = df.replace(np.nan,'', regex=True)#MUST REPLACE NAN AS NAN IS NOT ITERABLE

def buildValues(row): # build the values json field
    v = '{"observer": "' + row['observer'] \
    + '", "label1": "' + row['label1'] \
    + '", "value1": "' + row['value1'] \
    + '", "label2": "' + row['label2'] \
    + '", "value2": "' + row['value2'] \
    + '", "label3": "' + row['label3'] \
    + '", "value3": "' + row['value3'] \
    + '", "start_date": "' + row['start_date'] \
    + '", "end_date": "' + row['end_date'] \
    + '", "description": "' + row['description'] \
    + '", "photo1": "' + row['photo1'] \
    + '", "photo2": "' + row['photo2'] \
    + '"}'
   ...


I am working on the file "Classifying challenge data.ipynb".
The first task is:
http://www.gregreda.com/2013/10/26/working-with-pandas-dataframes/
Can you write Python code for it?

food_analysis = df[(df['obs_type'] == "food_analysis")]
food_analysis.to_csv('/Users/Peter/Documents/scc/challenge/obs_types/food_analysis.csv', index=False)


I am working on the file "Classifying_datasets/Convolutional_Neural_Networks/scikit_tutorial.ipynb".
The first task is:
Naive Bayes Using Scikit_Lerarn
Can you write Python code for it?

from sklearn.datasets import fetch_20newsgroups

news = fetch_20newsgroups(subset='all')

print type(news.data), type(news.target), type(news.target_names)

print news.target_names

len(news.data)

len(news.target)

news.data[0] #Content of the data at 0th index

news.target[0], news.target_names[news.target[0]] # Target_Name


I am working on the file "Classifying_datasets/Convolutional_Neural_Networks/scikit_tutorial.ipynb".
The first task is:
Pre-Processing The Data
machine learning algorithms can work only on numeric data, so our next step will be to convert our text-based dataset to a numeric dataset
Can you write Python code for it?

SPLIT_PERC = .75
split_size = int(len(news.data)*SPLIT_PERC)
X_train = news.data[:split_size]
X_test = news.data[split_size:]
y_train = news.target[:split_size]
y_test = news.target[split_size:]

from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer, HashingVectorizer, CountVectorizer

clf_1 = Pipeline([('vect', CountVectorizer()), ('clf', MultinomialNB())])

clf_2 = Pipeline([('vect', HashingVectorizer(non_negative=True)), ('clf', MultinomialNB())])

clf_3 = Pipeline([('vect', TfidfVectorizer()), ('clf', MultinomialNB())])

from sklearn.cross_validation import cross_val_score, KFold
from scipy.stats import sem

clfs = [clf_1, clf_2, clf_3]
for clf in clfs:
    print clf
    evaluate_cross_validation(clf, news.data, news.target, 5)


I am working on the file "Classifying_datasets/MNIST/mnist-with-keras-for-beginners-99457.ipynb".
The first task is:
Activations Look Like What?
It looks like diversity of the similar patterns present on multiple classes effect the performance of the classifier although CNN is a robust architechture.
Can you write Python code for it?

test_im = X_train[154]
plt.imshow(test_im.reshape(28,28), cmap='viridis', interpolation='none')


I am working on the file "Classifying_datasets/MNIST/mnist-with-keras-for-beginners-99457.ipynb".
The first task is:
Basic Simple Plot And Evaluation
Can you write Python code for it?

final_loss, final_acc = model.evaluate(X_val, Y_val, verbose=0)
print("Final loss: {0:.6f}, final accuracy: {1:.6f}".format(final_loss, final_acc))

# Look at confusion matrix
#Note, this code is taken straight from the SKLEARN website, an nice way of viewing confusion matrix.
def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]

    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, cm[i, j],
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')

# Predict the values from the validation dataset
Y_pred = model.predict(X_val)
# Convert predictions classes to one hot vectors
Y_pred_classes = np.argmax(Y_pred, axis = 1)
# Convert validation observations to one hot vectors
Y_true = np.argmax(Y_val, axis = 1)
# compute the confusion matrix
confusion_mtx = confusion_matrix(Y_true, Y_pred_classes)
# plot the confusion matrix
plot_confusion_matrix(confusion_mtx, classes = range(10))

print(h.history.keys())
accuracy = h.history['acc']
val_accuracy = h.history['val_acc']
loss = h.history['loss']
val_loss = h.history['val_loss']
epochs = range(len(accuracy))
plt.plot(epochs, accuracy, 'bo', label='Training accuracy')
plt.plot(epochs, val_accuracy, 'b', label='Validation accuracy')
plt.title('Training and validation...


I am working on the file "Classifying_datasets/MNIST/mnist-with-keras-for-beginners-99457.ipynb".
The first task is:
Classifcation Report
Can you write Python code for it?

#get the predictions for the test data
predicted_classes = model.predict_classes(X_test)

#get the indices to be plotted
y_true = test.iloc[:, 0]
correct = np.nonzero(predicted_classes==y_true)[0]
incorrect = np.nonzero(predicted_classes!=y_true)[0]

from sklearn.metrics import classification_report
target_names = ["Class {}".format(i) for i in range(num_classes)]
print(classification_report(y_true, predicted_classes, target_names=target_names))

submissions=pd.DataFrame({"ImageId": list(range(1,len(predicted_classes)+1)),
                         "Label": predicted_classes})
submissions.to_csv("asd.csv", index=False, header=True)

model.save('my_model_1.h5')
json_string = model.to_json()


I am working on the file "Classifying_datasets/MNIST/mnist-with-keras-for-beginners-99457.ipynb".
The first task is:
Linear Model
Can you write Python code for it?

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),activation='relu',kernel_initializer='he_normal',input_shape=input_shape))
model.add(Conv2D(32, kernel_size=(3, 3),activation='relu',kernel_initializer='he_normal'))
model.add(MaxPool2D((2, 2)))
model.add(Dropout(0.20))
model.add(Conv2D(64, (3, 3), activation='relu',padding='same',kernel_initializer='he_normal'))
model.add(Conv2D(64, (3, 3), activation='relu',padding='same',kernel_initializer='he_normal'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Conv2D(128, (3, 3), activation='relu',padding='same',kernel_initializer='he_normal'))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(BatchNormalization())
model.add(Dropout(0.25))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.RMSprop(),
              metrics=['accuracy'])

learning_rate_reduction = ReduceLROnPlateau(monitor='val_acc',
                                            patience=3,
                                            verbose=1,
                                            factor=0.5,
                                            min_lr=0.0001)

datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=15, # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
       ...


I am working on the file "Classifying_datasets/MNIST/mnist-with-keras-for-beginners-99457.ipynb".
The first task is:
Loading The Dataset
Can you write Python code for it?

#loading the dataset.......(Train)
train = pd.read_csv("../input/train.csv")
print(train.shape)
train.head()

z_train = Counter(train['label'])
z_train

sns.countplot(train['label'])

#loading the dataset.......(Test)
test= pd.read_csv("../input/test.csv")
print(test.shape)
test.head()

x_train = (train.ix[:,1:].values).astype('float32') # all pixel values
y_train = train.ix[:,0].values.astype('int32') # only labels i.e targets digits
x_test = test.values.astype('float32')

%matplotlib inline
# preview the images first
plt.figure(figsize=(12,10))
x, y = 10, 4
for i in range(40):
    plt.subplot(y, x, i+1)
    plt.imshow(x_train[i].reshape((28,28)),interpolation='nearest')
plt.show()


I am working on the file "Classifying_datasets/MNIST/mnist-with-keras-for-beginners-99457.ipynb".
The first task is:
Printing the shape of the Datasets
Can you write Python code for it?

print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')


I am working on the file "Classifying_datasets/Statoil/iceberg_v2.ipynb".
The first task is:
Below picture is not an iceberg
Can you write Python code for it?


I am working on the file "Classifying_datasets/Statoil/iceberg_v2.ipynb".
The first task is:
Get rgb of image using color composite function
Thanks to MadScientist for color composite.
Here is the kernal --
https://www.kaggle.com/keremt/getting-color-composites
Can you write Python code for it?

def color_composite(data):
    rgb_arrays = []
    for i, row in data.iterrows():
        band_1 = np.array(row['band_1']).reshape(75, 75)
        band_2 = np.array(row['band_2']).reshape(75, 75)
        band_3 = band_1 / band_2

        r = (band_1 + abs(band_1.min())) / np.max((band_1 + abs(band_1.min())))
        g = (band_2 + abs(band_2.min())) / np.max((band_2 + abs(band_2.min())))
        b = (band_3 + abs(band_3.min())) / np.max((band_3 + abs(band_3.min())))

#         r = ((band_1 - np.mean(band_1)) / (np.max(band_1) - np.min(band_1)))
#         g = ((band_2 - np.mean(band_2)) / (np.max(band_2) - np.min(band_2)))
#         b = ((band_3 - np.mean(band_3)) / (np.max(band_3) - np.min(band_3)))

        rgb = np.dstack((r, g, b))
        rgb_arrays.append(rgb)
    return np.array(rgb_arrays)

# Trained with data about rgb
rgb_train = color_composite(train)
rgb_train.shape

# Test with data about rgb
rgb_test = color_composite(test)
rgb_test.shape


I am working on the file "Classifying_datasets/Statoil/iceberg_v2.ipynb".
The first task is:
Loss is decreasing with increaseing iterations
Fine tuning and differential annealing
Since, the images that we have do not have very clear and sharp features which might be present in top layers of our pre-trained neural network. Let's unfreeze our layers and calculate activations
Can you write Python code for it?

lr = 0.01

lrs=np.array([lr/9,lr/3,lr]) # use lr/100 and lr/10 respectively if images would have been larger in sizes

%time learn.fit(lrs, 5, cycle_len=1, cycle_mult=2, cycle_save_name='resnet50')

learn.sched.plot_loss()

%time learn.fit(lrs, 4, cycle_len=1, cycle_mult=2)

learn.sched.plot_loss()


I am working on the file "Classifying_datasets/Statoil/iceberg_v2.ipynb".
The first task is:
Observation from images --
• Ships have a trace of bright lights around them which will be taken as a feature in CNN
• Ships are more consistent with shapes
• Icebergs shape vary more than ships
saving images in directories (train, valid, test)
Can you write Python code for it?

# # making directories for training resnet (as it need files to be in right dir)

os.makedirs(f'{path}composites', exist_ok= True)
os.makedirs(f'{path}composites/train', exist_ok=True)
os.makedirs(f'{path}composites/valid', exist_ok=True)
os.makedirs(f'{path}composites/test', exist_ok=True)

dir_list = [f'{path}composites/train', f'{path}composites/valid']

for i in dir_list:
    os.makedirs(f'{i}/ship')
    os.makedirs(f'{i}/iceberg')


I am working on the file "Classifying_datasets/Statoil/iceberg_v2.ipynb".
The first task is:
TTA (Test Time Augmentation) simply makes predictions not just on the images in your validation set, but also makes predictions on a number of randomly augmented versions of them
Predictions
Can you write Python code for it?

# from here we know that 'icebergs' is label 0 and 'ships' is label 1.
data.classes

# this gives prediction for validation set. Predictions are in log scale
log_preds = learn.TTA(is_test=True) # If need TTA
#log_preds = learn.predict(is_test=True) # if don't need TTA
log_preds

probs_submit1 = np.exp(log_preds[0][:,0])
probs_submit1[:]

# getting ids from test list
id_raw = data.test_dl.dataset.fnames
id_raw[1]

# using regex to take numbers from ids
id_pro = result_array = np.empty((0, len(id_raw)))
for i in range(len(id_raw)):
    stuff = int(re.findall(r'\d+', id_raw[i])[0])
    #print(type(str(stuff)))
    id_pro = np.append(id_pro,int(stuff))

id_pro_list = []
for i in id_pro:
    id_pro_list.append(int(i))
id_pro_list[:4]

# joining id and probability
d = {'index': id_pro_list, 'is_iceberg': probs_submit1}
submit1_df = pd.DataFrame(data=d)

id_ = test['id']
id_pd = pd.DataFrame({'id': id_} )
submit1_df_sorted = submit1_df.sort_values('index')
submit1_df_sorted2 = pd.concat([id_, submit1_df_sorted.set_index('index')], axis = 1)
submit1_df_sorted2.dtypes
submit1_df.dtypes

d['is_iceberg1']

l=[]
for i in range(len(d['is_iceberg1'])):
    l.append(d['is_iceberg1'][i][0])

len(l)

d_new = {'index': id_pro_list, 'is_iceberg': l}
d_new['is_iceberg']

submit1_df = pd.DataFrame(data=d_new)

id_ = test['id']
id_pd = pd.DataFrame({'id': id_} )

submit1_df_sorted = submit1_df.sort_values('index')
submit1_df_sorted2 = pd.concat([id_, submit1_df_sorted.set_index('index')], axis = 1)

submit1_df_sorted2.dtypes

submit1_df.dtypes

#submit1_df.id = submit1_df.id.astype(str)

submit1_df_sorted2.to_csv('data/processed/resnet50.csv', index = False)

! head -5 data/processed/resnet50.csv

submit_check = pd.read_csv("data/processed/resnet50.csv")

submit_check.dtypes

submit_check[:10]

submit_check['is_iceberg']


I am working on the file "Classifying_datasets/Statoil/iceberg_v2.ipynb".
The first task is:
The reason of converting these images to .png is that the pretrained ConvLearner that I am going to call takes image as input
Can you write Python code for it?

! ls {path}composites

# split
train_y, valid_y = train_test_split(train.is_iceberg, test_size=0.10)

train_iceberg_index, train_ship_index, valid_iceberg_index, valid_ship_index  = train_y[train_y==1].index, train_y[train_y==0].index, valid_y[valid_y==1].index, valid_y[valid_y==0].index

#save train images
for idx in train_iceberg_index:
    img = rgb_train[idx]
    plt.imsave(f'{path}/composites/train/iceberg/' + str(idx) + '.png',  img)

for idx in train_ship_index:
    img = rgb_train[idx]
    plt.imsave(f'{path}/composites/train/ship/' + str(idx) + '.png',  img)

#save valid images
for idx in valid_iceberg_index:
    img = rgb_train[idx]
    plt.imsave(f'{path}/composites/valid/iceberg/' + str(idx) + '.png',  img)

for idx in valid_ship_index:
    img = rgb_train[idx]
    plt.imsave(f'{path}/composites/valid/ship/' + str(idx) + '.png',  img)

#save test images
for idx in range(len(test)):
    img = rgb_test[idx]
    plt.imsave(f'{path}/composites/test/' + str(idx) + '.png',  img)


I am working on the file "Classifying_datasets/Statoil/iceberg_v2.ipynb".
The first task is:
epochs = 3
cycle length = 1
cycle multiple = 2
Can you write Python code for it?

learn.fit(lr, 3, cycle_len=1, cycle_mult=2) # precompute was false # first fit

# stochastic descent with restart
learn.sched.plot_lr()

learn.sched.plot_loss()

learn.fit(lr,4, cycle_len=1, cycle_mult=2) # precompute was false # first fit

# stochastic descent with restart
learn.sched.plot_lr()

learn.sched.plot_loss()


I am working on the file "Classifying_datasets/Statoil/iceberg_v2.ipynb".
The first task is:
let's look at a random ships now (from png) to make sure images are saved in directory
Can you write Python code for it?

files = !ls {path2}valid/ship | head

img = plt.imread(f'{path2}valid/ship/{files[0]}')
plt.imshow(img)

! ls {path2}


I am working on the file "Classifying_datasets/Statoil/iceberg_v2.ipynb".
The first task is:
scp submit1 to local machine to upload on kaggle
Post analysis
Looking at correctly, incorrectly classified images
Analyzing results: looking at pictures
As well as looking at the overall metrics, it's also a good idea to look at examples of each of:
* A few correct labels at random
* A few incorrect labels at random
* The most correct labels of each class (ie those with highest probability that are correct)
* The most incorrect labels of each class (ie those with highest probability that are incorrect)
* The most uncertain labels (ie those with probability closest to 0.5).
Can you write Python code for it?

log_pred1 = learn.TTA() # If need TTA

preds = np.argmax(log_pred1[0], axis=1)  # from log probabilities to 0 or 1
probs = np.exp(log_pred1[0][:,1])        # pr(ship)

#probs = submit_check['is_iceberg']
#submit_check['C'] = np.where(submit_check['is_iceberg'] >= 0.5,1, 0)
#preds = np.array(submit_check['C'])

def rand_by_mask(mask): return np.random.choice(np.where(mask)[0], 4, replace=False)
def rand_by_correct(is_correct): return rand_by_mask((preds == data.val_y)==is_correct)

def plot_val_with_title(idxs, title):
    imgs = np.stack([data.val_ds[x][0] for x in idxs])
    title_probs = [probs[x] for x in idxs]
    print(title)
    return plots(data.val_ds.denorm(imgs), rows=1, titles=title_probs)

def plots(ims, figsize=(12,6), rows=1, titles=None):
    f = plt.figure(figsize=figsize)
    for i in range(len(ims)):
        sp = f.add_subplot(rows, len(ims)//rows, i+1)
        sp.axis('Off')
        if titles is not None: sp.set_title(titles[i], fontsize=16)
        plt.imshow(ims[i])

def load_img_id(ds, idx): return np.array(PIL.Image.open(path2+ds.fnames[idx]))

def plot_val_with_title(idxs, title):
    imgs = [load_img_id(data.val_ds,x) for x in idxs]
    title_probs = [probs[x] for x in idxs]
    print(title)
    return plots(imgs, rows=1, titles=title_probs, figsize=(16,8))

# 1. A few correct labels at random
plot_val_with_title(rand_by_correct(True), "Correctly classified")

# 2. A few incorrect labels at random
plot_val_with_title(rand_by_correct(False), "Incorrectly classified")

def most_by_mask(mask, mult):
    idxs = np.where(mask)[0]
    return idxs[np.argsort(mult * probs[idxs])[:4]]

def most_by_correct(y, is_correct):
    mult = -1 if (y==1)==is_correct else 1
    return most_by_mask((preds == data.val_y)==is_correct & (data.val_y == y), mult)

plot_val_with_title(most_by_correct(0, True), "Most correct icebergs")

plot_val_with_title(most_by_correct(1, True), "Most correct ships")


I am working on the file "ClausiusClapeyronModuleTutorial.ipynb".
The first task is:
Finally, let us convert the vapor pressures to mass in grams. Humidity in g/m3 is called the absolute humidity. We will do this using the ideal gas law, p=$\rho$RT. We need to solve this for the molar density and then convert it to a mass density.
Can you write Python code for it?

#First, let's enter some constants.
R = 8.314 #m3 Pa K−1 mol−1 We're going to need to do some more units conversions!
rho_noon = (ed_noon*1000)/(R*(TnoonC+273.15)) #now this is in moles/m^3
rho_night = (ed_night*1000)/(R*(TnightC+273.15))

#Now we will convert the molar densities to mass densities
MW_H2O = 18.01528 #grams of water per mole
m_noon = rho_noon*MW_H2O #grams per m^3
m_night = rho_night*MW_H2O
print(m_noon)
print(m_night)


I am working on the file "ClausiusClapeyronModuleTutorial.ipynb".
The first task is:
Is the atmosphere at equilibrium with respect to water vapor content? Let's find out by solving for the actual vapor pressure from the dewpoint. Note that we can do this by plugging the dewpoint in to our saturation vapor pressure approximator above.
Can you write Python code for it?

ed_act_noon = CC_esat(Tdp_noonC) #in kPa
ed_act_night = CC_esat(Tdp_nightC) #in kPa
print(ed_act_noon)
print(ed_act_night)


I am working on the file "ClausiusClapeyronModuleTutorial.ipynb".
The first task is:
Let's convert all of the Farenheit data to Celsius by creating a module (i.e., a function).
Can you write Python code for it?

def FtoC(TF):
    """
    Takes the temperature in Farenheit (TF) as input and returns it in Celsius (TC)
    """
    TC = (TF-32)*5/9
    return TC


I am working on the file "ClausiusClapeyronModuleTutorial.ipynb".
The first task is:
Let's now figure out actual vapor pressure from the relative humidity.
Can you write Python code for it?

ed_noon = esat_noon*RHnoon #in kPa
ed_night = esat_night*RHnight #in kPa
print(ed_noon)
print(ed_night)


I am working on the file "ClausiusClapeyronModuleTutorial.ipynb".
The first task is:
Now we can feed this equation today's temperatures to calculate the saturation vapor pressure.
Can you write Python code for it?

esat_noon = CC_esat(TnoonC) #saturation vapor pressure at noon
esat_night = CC_esat(TnightC) #saturation vapor pressure at 4 am
print(esat_noon)
print(esat_night)


I am working on the file "ClausiusClapeyronModuleTutorial.ipynb".
The first task is:
What is today's pressure in Pa?
Can you write Python code for it?

Pinches = 30.12 #pressure in inches
Pmm = 30.12*25.4 #There are 25.4 mm in an inch
PPa = Pmm*133.322
print(PPa)


I am working on the file "Clean HPCC Data.ipynb".
The first task is:
Save the data method wise to a folder in tsv.gz format
Can you write Python code for it?

#for i in names_list:
#    print(d[i])
import os
if not os.path.isdir('newBenchmark_results'):
    os.mkdir('newBenchmark_results')

gb = methodWiseData['GradientBoostingClassifier']
gb.to_pickle('newBenchmark_results/GradientBoostingClassifier_results.tsv.gz')

method_data = pd.read_pickle('newBenchmark_results/GradientBoostingClassifier-benchmark_results.tsv.gz')
method_data


I am working on the file "Clean HPCC Data.ipynb".
The first task is:
Save this result in the HPCC_benchmark_results folder
Can you write Python code for it?

import os
if not os.path.isdir('HPCC_benchmark_results'):
    os.mkdir('HPCC_benchmark_results')
result.to_pickle('HPCC_benchmark_results/GradientBoostingClassifier-hpcc_results.tsv.gz')

data = pd.read_pickle('HPCC_benchmark_results/GradientBoostingClassifier-hpcc_results.tsv.gz')
data


I am working on the file "Clean HPCC Data.ipynb".
The first task is:
Store the data method wise separately
Can you write Python code for it?

methodWiseData = {}
for name in methodNames_list:
    methodWiseData[name] = benchmark_data[(benchmark_data.Method_Name == name)]


I am working on the file "Cleaning of data.ipynb".
The first task is:
Fix the numeric values to actually be numeric
The numeric values are filled it by had. And that causes some to be in different formats.
This code detects the format and parses to a numeric value.
In the end, there are still some values that are not parsable, and need to be fixed by hand.
Can you write Python code for it?

# Make Project investment numeric
import re

re_avg = re.compile(r'(\d+)-(\d+)(E\d+)')

def fixnumb(inp: str):
    if isinstance(inp, float):
        return inp
    if not inp or not inp.strip():
        return ''
    x = inp.upper().replace('US$', '').replace(' ', '').replace(',', '.')
    x = x.replace('BILLION', 'E9').replace('MILLION', 'E6')
    try:
        return float(x)
    except ValueError as e:
        if re_avg.match(x):
            res = re_avg.search(x)
            left, right, sin = res.groups()
            left = float(left)
            right = float(right)
            avg = (left+right)/2
            try:
                return float(repr(avg) + sin)
            except ValueError as e:
                print('x: %r, a: %r, %s %s' % (x, avg, e, inp))
                return inp
        print('x: %r, %s %s' % (x, e, inp))
        return inp

doc['Projected investment'] = doc['Projected investment'].map(fixnumb)
doc[:3]


I am working on the file "Cleaning of data.ipynb".
The first task is:
Pivot Table
A pivot table is a realy good way to visualise different values.
In this table you can see the hectares per land grabber per country.
Can you write Python code for it?

pd.pivot_table(doc, values=['Hectares'], index=['Base', 'Landgrabber'])


I am working on the file "Clicks.ipynb".
The first task is:
Group returns a new table.  If we wanted to specify the formats on columns of this table, assign it to a name.
Can you write Python code for it?

# How does gender and clicks vary with age?
gi = impressed.group('Age Range', np.mean).select(['Age Range', 'Gender mean', 'Clicks mean'])
gi.set_format(['Gender mean', 'Clicks mean'], PercentFormatter)
gi


I am working on the file "Clicks.ipynb".
The first task is:
Now we can group the table by Age Range and count how many clicks come from each range.
Can you write Python code for it?

clicks_by_age = clicks.group('Age Range', sum)
clicks_by_age

clicks_by_age.select(['Age Range', 'Clicks sum', 'Impressions sum', 'Person sum']).barh('Age Range')


I am working on the file "Clicks.ipynb".
The first task is:
This workbook shows a example derived from the EDA exercise in Chapter 2 of Doing Data Science, by o'Neil abd Schutt
Can you write Python code for it?

clicks = Table.read_table("http://stat.columbia.edu/~rachel/datasets/nyt1.csv")
clicks


I am working on the file "Clicks.ipynb".
The first task is:
Well.  Half a million rows.  That would be painful in excel.
Add a column of 1's, so that a sum will count people.
Can you write Python code for it?

age_upper_bounds = [18, 25, 35, 45, 55, 65]

def age_range(n):
    if n == 0:
        return '0'
    lower = 1
    for upper in age_upper_bounds:
        if lower <= n < upper:
            return str(lower) + '-' + str(upper-1)
        lower = upper
    return str(lower) + '+'

# a little test
np.unique([age_range(n) for n in range(100)])

clicks["Age Range"] = clicks.apply(age_range, 'Age')
clicks["Person"] = 1
clicks


I am working on the file "Clock Arithmetic.ipynb".
The first task is:
Aha!  Now we're getting somewhere.  M(3) * M(7) returns M(1), so these two are inverses of one another.  One fact to notice immediately is neither has any factors in common with 10. In fact, both are prime numbers in the ordinary integer sense.  M(9) is not prime, but again, 9 has no factors in common with 10.  So does M(9) have an inverse?  Lets check:
Can you write Python code for it?


I am working on the file "Clock Arithmetic.ipynb".
The first task is:
And yet 561 is a composite number, with prime factors (3, 11, 17).
Another test for primality, called the AKS Test, is more fool-proof.  We'll need the coefficients of the polynomial (x - 1) ** p, where p is our candidate prime.  Those coefficients may be read of Pascal's Triangle.  If all coefficients but the first and last (which are 1) are divisible by p, then p is prime.
Lets try that out, first by writing a Python generator for successive rows of Pascal's Triangle:
Can you write Python code for it?

def pascal():
    row = [1]
    while True:
        yield row
        row = [(i+j) for i,j in zip(row + [0], [0] + row)]

gen = pascal()
for i in range(10):
    print(next(gen))


I am working on the file "Clock Arithmetic.ipynb".
The first task is:
For example, pow(M(79), -2) means find the inverse of M(79) e.g. M(79)**-1, and then raise the result to the 2nd power.
Can you write Python code for it?

class M:  # for "modulo"

    """
    Version 2, with inverting and powering added
    Version 3, not listed, is in groups.py and uses xgcd (next Chapter)
    """

    modulus = 10  # class level

    def __init__(self, val, modulus=None):
        if modulus:
            self.modulus = M.modulus = modulus # resettable
        else:
            self.modulus = M.modulus
        self.val = val % M.modulus

    def __add__(self, other):
        if self.modulus != other.modulus:
            raise ValueError
        return M(self.val + other.val, self.modulus)

    def __mul__(self, other):
        if self.modulus != other.modulus:
            raise ValueError
        return M(self.val * other.val, self.modulus)

    def __invert__(self):
        elems = {M(x) for x in totatives(self.modulus)}
        for m in elems:
            if (self * m).val == 1:
                return m

    def __pow__(self, exp):  # pow() and ** both trigger this method
        output = self
        if exp < 0:
            exp = abs(exp)  # make exp positive now
            output = ~self  # -1 means take the inverse
        elif exp == 0:
            output = M(1)   # return identity if exp is 0
        elif exp == 1:
            output = self   # return M if exp is 1
        if exp > 1:
            for _ in range(1, exp):  # use __mul__ (already defined)
                output *= self

        return output

    def __repr__(self):
        return "(" + str(self.val) + " mod " + str(M.modulus)+ ")"

M.modulus = 100
a = M(79)
a_inv = pow(a, -1)
a_inv

M(13) * M(13) * M(13)

M(13)**3       # confirm this is another way of saying it (we're testing)

M(13)**-1      # give me the inverse of M(13) please

M(13) * M(77)  # these must be inverses then


I am working on the file "Clock Arithmetic.ipynb".
The first task is:
Indeed it does.  M(9) * M(9) = M(1), meaning M(9) is its own inverse.
When positive integers have no factors in common, aside from 1, we say they're "relatively prime" or that they're "strangers" to one another.  3 and 10 are strangers, as are 9 and 10.  Sometimes we write 7 | 10 = 1, meaning their greatest factor in common is 1.  On the other hand, 6 | 10 = 2, as 2 divides into both without remainder.
What we're about to discover is the strangers to a modulus comprise a group, i.e. M(n) where n | N = 1 and M.modulus = N.  A Cartesian product of such "minions" would demonstrate closure, inverse for everyone, naturally a neutral element, and associativity i.e. makes no difference if you go M(a) * M(b) * M(c) by starting with either M(a) * M(b) or M(b) * M(c).  Commutativity is not a requirement for grouphood.
Finding Totatives
It'd sure be handy at this point, to have a function, gcd, that returns the greatest common divisor of two numbers.  Then we could find all the strangers to 10, that are positive integers less than 10, pretty easily. They would have no common divisor with 10.
We call these strangers the "totatives" of 10, and the number of totatives, is called the "totient" of 10.
Note that we're using list comprehension syntax in this example.
The for loop inside the square brackets makes x be every integer from 0 to modulus - 1, but then most of these get filtered out, because of factors in common with the modulus.
Can you write Python code for it?

import math
modulus = 10
[x for x in range(modulus) if math.gcd(x, modulus) == 1]

modulus = 12
[x for x in range(modulus) if math.gcd(x, modulus) == 1]


I am working on the file "Clock Arithmetic.ipynb".
The first task is:
Now that we have our inverse function, we can ask questions such as:

What is the inverse of M(5, 7)?
What is the inverse of M(7, 5)?
What is the inverse of M(97, 100)?
Can you write Python code for it?

print("What is the inverse of M(5, 7)?", ~M(5,7))
print("What is the inverse of M(7, 5)?", ~M(7,5))  # M(7,5) is same as M(2,5)
print("What is the inverse of M(97, 100)?", ~M(97, 100))


I am working on the file "Clock Arithmetic.ipynb".
The first task is:
OK, everything seems to be working, even though we haven't implemented powering yet.  Eventually we'd like to go pow(M(3), -1) to get the inverse of M(3), such that M(3) times its inverse equals the multiplicative identity M(1).
Looking for Groups
But wait, does every M-number, with modulus set to 10, have an inverse?  We can check that easily.  First, lets make a list of all 10 M-numbers, (0) through (9):
Can you write Python code for it?


I am working on the file "Clock Arithmetic.ipynb".
The first task is:
Primality Tests
Now that we have a powering method in place, lets take a look a Fermat's Little Theorem.  The logic here is useful, in the sense that we sometimes get tripped up by IF / THEN statements.
IF a number p is prime, THEN M(A, p) ** p == A, where 0 < A < p. If some number n passes this test for a given A, where gcd(A, n) == 1, then that's evidence n might be prime, but not proof.  If n in fact turns out to be composite, then a is called a Fermat Liar or Fermat Fooler.
Lets take a look.  Is 511 prime?  Lets find an A with no factors in common with 511.
Can you write Python code for it?

from math import gcd
gcd(511, 25)

n = M(25, 511) # n will power modulo 511
n ** 511


I am working on the file "Clock Arithmetic.ipynb".
The first task is:
Same thing.  We cycle around and around, always stopping at the same stations, like a model train going in a circle.  We never stop at station M(1).  So M(2) has no inverse either.  What about M(3)?
Can you write Python code for it?


I am working on the file "Clock Arithmetic.ipynb".
The first task is:
That seems to be working.  So Pascal's Triangle could be at the heart of a new iterator for producing successive prime numbers.
Can you write Python code for it?

def primes():
    yield 2
    p = 1
    gen = pascal()
    next(gen) # [1]
    next(gen) # [1, 1]
    while True:
        p += 2 # 3, 5... check odd rows only, once 2 yielded
        next(gen)     # skip even row
        r = set(next(gen)[1:-1]) # drop 1s on both ends and dedupe
        if sum([coeff%p for coeff in r]) == 0:
            yield p

ps = primes()
print(str([next(ps) for _ in range(100)]))


I am working on the file "Clock Arithmetic.ipynb".
The first task is:
That was pretty easy.  Right away we're getting totatives.  We could be using set comprehension syntax instead.
Using Python's "list comprehension" syntax, which allows an if clause for filtering, we were able to get the totatives of 10 and 12 respectively.  Both 10 and 12 have a totient of four, meaning each has four totatives.
What's true is that the M-numbers that are totatives of some modulus (say 10), collectively form a group under multiplication.
If we confine ourselves to strangers to the modulus, we'll have an inverse for every element, closure, associativity, and M(1) will be included.  That's a quick way to get a group.
We don't have closure for addition though.  If we want to include addition, along with its neutral element zero, then we'll need to make our modulus a prime number.  Then we will be guaranteed a field, a Galois Field to be more precise, though in German they say it differently.
Before we jump to fields though, lets build a function for computing totatives and use that to build some groups.
A "set comprehension" is just like a list comprehension in terms of syntax, but the curly braces make it for a set, instead of a list.
Sets are unordered and their elements are always unique (no duplicates).  Since we know totatives are unique, we might as well practice using a set object.
Can you write Python code for it?

def totatives(n):
    return {x for x in range(n) if math.gcd(x, n) == 1}

elems = {M(x, 12) for x in totatives(12)}
elems


I am working on the file "Clock Arithmetic.ipynb".
The first task is:
The number one million has a totient of 400,000, meaning that many numbers from one to one million are co-prime to it.
Adding a Power Method
We now have the technology (tool set) we need to add the __pow__ method to our M class. A negative power triggers finding an inverse of a, the -1 part, then we switch back to positively powering.  We're saying $M(n)^{-e}$ equals $(M(n)^{-1})^{e}$.
This is how we treat exponents normally, e.g. 2 ** -2 equals (2 ** -1)**2 or 1/4.
Can you write Python code for it?


I am working on the file "Clock Arithmetic.ipynb".
The first task is:
Try doing a number of these yourself.  Guess the answer before hitting the Shift-Enter key.
You may find your first years of formal schooling did not emphasize cyclic arithmetic, except in the two most common forms, of telling time and scheduling appointments.  We understand both the 24 hour day, and the 365 day year, comprise a modulus such that we're always talking about our degree of offset into each one.  Adding hours or days, per a timedelta object, keeps us within the scope of some calendar.
The unit circle of 360 degrees, or 2 pi radians, is also treated in a clock-like fashion.  Going around thousands of degrees from some present position, never takes us beyond 360.  We're confined to a finite domain.  Cyclic phenomena more generally are accommodated by our clock arithmetic, of complex numbers, of e to imaginary powers.
There's a kind of "closure" in this picture, which might be resonating with you by now, as one of those properties of a group.  Will we find groups in this Chapter?  You bet.  Groups also enjoy the symmetry of a circle in that each member is paired with another "180 degrees opposite" although sometimes the inverse might be the same as the self.
Again, we call this "clock arithmetic" because even if we say "20 hours from now", or "30 hours ago" we'll still be somewhere on the circle marked out into 12 intervals, each representing one hour.  The yearly calendar is likewise a kind of modulus.  No matter how many days we add, we're still somewhere between 0 and 365 days into some year.
We also call it "modulo arithmetic" meaning the computations are always vis-a-vis some fixed modulus.  This word "modulus" is our inspiration for the M-numbers coded below.
A Class for M-numbers
Lets build a class, the instances of which will multiply and add per some fixed modulus, meaning we're always factoring out the modulus and keeping the remainder.
Can you write Python code for it?

class M:  # for "modulo"

    modulus = 10  # class level

    def __init__(self, val, modulus=None):
        if modulus:
            self.modulus = M.modulus = modulus # resettable
        else:
            self.modulus = M.modulus
        self.val = val % M.modulus

    def __add__(self, other):
        if self.modulus != other.modulus:
            raise ValueError
        return M((self.val + other.val) % self.modulus)

    def __mul__(self, other):
        if self.modulus != other.modulus:
            raise ValueError
        return M((self.val * other.val) % self.modulus)

    def __pow__(self, exp):
        raise NotImplemented

    def __repr__(self):
        return "(" + str(self.val) + " mod " + str(self.modulus)+ ")"

a = M(8)
b = M(7)
print(a, b)
print("(8 mod 10) * (7 mod 10) = ", a * b)
print("(8 mod 10) + (7 mod 10) = ", a + b)


I am working on the file "Clock Arithmetic.ipynb".
The first task is:
What we just did there was compose a group of M-number objects, of twelves four strangers.  If you're using this Notebook live, here might be a good time to insert some code cells of your own, testing out whether the Cayley Table, i.e. the everything by everything multiplication table, really shows closure, and 1 in every row (proof of inverse element).
Can you write Python code for it?

M.modulus = 100
elems = {M(x, 100) for x in totatives(100)}  # set comprehension
elems  # print M-number totatives in no special order -- it's a set


I am working on the file "Closure a dekorator.ipynb".
The first task is:
A aj k ich hodnotam
Can you write Python code for it?

print(avg.__closure__)
print(avg.__closure__[0].cell_contents) # tato hodnota sa da aj zmenit, ale nerobte to.


I am working on the file "Closure a dekorator.ipynb".
The first task is:
A teraz to vyskusame
Can you write Python code for it?

measure_add = make_adder()
gc.collect()
print_memory_usage()
print(reduce(measure_add, [x*x for x in range(10000000)]))


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Ale my globalne premenne nemame radi. Takze toto nebudeme pouzivat
Co si zapamatat o rozsahu platnosti premennych?

funkcia vie citat premenne definovane mimo svojho tela
tieto premenne ale nevie menit.
ak by ich chcela menit, tak pri kompilacii do bitekodu sa vytvori lokalna premenna a ak ju pouzijeme skor ako jej priradime hodnotu, tak mame problem
teoreticky mozeme pouzit globalnu premennu, ale my nechceme

Dalsia vec, ktoru si potrebujeme vysvetlit na to aby sme spravili closure su vnorene funkcie
Ano, funkcie sa daju definovat vo vnutri inej funkcie
Can you write Python code for it?

def outer():
    def inner(a):
        return a + 7
    return inner


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Alebo nahradit niecim uplne inym
Can you write Python code for it?

def nahradzujuci_dekorator(param_fct):
    def nieco_uplne_ine():
        pass
    return nieco_uplne_ine


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Co dava zmysel ak sa vlastne deje toto
Can you write Python code for it?

def f1():
    print('running f1()')

f1 = register(f1)

def f2():
    print('running f2()')

f2 = register(f2)

def f3():
    print('running f3()')

f1()
f2()
f3()


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Co ked ma tych parametrov viac?
To iste ako v predchadzajucom pripade. Wrapper musi mat tie iste parametre.
Skusme to zovseobecnit pre funkcie s hociakym poctom atributov
Can you write Python code for it?

def param_decorator2(param_fct):
    def wrapper(*args): # wrapper musi mat tie iste parametre
        print('wrapper stuff')
        return param_fct(*args) # co sa stane ked tu nebude *?
    return wrapper

@param_decorator2
def my_print(string):
    print(string)

@param_decorator2
def my_print_more(string1, string2, string3):
    print(string1, string2, string3)

@param_decorator2
def my_print_many(*args):
    print(*args)

my_print('hello')
my_print_more('hello', 'hello2', 'hello3')
my_print_many('hello', 'hello2', 'hello3', 'hello4', 'hello5')


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Da sa pristupit k nazvom premennych a aj volnych premennych
Can you write Python code for it?


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Daju sa pouzit napriklad na kontrolu vstupov odelenu od samotenho vypoctu
Can you write Python code for it?

# https://realpython.com/blog/python/inner-functions-what-are-they-good-for/
def factorial(number):

    # error handling
    if not isinstance(number, int):
        raise TypeError("Sorry. 'number' must be an integer.")
    if not number >= 0:
        raise ValueError("Sorry. 'number' must be zero or positive.")

    # logika spracovania je pekne sustredena na jednom mieste
    def inner_factorial(number):
        if number <= 1:
            return 1
        return number*inner_factorial(number-1)
    return inner_factorial(number)

# call the outer function
print(factorial(4))


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Dekorovanim menim niektore atributy funkcie
Can you write Python code for it?

def some_fct():
    """doc string of some_fct"""
    print("some stuff")

some_fct()
print(some_fct.__name__)
print(some_fct.__doc__)
print(some_fct.__module__)

def decorator(f):
    def wrapper_fct(*args, **kwargs):
        """wrapper_fct doc string"""
        return f(*args, **kwargs)
    return wrapper_fct

@decorator
def some_fct():
    """doc string of some_fct"""
    print("some stuff")

some_fct()
print(some_fct.__name__)
print(some_fct.__doc__)
print(some_fct.__module__)


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Funkcia vie citat premennu definovanu mimo svojho tela
Can you write Python code for it?

b = 6

def f1(a):
    print(a)
    print(b)

f1(3)

b = 6
def f2(a):
    print(a)
    print(b)
    b = 9

f2(3)

b = 6
def f2(a):
    print(a)
    print(b) # tuna nevypisujeme obsah premennej z prveho riadku ale tej, ktora je az na dalsom riadku. To logicky nemoze prejst
    b = 9 # akonahle raz vo funkcii priradujete do premennej, tak sa vytvori nova, lokalna pri kompilacii do bitekodu.

f2(3)


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Ked uz mame tu funkciu, tak s nou mozeme nieco aj spravit - napriklad obalit niecim inym
Can you write Python code for it?

def zaujimavejsi_dekorator(param_fct):
    def inner():
        do_stuff()
        result = param_fct()
        do_another_stuff()
        return result
    return inner


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Minule som vam slubil vysvetlenie ako opravit jeden hack
Mali sme kod, ktory meral mnozstvo pamati spotrebovanej pri pocitani s generatorom a bez neho
Funkcia measure_add mala vedlajsi efekt, ktory vypisoval spotrebu pamati kazdych 200 000 volani. Potrebovala teda pocitadlo, ktore si uchovavalo stav medzi volaniami. Pouzili sme mutable objekt na to aby sme nemuseli pouzit priradovanie a nesnazili sa pristupit k premennej pred jej definicou alebo aby sme nedefinovali globalnu premennu..
Can you write Python code for it?

from functools import reduce
import gc
import os
import psutil
process = psutil.Process(os.getpid())

def print_memory_usage():
    print(process.memory_info().rss)

counter = [0] # Toto je ta globalny mutable objekt
def measure_add(a, result, counter=counter):
    if counter[0] % 200000 == 0:
        print_memory_usage()
    counter[0] = counter[0] + 1
    return a + result

gc.collect()
counter[0] = 0
print_memory_usage()
print(reduce(measure_add, [x*x for x in range(1000000)]))


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Na spocitanie priemeru nepotrebujeme cely zoznam. Staci nam suma a pocet.
Tento priklad je ale pokazeny. Kto vie preco? Uz som to naznacil viac krat.
Can you write Python code for it?

def make_averager():
    count = 0
    total = 0
    def averager(new_value):
        count += 1
        total += new_value
        return total / count
    return averager

avg = make_averager()
avg(10)


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Na vypocet potrebujeme uchovavat stav: sumu a pocet hodnot. Kde sa uklada stav?

V globalnej premennej? Nie, nechceme si zapratat priestor mien nejakymi nahodnymi premennymi, ktore by nam mohol hocikto prepisat.
Potrebuje nieco, co nam tie premenne schova.

Nieco taketo by sa dalo implementovat pomocou uzaveru
Can you write Python code for it?

def make_averager():
    series = [] # tato premenna je platna len vo funkcii make_averager. Je pre nu lokalna. Mimo nej neexistuje.
    def averager(new_value):
        series.append(new_value) # vieme pristupit k premennej definovanej vyssie.
        # Kedze list je mutable, tak ho vieme aj zmenit. Pozor, nemenime premennu, menime objekt!
        # Aby sme menili premennu, tak by tu muselo byt =
        total = sum(series)
        return total/len(series)
    return averager

avg = make_averager()
print(avg(10))
print(avg(11))
print(avg(12))


I am working on the file "Closure a dekorator.ipynb".
The first task is:
No a co pomenovane atributy?
Can you write Python code for it?

def my_print_optional(first, second='second', third='third'):
    print(first, second, third)

my_print_optional('1', '2', '3')
my_print_optional('1', '2')
my_print_optional('1')
my_print_optional('1', third='3', second='2')
my_print_optional('1', third='3')

def param_decorator3(param_fct):
    def wrapper(*args, **kwargs): # wrapper musi mat tie iste parametre
        print('wrapper stuff')
        return param_fct(*args, **kwargs)
    return wrapper

@param_decorator3
def my_print_optional(first, second='second', third='third'):
    print(first, second, third)

my_print_optional('1', '2', '3')
my_print_optional('1', '2')
my_print_optional('1')
my_print_optional('1', third='3', second='2')
my_print_optional('1', third='3')


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Obalujuci dekorator prida nieco pred a/alebo za volanie funkcie
Can you write Python code for it?

def obalujuci_dekorator(param_fct):
    def inner():
        before_call()
        result = param_fct()
        after_call()
        return result
    return inner


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Parametrizovany dekorator
Can you write Python code for it?

def vonkajsi_decorator(argument):
    def decorator(param_funct):
        def fct_wrapper(*args, **kwargs):
            before_stuff()
            something_with_argument(argument)
            funct(*args, **kwargs)
            after_stuff()
        return fct_wrapper
     return decorator

@vonkajsi_dekorator(parameter)
def funkcia():
    pass


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Pomocou Closure sa da vytvorit napriklad aj jednoducha trieda
Can you write Python code for it?

class A:
    def __init__(self, x):
        self._x = x

    def incr(self):
        self._x += 1
        return self._x

obj = A(0)

obj.incr()


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Pomocou closure napriklad takto
Can you write Python code for it?

def A(x):
    def incr():
        nonlocal x
        x +=1
        return x
    return incr

obj = A(0)

obj()


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Potrebujeme nejaku strukturu, kde si budeme ukladat priebezne vysledky

Napriklad slovnik
Can you write Python code for it?

def memoize(f):
    memo = {}
    counter = 0
    def wrapper(x):
        if x not in memo:
            nonlocal counter # toto by tu nemuselo byt, ale ja chcem vediet kolko som si usetril volani
            memo[x] = f(x)
            counter += 1 # toto by tu nemuselo byt
        return memo[x]
    return wrapper

@memoize
def memoized_fib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return memoized_fib(n-1) + memoized_fib(n-2)

memoized_fib(10)
print(memoized_fib.__closure__[0].cell_contents)

@counter_decorator
def counted_fib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return counted_fib(n-1) + counted_fib(n-2)

counted_fib(10)
print(counted_fib.__closure__[0].cell_contents)


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Pouzit to viem napriklad na vytvorenie dekoratora, ktory mi bude logovat volania funkcie a ja si zvolim uroven logovania
Can you write Python code for it?

def log(level, message):
    print("{0}: {1}".format(level, message))

def log_decorator(level):
    def decorator(f):
        def wrapper(*args, **kwargs):
            log(level, "Function {} started.".format(f.__name__))
            result = f(*args, **kwargs)
            log(level, "Function {} finished.".format(f.__name__))
            return result
        return wrapper
    return decorator

@log_decorator('debug')
def my_print(*args):
    print(*args)

my_print('ide to?')


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Registracny dekorator vykona nieco pri registracii funkcie

vykona nieco pri registracii funkcie v case importovania a nie vykonavania samotenej funkcie.
kludne si moze dekorator udrzovat nejaky stav pomocou lokalnych premennych
Can you write Python code for it?

def registracny_dekorator(param_func):
    when_registering_stuff()
    return param_func


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Sumarizujeme -  Rozne mozne formy dekoratorov
Nahradzajuci generator nahradi funkciu uplne niecim inym
Can you write Python code for it?

def nahradzujuci_dekorator(param_fct):
    def nieco_uplne_ine():
        pass
    return nieco_uplne_ine


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Takze co je to teda ten uzaver?
Funkcia, ktora pouziva neglobalnu premennu definovanu mimo svojho tela
Naco?

Specializovanie vseobecnej funkcie (Partial function application)
Udrziavanie stavu medzi volaniami funkcie

Ano, bude to spinava funkcia
Obcas sa ale uchovavaniu stavu nevyhneme
Ideme spravit funkcionalny sposob ako si uchovavat stav
Chcem ukazat ako sa daju funkcionalne crty pouzit na vylepsenie imperativneho kodu

Specializovanie vseobecnej funkcie (Partial function application)
Teraz len jeden priklad. Poriadne sa tomu budem venovat zajtra
Can you write Python code for it?

def make_power(a):
    def power(b):
        return b ** a
    return power

square = make_power(2)
square(3)


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Teraz si mozeme vyrobit napriklad uplne vseobecny dekorator, ktory bude pocitat pocty volani nejakej funkcie
Can you write Python code for it?

def counter_decorator(fct):
    counter = 0
    def wrapper(*args, **kwargs):
        nonlocal counter
        counter += 1
        return fct(*args, **kwargs)
    return wrapper

@counter_decorator
def counted_fib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return counted_fib(n-1) + counted_fib(n-2)

counted_fib(10)
print(counted_fib.__closure__[0].cell_contents)


I am working on the file "Closure a dekorator.ipynb".
The first task is:
Vnutorna funkcia nie je z vonka dostupna. Existuje len v ramci tela vonkajsej funkcie. Teda pokial ju nevratime ako navratovu hodnotu.
Can you write Python code for it?

def outer():
    def inner(a):
        return a + 7
    return inner

inner(3) # zvonka funkcia nieje dostupna a je teda chranena (nikto k nej nemoze a nezaspini nam priestor mien)

pom = outer()
pom


I am working on the file "Closure a dekorator.ipynb".
The first task is:
dalo by sa to este vylepsit tak, aby som mal praktickejsi pristup k tomu pocitadlu, ale nateraz mi to staci
Pocita sa celkovy pocet volani funkcie
Can you write Python code for it?


I am working on the file "Closure.ipynb".
The first task is:
In the example above the print_msg() function was called with the string "Hello" and the returned function was bound to the name another. On calling another(), the message was still remembered although we have already finished executing the print_msg() function.
This technique by which some data ("Hello") get attached to the code is called closure in python.
This value in the enclosing scope is remembered even when the variable goes out of scope or the function itself is removed from the current namespace.
Can you write Python code for it?


I am working on the file "Closure.ipynb".
The first task is:
We can see that the nested function printer() was able to access the non-local variable msg of the enclosing function.
In the example above, what would happen if the last line of the function print_msg() returned the printer() function instead of calling it? This means the function was defined as follows:
Can you write Python code for it?

def print_msg(msg):     # This is the outer enclosing function
    def printer():      # This is the nested function
        print(msg)
    return printer      # This is changed from the above example

another = print_msg("Hello")
another()


I am working on the file "Cloud Chamber Control.ipynb".
The first task is:
Here is an example of a function to add the cloud incrementally into the chamber
Can you write Python code for it?

def pattern_add(on_time,off_time,rounds):
    """pattern add controls the airbrush in a pattern
    it takes the parameters of the on_time, the off_time,
    and how many times that it should cycle between the two"""
        for i in range(rounds):
            for j in range(off_time):
                arduino.write(bytes([WAIT])

            arduino.write(bytes([AIRBRUSH_ON]))
            for m in range(on_time):
                arduino.write(bytes([WAIT])
            arduino.write(bytes([AIRBRUSH_OFF]))

pattern_add(3,2,5)
#this should cycle through a pattern w/ 3 seconds on, 2 seconds off, for 5 rounds

#always close serial connection when finished
arduino.close()

## TODO
# Add a way for the program to take brightness data and use that information
#to determine how long of a pulse is needed to bring the cloud to a stable state


I am working on the file "Cloud Chamber Test Analysis.ipynb".
The first task is:
Initial Data Reading
Can you write Python code for it?

#SG Here is the cloud chamber data collected of brightness vs. time when the brightness of a laser
#incident on the cloud chamber is without any added mist for 5 seconds, then 5 seconds of mist.
plt.plot(cloud_data[1:,0],cloud_data[1:,1])
plt.plot(cloud_data[1:,0],cloud_data[1:,3])
plt.plot(cloud_data[1:,0],cloud_data[1:,5])
plt.plot(cloud_data[1:,0],cloud_data[1:,7])
plt.plot(cloud_data[1:,0],cloud_data[1:,9])
plt.plot(cloud_data[1:,0],cloud_data[1:,11])
plt.xlabel('time(seconds)')
plt.ylabel('brightness(lux)')


I am working on the file "Cloud Chamber Test Analysis.ipynb".
The first task is:
Linearize the function
Can you write Python code for it?

#assuming that it is in the form returning_brightness= a*e^(bx)

plt.plot(cloud_data[1000:2200,0],[np.log(x) for x in Trial_1])
plt.xlabel('time(seconds)')
plt.ylabel('natural log of brightness(lux)')

plt.plot(cloud_data[1000:2200,0],[np.log(x) for x in Trial_2])
plt.xlabel('time(seconds)')
plt.ylabel('natural log of brightness(lux)')

plt.plot(cloud_data[1000:2100,0],[np.log(x) for x in Trial_3])
plt.xlabel('time(seconds)')
plt.ylabel('natural log of brightness(lux)')

plt.plot(cloud_data[1000:2000,0],[np.log(x) for x in Trial_4])
plt.xlabel('time(seconds)')
plt.ylabel('natural log of brightness(lux)')

plt.plot(cloud_data[1000:2000,0],[np.log(x) for x in Trial_5[]])
plt.xlabel('time(seconds)')
plt.ylabel('natural log of brightness(lux)')

plt.plot(cloud_data[1000:2200,0],[np.log(x) for x in Trial_6])
plt.xlabel('time(seconds)')
plt.ylabel('natural log of brightness(lux)')


I am working on the file "Cloud Chamber Test Analysis.ipynb".
The first task is:
Okay so this appears to be linear, but we will test the correlation coefficient to test that
Can you write Python code for it?

# The values needed to be sliced further, to not have the turbulent sections interfering with the
# correlation coefficent

start=1650
stop=2000
#slicing different values for areas of interest

plt.plot(cloud_data[start:stop,0],[np.log(x) for x in cloud_data[start:stop,1]])
plt.xlabel('time(seconds)')
plt.ylabel('brightness(lux)')

R_1= np.corrcoef(cloud_data[1650:2000,0],[np.log(x) for x in cloud_data[1650:2000,11]])[0,1]

R_2 = np.corrcoef(cloud_data[1550:2000,0],[np.log(x) for x in cloud_data[1550:2000,3]])[0,1]

R_3 = np.corrcoef(cloud_data[1450:2000,0],[np.log(x) for x in cloud_data[1450:2000,5]])[0,1]
R_4 = np.corrcoef(cloud_data[1450:2000,0],[np.log(x) for x in cloud_data[1450:2000,7]])[0,1]
R_5 = np.corrcoef(cloud_data[1450:2000,0],[np.log(x) for x in cloud_data[1450:2000,9]])[0,1]
R_6= np.corrcoef(cloud_data[1450:2000,0],[np.log(x) for x in cloud_data[1450:2000,11]])[0,1]

#all high 0.99+ R values, support that this is a exponential function.

start=1650
stop=2000
#slicing different values for areas of interest

for i in range(12):
    if i%2 != 0:
        plt.plot(cloud_data[start:stop,0],[np.log(x) for x in cloud_data[start:stop,i]])
plt.xlabel('time(seconds)')
plt.ylabel('natural log of brightness(lux)')


I am working on the file "Cloud Chamber Test Analysis.ipynb".
The first task is:
Slicing Sections of Interest
Can you write Python code for it?

start=1000
stop=2000
#slicing different values for areas of interest

plt.plot(cloud_data[start:stop,0],cloud_data[start:stop,1])
plt.xlabel('time(seconds)')
plt.ylabel('brightness(lux)')
#a visual of what is the area of interest

#hardwiring once that area is found
Trial_1=cloud_data[1000:2200,1]
Trial_2=cloud_data[1000:2200,3]
Trial_3=cloud_data[1000:2100,5]
Trial_4=cloud_data[1000:2000,7]
Trial_5=cloud_data[1000:2000,9]
Trial_6=cloud_data[1000:2200,11]


I am working on the file "Cluster experiment 1.ipynb".
The first task is:
Attempting K-Means
What sort of clustering algorithm to employ is actually a good question. K-means can give fairly meaningless responses if the data is of a given sort. Generally, it can be useful but cannot be used blindly.
Given the data above, it might be a good start however.
Can you write Python code for it?

# Build the clustering and show the individual clusters as best we can:
from sklearn.cluster import KMeans

estimator = KMeans(n_clusters=8)

clusters = estimator.fit(marray)
labels = estimator.labels_

def isol(label, labels):
    for l in labels.astype(np.float):
        if l != label:
            yield "#444444"
        else:
            yield "#FF3355"

#for label in set(labels):
#    print(label)
#    plt.scatter(marray[:,0], marray[:,1], c=list(isol(label, labels)))
#    plt.show()


I am working on the file "Cluster experiment 1.ipynb".
The first task is:
What do these 'vectors' look like? What do the columns refer to?
Can you write Python code for it?

data, transform, id_list = ds
print(data.toarray())

print(transform.get_feature_names())


I am working on the file "Cluster experiment 1.ipynb".
The first task is:
x1 vs x2 varience?
What is the rough shape of this data? The varience of x1 and x2 are equivalent to the left and right alignment of the text varies in a given block of text.
Can you write Python code for it?