I am working on the file "Class02/Class02.ipynb".
The first task is:
I've put all the steps together in one cell and commented on each step.
Can you write Python code for it?

# Step 1: Split off the test data
dia_train, dia_test = train_test_split(diabetes, test_size=0.2, random_state=23)

# Step 2: Create linear regression object
dia_model = LinearRegression()

# Step 3: Train the model using the training sets
features = dia_train[['BMI']].values
labels = dia_train['Target'].values

# Step 4: Fit the model
dia_model.fit(features,labels)

# Step 5: Get the predictions
testinputs = dia_test[['BMI']].values
predictions = dia_model.predict(testinputs)
actuals = dia_test['Target'].values

# Step 6: Plot the results
plt.scatter(testinputs, actuals, color='black', label='Actual')
plt.plot(testinputs, predictions, color='blue', linewidth=1, label='Prediction')
plt.xlabel('BMI') # Label the x axis
plt.ylabel('Target') # Label the y axis
plt.legend(loc='upper left', shadow=False, scatterpoints=1)

# Step 7: Get the RMS value
print("RMS Error: {0:.3f}".format( np.sqrt(np.mean((predictions - actuals) ** 2))))


I am working on the file "Class02/Class02.ipynb".
The first task is:
It is pretty clear that there is a linear trend here. If I wanted to predict what would happen if we tried the input of x=0.6, it would be a good guess to pick something like y=1.6 or so. Training the computer to do this is what we mean by Machine Learning.
To formalize this a little bit, it consists of four steps:

We start with relevant historical data. This is our input to the machine learning algorithm.
Choose an algorithm. There are a number of possibilities that we will cover over the course of the semester.
Train the model. This is where the computer learns the pattern.
Test the model. We now have to check to see how well the model works.

We then refine the model and repeat the process until we are happy with the results.
The Testing Problem
There is a bit of a sticky point here. If we use our data to train the computer, what do we use to test the model to see how good it is? If we use the same data to test the model we will, most likely, get fantastic results! After all, we used that data to train the model, so it should (if the model worked at all) do a great job of predicting the results.
However, this doesn't tell us anything about how well the model will work with a new data point. Even if we get a new data point, we won't necessarily know what it is supposed to be, so we won't know how well the model is working. There is a way around all of this that works reasonably well. What we will do is set aside a part of our historical data as "test" data. We won't use that data to train the model. Instead, we will use it to test the model to see how well it works. This gives us a good idea of how the model will work with new data points. As a rule of thumb, we want to reserve about 20% of our data set as testing data.
There is a library that does this for us in Python called train_test_split. The documentation is here: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html. I want you to get used to looking up the documentation yourself to see how the function...
Can you write Python code for it?

from sklearn.model_selection import train_test_split

faketrain1, faketest1 = train_test_split(fakedata1, test_size=0.2, random_state=23)
faketrain1.plot(x='input',y='output',kind='scatter')
faketest1.plot(x='input',y='output',kind='scatter')


I am working on the file "Class02/Class02.ipynb".
The first task is:
Not too surprising that the RMS error isn't very good. This is the real world after all. However, we saw in Class 01 that there may be some dependence on some of the other variables like the LDL. We can try a linear regression with both of them as inputs. I have to change the code a little to do this. Compare this with the previous cell to see what needs to change.
Can you write Python code for it?

# Step 2: Create linear regression object
dia_model2 = LinearRegression()

# Possible columns:
# 'Age', 'Sex', 'BMI', 'BP', 'TC', 'LDL', 'HDL', 'TCH', 'LTG', 'GLU'
#
inputcolumns = [ 'BMI', 'HDL']

# Step 3: Train the model using the training sets
features = dia_train[inputcolumns].values
labels = dia_train['Target'].values

# Step 4: Fit the model
dia_model2.fit(features,labels)

# Step 5: Get the predictions
testinputs = dia_test[inputcolumns].values
predictions = dia_model2.predict(testinputs)
actuals = dia_test['Target'].values

# Step 6: Plot the results
#
# Note the change here in how we plot the test inputs. We can only plot one variable, so we choose the first.
# Also, it no longer makes sense to plot the fit points as lines. They have more than one input, so we only visualize them as points.
#

plt.scatter(testinputs[:,0], actuals, color='black', label='Actual')
plt.scatter(testinputs[:,0], predictions, color='blue', label='Prediction')
plt.legend(loc='upper left', shadow=False, scatterpoints=1)

# Step 7: Get the RMS value
print("RMS Error: {0:.3f}".format( np.sqrt(np.mean((predictions - actuals) ** 2))))


I am working on the file "Class02/Class02.ipynb".
The first task is:
This looks pretty good. We can go one step futher and define a quantitative measure of the quality of the fit. We will subtract the difference between the prediction and the actual value for each point. We then square all of those and average them.  Finally we take the square root of all of that. This is known as the RMS error (for Root Mean Squared).
Can you write Python code for it?

import numpy as np
print("RMS Error: {0:.3f}".format( np.sqrt(np.mean((predictions - actuals) ** 2))))


I am working on the file "Class02/Class02.ipynb".
The first task is:
You can see that, with a 20% split, our small fake dataset doesn't have very many points. Really we shouldn't be working with less than 100 points for anything we do. Any fewer than that and the statistics just start breaking. Ideally we'd have tens of millions of data points. We'll talk later about how to get that much data, but we'll start small for now. We'll load in the Class02_fakedata2.csv file and split it 80/20 training/testing datasets.
Can you write Python code for it?

fakedata2 = pd.read_csv('Class02_fakedata2.csv')
faketrain2, faketest2 = train_test_split(fakedata2, test_size=0.2, random_state=23)
faketrain2.plot(x='input',y='output',kind='scatter')
faketest2.plot(x='input',y='output',kind='scatter')


I am working on the file "Class03/Class03.ipynb".
The first task is:
Another useful function is adding up columns. Note that we need to tell pandas to run through each row by adding the argument axis=1 to the apply function. Otherwise it tries to add up each column. This might be something you might want to do, too, though the easiest way to do that is to use the pandas sum function for the column.
Can you write Python code for it?

sampledata['IntSUM'] = sampledata[['IntCol','FloatCol']].apply(np.sum,axis=1)
print(sampledata[['IntCol','FloatCol','IntSUM']])

sampledata['IntCol'].sum()


I am working on the file "Class03/Class03.ipynb".
The first task is:
Because this is "fake" data, I put in a functional dependence here. The float column looks like it is some function of the integer column. It is almost always a good idea to visualize your data early on to see what it looks like graphically!
Text
Pandas can store text in its columns. Because there are a number of different types of text objects, by default pandas will store text as an object which just means it doesn't know which of the types it really is. Text can, in principle, be anything you want it to be, so it is both the most flexible and the most challenging data type.
Can you write Python code for it?


I am working on the file "Class03/Class03.ipynb".
The first task is:
Before we move on, I'd like to take a quick look at the data graphically.
Can you write Python code for it?


I am working on the file "Class03/Class03.ipynb".
The first task is:
Categorical
A categorical data type is a finite set of different objects. These objects are represented internally as integers but may be displayed as text or other generic objects. To make things simple, we'll start with a categorical object that has three possible values: "yes", "no", and "maybe". Internally, pandas will represent these as integers 0,1, and 2. But it knows that this is a categorical data type, so it keeps track of the text value associated with the integer and displays that for the user.
Can you write Python code for it?


I am working on the file "Class03/Class03.ipynb".
The first task is:
Common functions
There are a number of common functions that we could use inside the apply. For example, if we wanted to get the square root of each entry, this is what it would look like. We are using the function np.sqrt from the numpy library. We already imported this library, but if we didn't, we'd need to import numpy as np before running this function.
Can you write Python code for it?

sampledata['FloatSQRT'] = sampledata['FloatCol'].apply(np.sqrt)
print(sampledata[['FloatCol','FloatSQRT']])


I am working on the file "Class03/Class03.ipynb".
The first task is:
Converting to categorical
We already saw how to convert text columns to categorical columns. We can also covert other data types to categorical columns. For example, we could bin a float column into regularly sized bins, then create a categorical column from those bins.
Word/Text cleaning
Finally, it is often useful to clean up text entries before trying to turn them into features. For example, we may want to remove all punctuation, capital letters, or other special characters. We may also want to consider all of the forms of a word as the same word. For example, we may want to have both "dog" and "dogs" as the same feature. Or we may want "wonder" and "wonderful" as the same feature. There are a couple of text processing tools in python that simplify this work considerably.
I created a small dataset to work with. We'll use one of the rows to test our text cleaning process.
Can you write Python code for it?

textDF = pd.read_csv('Class03_text.tsv',sep='\t')
testcase = textDF['review'][3]
testcase


I am working on the file "Class03/Class03.ipynb".
The first task is:
Data Cleaning Example In-class Activity
The tutorial on cleaning messy data is located here: http://nbviewer.jupyter.org/github/jvns/pandas-cookbook/blob/v0.1/cookbook/Chapter%207%20-%20Cleaning%20up%20messy%20data.ipynb
Follow the tutorial, looking at the data and how to do a preliminary clean to eliminate entries that aren't correct or don't help. The data file can be loaded from the SageMath folder. I've reduced the number of column features in the data set to make it a bit easier to work with.
Can you write Python code for it?


I am working on the file "Class03/Class03.ipynb".
The first task is:
Feature extraction
We can often pull additional features from what we currently have. This involves doing a column-wise processing step, but with the additional component of doing a transformation or extraction from the data. We'll look at a couple of techniques to do this.
Date/day/week features
We already saw how to take a text column that is a date and turn it into a datetime data type. The to_datetime() function has the capability of parsing many different string formats. I recommend looking at the documentation for the function to learn how to do parsing of more specific date time formats.
Once we have a datetime data type, we can use other functions to get, for example, the day of the week or the week of the year for any given date. This may be useful for looking at weekly patterns or yearly patterns. The full list of features we can easily extract is found in the documentation. We use the apply function with the simple in-line lambda function to get the date or time features. Another use for this might be to identify holidays- for example, Memorial day is always on the same relative day of the year (last Monday in May). We could use these functions to identify which days are national or bank holidays.
Can you write Python code for it?

# Get the day of the week for each of the data features. We can get either a numerical value (0-6) or the names
sampledata['DayofWeek'] = sampledata['DateCol2'].apply(lambda x: x.weekday_name)
# Or the week number in the year
sampledata['WeekofYear'] = sampledata['DateCol2'].apply(lambda x: x.week)

print(sampledata[['DayofWeek', 'WeekofYear']])


I am working on the file "Class03/Class03.ipynb".
The first task is:
It is also useful to plot the geographical data. There are python libraries that make this easy to do.
Can you write Python code for it?

from mpl_toolkits.basemap import Basemap
import numpy as np

# Draw the base map of the world
m = Basemap(projection='robin',lon_0=0,resolution='c')
# Draw the continent coast lines
m.drawcoastlines()
# Color in the water and the land masses
m.fillcontinents(color='red',lake_color='aqua')
# draw parallels and meridians.
m.drawparallels(np.arange(-90.,120.,30.))
m.drawmeridians(np.arange(0.,360.,60.))
#m.drawmapboundary(fill_color='aqua')

# Prep the data for plotting on the map
x,y = m(sampledata['LonCol'].values, sampledata['LatCol'].values)
# Plot the data points on the map
m.plot(x,y, 'bo', markersize=10)


I am working on the file "Class03/Class03.ipynb".
The first task is:
Now that we have the datetime column, I'd like to plot the data as a function of date. This is often a useful thing to do with time series data. We'll need to import the matplotlib library and use a trick to format the data by date. Here's the code that makes it work.
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline
# We will plot the data values and set the linestyle to 'None' which will not plot the line. We also want to show the individual data points, so we set the marker.
plt.plot(sampledata['DateCol2'].values, sampledata['FloatCol'].values, linestyle='None', marker='o')
# autofmt_xdate() tells the computer that it should treat the x-values as dates and format them appropriately. This is a figure function, so we use gcf() to "get current figure"
plt.gcf().autofmt_xdate()


I am working on the file "Class03/Class03.ipynb".
The first task is:
Text regex features
Another type of text feature extraction using a regex or regular expression pattern recognition code. The date/time conversion uses one form of this, but we can be more general in identifying patterns. There are some very useful tools for testing your pattern. I like the tester at https://regex101.com/. I use it whenever I build a pattern recognition string.
Can you write Python code for it?

# This simple text pattern gathers all the letters up to (but not including) the last 'e' in the text entry. There are lots of other pattern recognition tools to extract features from text.
# Note that it returns "NaN" if there are no 'e's in the text string. We could use that to find all the strings without an 'e' in them.
sampledata['TextCol'].str.extract("(.*)e", expand=True)


I am working on the file "Class03/Class03.ipynb".
The first task is:
The first thing we notice is that there are hypertext bits in the text (the <br /> items). We want to clean all of those out. The BeautifulSoup function does this for us.
Can you write Python code for it?

from bs4 import BeautifulSoup
cleantext = BeautifulSoup(testcase,"html5lib").text
cleantext


I am working on the file "Class03/Class03.ipynb".
The first task is:
The next two steps we'll do at once because we need to split up the text into individual words to do them. The split() function breaks up the string into an array of words. We will then eliminate any words that are stopwords in English. These are words like "and", "or", "the" that don't communciate any information but are necessary for language.
The other thing we'll do is cut the words down to their root stems. This will get rid of plurals or other modifications of words.
Can you write Python code for it?

import nltk
from nltk.corpus import stopwords # Import the stop word list

words = lowercase.split()
meaningfulwords = [w for w in words if not w in stopwords.words("english")]

from nltk.stem import SnowballStemmer
snowball_stemmer = SnowballStemmer("english")

stemmedwords = [snowball_stemmer.stem(w) for w in meaningfulwords ]

print(" ".join(meaningfulwords))
print("\n")
print(" ".join(stemmedwords))

# Now we make a function that we can apply to every entry in the dataframe

def cleantext(textinput):

    # First Pass: remove any html tags
    from bs4 import BeautifulSoup
    cleantext = BeautifulSoup(textinput,"html5lib").text

    # Second pass: remove non-letters and make everything lower case
    import re
    testcase = re.sub("[^a-zA-Z]"," ",cleantext)
    lowercase = testcase.lower()

    # Third pass: remove all stop words (non-essential words)
    from nltk.corpus import stopwords # Import the stop word list
    words = lowercase.split()
    meaningfulwords = [w for w in words if not w in stopwords.words("english")]

    # Fourth pass: get the word stems so that plurals, etc. are reduced
    from nltk.stem import SnowballStemmer
    snowball_stemmer = SnowballStemmer("english")
    stemmedwords = [snowball_stemmer.stem(w) for w in meaningfulwords ]

    # Put the words back together again with a single space beteen them
    return " ".join(stemmedwords)

textDF['cleaned'] = textDF['review'].apply(cleantext)
textDF


I am working on the file "Class03/Class03.ipynb".
The first task is:
They are currently stored as objects, not as datetimes. We need to convert this column as well, but we'll use a special pandas function to do that. Take a quick look at the reference page for this function to see what else it can do. Note that the new column has type datetime64[ns]. That means that the date format is capable of counting nanoseconds. We won't use all of that capability, but pandas used that format because our dates are accurate to the second.
Can you write Python code for it?

sampledata["DateCol2"] = pd.to_datetime(sampledata["DateCol"])
sampledata.dtypes

#We print out the column to see what it looks like
sampledata["DateCol2"]


I am working on the file "Class03/Class03.ipynb".
The first task is:
Unique values
Sometimes it is helpful to know what unique values are in a column. Especially when there are many rows (millions), it is impractical to manually scan through the columns to look for unique values. However, we can use a pandas function unique() to do just that. We will see this is particularly helpful in doing data cleaning to identify rows with problems in the data.
Can you write Python code for it?


I am working on the file "Class03/Class03.ipynb".
The first task is:
We can also get a list of the categories that pandas found when converting the column. These are in order- the first entry corresponds to 0, the second to 1, etc.
Can you write Python code for it?


I am working on the file "Class03/Class03.ipynb".
The first task is:
We can now look at how the data are stored as categorical data. We can get thi internal codes for each of the entries like this:
Can you write Python code for it?


I am working on the file "Class03/Class03.ipynb".
The first task is:
We may encounter situations where we want to plot the data and visualize each category as its own color. We saw how to do this back in Class01.
Can you write Python code for it?

import seaborn as sns
sns.set_style('white')
sns.lmplot(x='IntCol', y='FloatCol', data=sampledata, hue='CatCol2', fit_reg=False)


I am working on the file "Class03/Class03_supplement.ipynb".
The first task is:
Ordered Categorical Values
The 'Rank' column are ranked categorical values where the ranking matters on a linear scale. So we can create a categorical column for these values right away. We are lucky here that the values are in alphabetical order - pandas can pick out that order and use it for us.
Can you write Python code for it?

sampledata['CatRank'] = sampledata['Rank'].astype('category')
print(sampledata["CatRank"].cat.categories)
sampledata["CatRank"][1:10].cat.codes


I am working on the file "Class03/Class03_supplement.ipynb".
The first task is:
Unordered Categorical Values
Let's now put the states into a categorical column. Even though Pandas will sort them, there is no real 'rank' for the states
Can you write Python code for it?

sampledata['CatState'] = sampledata['State'].astype('category')
print(sampledata["CatState"].cat.categories)
sampledata["CatState"][1:10].cat.codes


I am working on the file "Class03/Class03_supplement.ipynb".
The first task is:
We now want to join this back with the original set of features so that we can use it instead of the ranked column of data. Here's one way to do that.
Can you write Python code for it?

sampledata2 = sampledata.join(dummydf)
sampledata2.head()


I am working on the file "Class03/Class03_supplement.ipynb".
The first task is:
We now want to select out all 50 columns from the dummy variable. There is a python way to do this easily, since we used the prefix 'S_' for each of those columns
Can you write Python code for it?

inputcolumns = ['DaysSinceStart','RankCode'] + [col for col in sampledata2.columns if 'S_' in col]

train2, test2 = train_test_split(sampledata2, test_size=0.2, random_state=23)

# Step 1: Create linear regression object
regr2= LinearRegression()

features = train2[inputcolumns].values
labels = train2['Output'].values

regr2.fit(features,labels)

# Step 5: Get the predictions
testinputs = test2[inputcolumns].values
predictions = regr2.predict(testinputs)
actuals = test2['Output'].values

plt.scatter(testinputs[:,0], actuals, color='black', label='Actual')
plt.scatter(testinputs[:,0], predictions, color='blue', label='Prediction')
plt.legend(loc='upper left', shadow=False, scatterpoints=1)

# Step 7: Get the RMS value
print("RMS Error: {0:.3f}".format( np.sqrt(np.mean((predictions - actuals) ** 2))))


I am working on the file "Class04/Class04.ipynb".
The first task is:
Assignment
You assignment this week is to run through both the Perceptron and the Naïve Bayes classifiers with your classification data. Evaluate both models using each of the metrics we've learned about and compare the performance of the models.
If you find that the model fit is taking a long time, you should note that in your assignment as well. How long a model takes to learn is an important parameter. There is a simple way of timing the model performance. We'll run both models again and compare their timing. For the small number of data points we have in this dataset, the timing isn't very different. That may not be the case for your models.
Can you write Python code for it?

import time
# Perceptron Model
start1 = time.time()
model2.fit(features_train,labels_train)
stop1 = time.time()
print("Elapsed time: {} seconds".format(stop1-start1))

# Naïve Bayes model
start2 = time.time()
nb_model.fit(features_train,labels_train)
stop2 = time.time()
print("Elapsed time: {} seconds".format(stop2-start2))


I am working on the file "Class04/Class04.ipynb".
The first task is:
At this point, let's go back to the entire test dataset and fit the decision boundary for it. We'll also look at the out-of-sample performance by plotting the test data instead of the train data.
Can you write Python code for it?

train, test = train_test_split(speeddf, test_size=0.2, random_state=23)

model2 = Perceptron()

features_train = train[['Grade','Bumpiness']].values
labels_train = train['Speed'].values
features_test = test[['Grade','Bumpiness']].values
labels_test = test['Speed'].values

model2.fit(features_train,labels_train)

Zpred = pd.Series(model2.predict(np.c_[xx.ravel(), yy.ravel()]), dtype='category').cat.codes.values
Z = Zpred.reshape(xx.shape)

# First plot our points
lm = sns.lmplot(x='Grade', y='Bumpiness', data=test, hue='Speed', fit_reg=False)
sns.despine(ax=lm.ax, top=False, right=False)
plt.pcolormesh(xx, yy, Z, cmap= plt.cm.cool, alpha=0.1)


I am working on the file "Class04/Class04.ipynb".
The first task is:
How confident is the model of that prediction? Let's get the prediction proabilities for that point.
Can you write Python code for it?


I am working on the file "Class04/Class04.ipynb".
The first task is:
How do we decide where in this region to put the boundary? There are a couple of different algorithms that will do the job for us. We're not going to spend time describing how they work - you can look them up if you are interested in the mathematics. Instead, we'll look at how to apply them and look at how well they work.
Perceptron
The first algorithm is called the Perceptron (information on how it works is found on Wikipedia: https://en.wikipedia.org/wiki/Perceptron#Learning_algorithm). The documentation for the Scikit Learn Perceptron is found here. We'll use a syntax very similar to the pattern we used in Class02. First, we split the data into training and testing sets.
Can you write Python code for it?

from sklearn.model_selection import train_test_split

trainsub, testsub = train_test_split(speedsub, test_size=0.2, random_state=23)


I am working on the file "Class04/Class04.ipynb".
The first task is:
Note that the line isn't very good - remember that we only used a subset of the data to fit the decision boundary. But it still lies in the expected range.
There is another way we could plot this: we could split our figure into small boxes, then make a prediction for each box. We then plot all the decisions in two different colors, showing the prediction for each box. This gives us a more general tool for plotting not only linear boundaries, but any possible decision boundary.
Can you write Python code for it?

# Plot the decision boundary. For that, we will assign a color to each
# point in the mesh
x_min = 0.0; x_max = 1.0 # Mesh x size
y_min = 0.0; y_max = 1.0  # Mesh y size
h = .01  # step size in the mesh
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

# Now predict the results at each point and get the categorical values
Zpred = model.predict(np.c_[xx.ravel(), yy.ravel()])
Zseries = pd.Series(Zpred, dtype='category')
Zvalues = Zseries.cat.codes.values
Z = Zvalues.reshape(xx.shape)

# First plot our points
lm2 = sns.lmplot(x='Grade', y='Bumpiness', data=speedsub, hue='Speed', fit_reg=False)
sns.despine(ax=lm2.ax, top=False, right=False)

# Now add in the decision boundary
plt.pcolormesh(xx, yy, Z, cmap= plt.cm.cool, alpha=0.1)


I am working on the file "Class04/Class04.ipynb".
The first task is:
Now we import the model and train it, just like we did with the linear regression.
Can you write Python code for it?

from sklearn.linear_model import Perceptron

# Step 1: Create linear regression object
model = Perceptron()

# Step 2: Train the model using the training sets
features = trainsub[['Grade','Bumpiness']].values
labels = trainsub['Speed'].values

model.fit(features,labels)
print("Model Coefficients: {}".format(model.coef_))
print("Model Intercept: {}".format(model.intercept_))


I am working on the file "Class04/Class04.ipynb".
The first task is:
So we see that the model has a pretty high probabily of getting the label correct in both corners, but closer to the decision boundary the probability of each label approaches the midpoint of 50%.
Logloss Metric
We've got one more metric we can use for models that give us access to the prediction probabilities. This metric has the property that if all the points are correctly predicted, it will be 0.0. The closer to zero you are, the better the model is doing at predicting the correct outcomes. It is a class-independent metric and works for models with more than two classes, too.
Can you write Python code for it?

logloss = metrics.log_loss(labels_test, y_proba_nb)
print("Log loss: {}".format(logloss))


I am working on the file "Class04/Class04.ipynb".
The first task is:
So, there are a few things to note here. First, the Perceptron has given us a boundary that works fairly well. However, it isn't perfect. There are a few points that are labeled "fast" that will now be classified as "slow". It would be nice to have a way to quantify how well the classifier has performed. We'll look at a new set of tools to do that.
Evaluation Metrics
First, we review the evaluation metric we've already seen: the RMS value for the linear regression. Recall from Class 02 that we calculated this by taking our model prediction, subtracting the actual value, squaring the difference, then averaging over all points in the test set. Finally, we took the square root of this to get the RMS: "[Square]Root [of the] Mean-Squared". A perfect fit would give an RMS of 0.0 and larger RMS values mean that the fit is not performing as well.
There are more ways to evaluate the performance of a classifier model. They all start with the confusion matrix, so we'll start there.
The Confusion Matrix
The first thing we do is recognize that there are, for a binary, or two-state classifier, four possible outcomes when we evaluate each test point:
1. The prediction says "slow" and the actual label says "slow"
2. The prediction says "fast", but the actual label says "slow"
3. The prediction says "slow", but the actual label says "fast"
1. The prediction says "fast" and the actual label says "fast"
The first and last possibilies indicate that the prediction did a good job, but the other two mean there were problems. Let's make this into a table:
|           | | Predicted | Predicted|
|:--------: |:-----:|:-----:|:-----:|
|  | |  Slow | Fast |
|Actual |Slow | #1  | #2 |
|Actual | Fast | #3 | #4 | |
Now we need to count how many of each possibility there were using the test data. There is, naturally, a tool to do this for us.
Can you write Python code for it?

from sklearn.metrics import confusion_matrix
class_labels = ["slow", "fast"]
y_pred = model2.predict(features_test)
cnf_matrix = confusion_matrix(labels_test, y_pred,labels=class_labels)
print(cnf_matrix)


I am working on the file "Class04/Class04.ipynb".
The first task is:
So, we can see that, for this point, the model outputs a 68% chance that the point should be classified as "fast" and, therefore, a 32% chance that it is "slow". We can plot the confidence intervals for these points to show how the model is mapping input values to output values. (Note: there may be a pandas warning... it doesn't appear to affect the outcome, so don't worry about it.)
Can you write Python code for it?

#first, get all the predictions
y_proba_nb = nb_model.predict_proba(features_test)

test['fastprob'] = y_proba_nb[:,0]

cm = plt.cm.get_cmap('YlGn')
sc = plt.scatter(x=test['Grade'], y=test['Bumpiness'], c=test['fastprob'] , vmin=0, vmax=1, s=35, cmap=cm)
cbr = plt.colorbar(sc)
cbr.set_label('Probability of "fast"')
plt.xlabel('Grade')
plt.xlabel('Bumpiess')


I am working on the file "Class04/Class04.ipynb".
The first task is:
There are a couple of things to note here: first: the decision boundary is curved! However, it is a fairly simple curve in that it doesn't wiggle very much - it is a smooth arc. This is related to the class Learning Principle of Occam's Razor. A straight-line is the simplest possible decision boundary and, therefore, is valued highly from the perspective of keeping the model as simple as possible. A smooth curve is slightly more complicated, but still fairly simple. The question is: do we gain out-of-sample performance by adding in the complexity of making the decision boundary curve?
That brings us to the second point: the confusion matrix now shows us that we have mis-classified 17 points. We compare that to the Perceptron model where we mis-classified 20 points. So we've done a little bit better in terms of out-of-sample performance, which is good. Let's take a look at the other metrics to see how they compare.
Can you write Python code for it?

recall_score = metrics.recall_score(labels_test, y_pred_nb,labels=class_labels,average=None)
prec_score = metrics.precision_score(labels_test, y_pred_nb,labels=class_labels,average=None)
f1_score = metrics.f1_score(labels_test, y_pred_nb,labels=class_labels,average=None)

acc_score = metrics.accuracy_score(labels_test, y_pred_nb)
matt_score = metrics.matthews_corrcoef(labels_test, y_pred_nb)

print("Class-dependent Metrics")
print("Sensitivity/Recall Score: {}".format(recall_score))

print("Precision Score: {}".format(prec_score))
print("F1 Score: {}".format(f1_score))

print("\nClass-independent Metrics")
print("Accuracy Score: {}".format(acc_score))
print("Matthews Correlation Coefficient (MCC): {}".format(matt_score))


I am working on the file "Class04/Class04.ipynb".
The first task is:
We can also visualize this as a graphic, showing a shade of color for each of the different values. This is especially useful when we have more than two classes. Because we'll use this again, we define a function that takes the class labels and confusion matrix as inputs and creates the plot.
Can you write Python code for it?

def show_confusion_matrix(cnf_matrix, class_labels):
    plt.matshow(cnf_matrix,cmap=plt.cm.YlGn,alpha=0.7)
    ax = plt.gca()
    ax.set_xlabel('Predicted Label', fontsize=16)
    ax.set_xticks(range(0,len(class_labels)))
    ax.set_xticklabels(class_labels)
    ax.set_ylabel('Actual Label', fontsize=16, rotation=90)
    ax.set_yticks(range(0,len(class_labels)))
    ax.set_yticklabels(class_labels)
    ax.xaxis.set_label_position('top')
    ax.xaxis.tick_top()

    for row in range(len(cnf_matrix)):
        for col in range(len(cnf_matrix[row])):
            ax.text(col, row, cnf_matrix[row][col], va='center', ha='center', fontsize=16)

show_confusion_matrix(cnf_matrix,class_labels)


I am working on the file "Class04/Class04.ipynb".
The first task is:
We will start with a subset of this data to illustrate what we are trying to do here. We use the sample() function to get a small piece of the data (we use the random_state option to make sure we use the same set of data every time, otherwise the data will change).
Can you write Python code for it?

speedsub = speeddf.sample(16,random_state=55)
sns.lmplot(x='Grade', y='Bumpiness', data=speedsub, hue='Speed', fit_reg=False)
sns.despine(top=False, right=False)


I am working on the file "Class04/Class04.ipynb".
The first task is:
What we want to do is have the computer learn where the boundary lies between the fast data points and the slow data points. That way we can input in any grade and any bumpiness and the computer will tell us whether to go fast or slow. It looks like there is a region between the two sets of data where we could potentially put our boundary.
Can you write Python code for it?

lm = sns.lmplot(x='Grade', y='Bumpiness', data=speedsub, hue='Speed', fit_reg=False)
sns.despine(ax=lm.ax, top=False, right=False)

from matplotlib.patches import Polygon
from matplotlib.collections import PatchCollection
patches=[]
polygon = Polygon([[.92,0],[1,0],[1,.24],[0,.9],[0,.67]], True)
patches.append(polygon)
p = PatchCollection(patches, alpha=0.4)
lm.ax.add_collection(p)


I am working on the file "Class05/Class05.ipynb".
The first task is:
A Second dataset
Now we need another dataset to work with. Let's try pulling the Dow Jones Industrial average index for each day. We'll download the data from here. We will also use a nice tool in the read_csv() function: it can parse a date column and put it in the appropriate datetime format all in one step.
Can you write Python code for it?

djiadf = pd.read_csv('Class05_DJIA_data.csv',parse_dates=[0])
print(djiadf.dtypes)
djiadf.head()

# We'll rename the second column to something a little more descriptive
djiadf.rename(columns={'VALUE':'DowJonesAvg'},inplace=True)
#Let's check for problems:
djiadf.isnull().values.any()

# Looks like there are two problem rows. Let's identify which rows by indexing the dataframe on the isnull() data
djiadf[djiadf['DowJonesAvg'].isnull()]

# These were both holidays and shouldn't be included in the data set anyway! We can drop them in place, modifying our dataframe.
djiadf.dropna(inplace=True)
djiadf.isnull().values.any()


I am working on the file "Class05/Class05.ipynb".
The first task is:
And, of course, let's plot it to see what it looks like.
Can you write Python code for it?

# We will plot the data values and set the linestyle to 'None' which will not plot the line. We also want to show the individual data points, so we set the marker.
plt.plot(djiadf['DATE'].values, djiadf['DowJonesAvg'].values, linestyle='None', marker='o')
# autofmt_xdate() tells the computer that it should treat the x-values as dates and format them appropriately. This is a figure function, so we use gcf() to "get current figure"
plt.gcf().autofmt_xdate()


I am working on the file "Class05/Class05.ipynb".
The first task is:
Choosing the Merge Index
How do you choose the merge index? We used both the visit_number and the patient/doctor_id values. The key is to be able to merge on some common feature. This may be date (like we show below), but it could also be on zip code, a city name, the month number, financial quarter, or any other piece of information that you have common between your two datasets. You need to make sure that the merge indices have the same data type and are in the same format. For example, the computer doesn't know that 'DALLAS' and 'Dallas' are the same city. We can use the tools we've previously covered to clean up the data prior to attempting the merge.
Merging the Store and DJIA data
Since we want to keep all of the store data and merge in the DJIA data where we can, we are going to try a left join. We are also going to try joining on the date - they should match up reasonably well.
Can you write Python code for it?

storeDJ = pd.merge(topstoredf, djiadf, left_on='SalesDate',right_on='DATE',how='left')
storeDJ.head()


I am working on the file "Class05/Class05.ipynb".
The first task is:
How do we tack df_b on the end of df_a? This is useful if you have pieces of a dataset that you want to combine together. Note that pandas doesn't, by default, reset the index for the new dataframe. We'd have to do that afterwards if we want to. Notice that there are now NaN values in the table! The first dataframe didn't have the doctor_id column, so pandas filled in those values with NaN. Same for the second dataframe and the patient_id values. We could drop those columns if we wanted to.
Can you write Python code for it?


I am working on the file "Class05/Class05.ipynb".
The first task is:
If we want to get the first entry in this new dataframe, we have to address it by its index, which in this case is 2640 and not 0.
Can you write Python code for it?


I am working on the file "Class05/Class05.ipynb".
The first task is:
Joining Data
We now want to join the data from the store and the Dow Jones Industrial average. This will let us plot the two together to see if there is any correlation between the DJIA and the purchases that the store is making.
In order to join two dataframes we need to know how to align the rows or how to match up the data. The easiest way to do this is to try it out with an example to see how it works. We are going to follow this tutorial on merging data in pandas. We start by creating two small test dataframes.
Can you write Python code for it?

raw_data = {
        'patient_id': ['A', 'B', 'C', 'D', 'E'],
        'first_name': ['Alex', 'Amy', 'Allen', 'Alice', 'Ayoung'],
        'last_name': ['Anderson', 'Ackerman', 'Ali', 'Aoni', 'Atiches'],
        'visit_number': [1,2,3,4,5],}
df_a = pd.DataFrame(raw_data, columns = ['patient_id', 'first_name', 'last_name','visit_number'])
df_a

raw_data = {
        'doctor_id': ['G', 'H', 'I', 'A', 'B'],
        'first_name': ['Billy', 'Brian', 'Bran', 'Bryce', 'Betty'],
        'last_name': ['Bonder', 'Black', 'Balwner', 'Brice', 'Btisan'],
        'visit_number': [4,5,6,7,8]}
df_b = pd.DataFrame(raw_data, columns = ['doctor_id', 'first_name', 'last_name','visit_number'])
df_b


I am working on the file "Class05/Class05.ipynb".
The first task is:
Not surprisingly, we can do the right-side, too.
Can you write Python code for it?


I am working on the file "Class05/Class05.ipynb".
The first task is:
Now we can do what we wanted to do: plot the data to see if there is a correlation here!
Can you write Python code for it?


I am working on the file "Class05/Class05.ipynb".
The first task is:
Now we get the first entry by addressing the dataframe at location 0.
Can you write Python code for it?


I am working on the file "Class05/Class05.ipynb".
The first task is:
Resetting the index
We haven't really paid much attention to the pandas index column up to this point, but now we'll take a look at it. The far left column of our new dataframe contains the indices for the dataframe- note that they are neither ordered nor sequential. After subsetting and re-ordering the dataset, they are a mess! We can fix this by resetting the index of the dataframe. This is helpful when we want to move forward with a subset of data. We'll pass two parameters to the reset_index() function. The first tells pandas to change our dataframe (instead of creating a copy), the second tells pandas that we don't care about the old index and so we should drop it.
Can you write Python code for it?


I am working on the file "Class05/Class05.ipynb".
The first task is:
So we now have a much smaller dataframe that only has the rows where visit_number was the same in both. Note that pandas also has changed the column names. This is because there were duplicate names and it can't have that to maintain unique addressing. Look at the documentation for pandas to see how to change the suffixes that pandas appends.
What if, instead, we wanted to merge rows where the patient_id from one matches the doctor_id from the other? We can do that, too. The two dataframes are given the designations left and right for the first and second dataframes, respectively. So we need to identify the columns in each to match.
Can you write Python code for it?


I am working on the file "Class05/Class05.ipynb".
The first task is:
Sorting Data
We want to see the stores with the largest purchases. To do this, we will sort the SalesToStore column in descending order (which puts the largest values at the top of the list). We use a pandas function sort_values() that takes the column name that you want to sort by as an argument. We also add the argument ascending=False to tell pandas to give us the results in descending order.
Can you write Python code for it?

storesum = storegroups.sum()

# Let's sort this list by the sales number in descending order:
storelist = storesum.sort_values('SalesToStore',ascending=False)

# And list the top stores:
storelist.head(10)


I am working on the file "Class05/Class05.ipynb".
The first task is:
There is a lot of duplicated information here. We only really care about the SalesDate and the SalesToStore column. So we'll drop the other columns in this dataframe and assign it the same name (effectively dropping the columns).
Can you write Python code for it?

topstoredf = topstoredf.drop(['Date', 'Store Number', 'City', 'Zip Code', 'County', 'Total Sales to Store (Dollars)'], axis=1)


I am working on the file "Class05/Class05.ipynb".
The first task is:
We now want the sum of all of the SalesToStore values in each group. We can do this easily since it is the only numerical value in the group. We apply the sum() function of the group to get the sum. There are a handful of other built-in functions. You can get a list of the functions by pressing the <TAB> key after typing storegroups. Try it to see the list of functions. Documentation for these functions is provided on the DataFrame document page.
Can you write Python code for it?


I am working on the file "Class05/Class05.ipynb".
The first task is:
We only lose 25 rows out of 13,000. I'm going to go with that- it simplifies further computations.
Grouping and subsetting data
We now want to narrow down the data in order to make it easier to work with joining datasets. Let's get the top-purchasing store and use just that store for our further work. We first need to do an analysis step: we want to add up all the sales for each store. In other words, we want to group by store, then sum up the StoreSales.
The first step is to use the groupby() function to group the data into sections that all have something in common. For our case, we will group by the "Store Number". Pandas will go through that column, find all the unique values, then group them together and return a groupby object. We'll look at the first group to see what is in it.
Can you write Python code for it?

storegroups = iowadfcl.groupby('Store Number')


I am working on the file "Class05/Class05.ipynb".
The first task is:
What if we want to join together the rows that have the same visit_number? Looking at the table, we see that Billy Bonder and Alice Aoni have the same visit_number. Let's merge the two dataframes together to see how that works.
Can you write Python code for it?


I am working on the file "Class05/Class05.ipynb".
The first task is:
What if we want to keep all of the entries in the left dataframe and only add in values on the right where they match? We can do that, too!
Can you write Python code for it?


I am working on the file "Class06/Class06.ipynb".
The first task is:
As a reminder, let's review the linear regression along with its graph.
Can you write Python code for it?

from sklearn.linear_model import LinearRegression
regr = LinearRegression()
features = faketrain2[['input']].values
labels = faketrain2['output'].values
regr.fit(features,labels)

testinputs = faketest2[['input']].values
predictions = regr.predict(testinputs)
actuals = faketest2['output'].values
print("RMS Error: {0:.3f}".format( np.sqrt(np.mean((predictions - actuals) ** 2))))

# Prepare the model line
X_plot =np.linspace(0, 1, 1000)
Y_pred = regr.predict(X_plot[:,None])

trainfig, ax = plt.subplots()

# First plot our points
ax.scatter(x=testinputs, y=actuals)

ax.plot(X_plot,Y_pred,c='r')


I am working on the file "Class06/Class06.ipynb".
The first task is:
Ok, we now try the support vector regression model.
Can you write Python code for it?

from sklearn.svm import SVR
svrmodel = SVR(C=1.0,gamma=1.0)

svrmodel.fit(features,labels)

svrpredictions = svrmodel.predict(testinputs)

print("RMS Error: {0:.3f}".format( np.sqrt(np.mean((svrpredictions - actuals) ** 2))))

# Prepare the model line
Y_pred = svrmodel.predict(X_plot[:,None])

trainfig, ax = plt.subplots()
# First plot our points
ax.scatter(x=testinputs, y=actuals)
ax.plot(X_plot,Y_pred,c='r')


I am working on the file "Class06/Class06.ipynb".
The first task is:
Reducing the penalty looks like it added a bit of curvature, but it increased the number of support points and lowered the MCC. Let's try the other way.
Programming Aside Note:
Take a quick look at the previous two blocks of code: they were identical except for a couple of parameters at the top. This is a good indication that we should write our own function to simplify running more examples. Python makes that really easy: we add one line at the top, change the parameter inputs to the model, and indent the rest of the code.
Can you write Python code for it?

def testsvcmodel(inputC,inputgamma):
    svcmodelf = SVC(kernel='rbf',C=inputC, gamma=inputgamma)
    svcmodelf.fit(features_train,labels_train)
    y_pred = svcmodelf.predict(features_test)
    # Prepare the boundary
    Z = pd.Series(svcmodelf.predict(np.c_[xx.ravel(), yy.ravel()]), dtype='category').cat.codes.values.reshape(xx.shape)
    support_points = features_train[svcmodelf.support_]

    figf = plt.figure()
    ax1 = figf.add_subplot(121)
    cnf_matrix = confusion_matrix(labels_test, y_pred,labels=class_labels)
    show_confusion_matrix(cnf_matrix,class_labels,ax=ax1)
    matt_score = metrics.matthews_corrcoef(labels_test, y_pred)
    print("Matthews Correlation Coefficient (MCC): {}".format(matt_score))

    ax2 = figf.add_subplot(122)
    # First plot our points
    ax2.pcolormesh(xx, yy, Z, cmap= plt.cm.cool, alpha=0.1)
    ax2.set_title("N support points: {}".format(len(support_points)))
    ax2.set_aspect(1)
    ax2.set_xlabel('Grade')
    ax2.set_ylabel('Bumpiness')
    # Circle the support points

    ax2.scatter(x=support_points[:,0],y=support_points[:,1],s=10, facecolors='none', edgecolors='r')

# Now we just need to call the function!
testsvcmodel(inputC=100,inputgamma=1)


I am working on the file "Class06/Class06.ipynb".
The first task is:
So we've improved performance again! But this time, look at the support points and the decision boundary: they are both a mess! This is a complicated model that is trying hard to fit a very particular shape that doesn't look like it is really our dataset. In other words, it looks like it is trying to fit to the noise in the data as opposed to fitting the underlying model. This is what we mean by overfitting and it is going the wrong way! This type of model will most likely perform poorly when we head out to try it in the "real world".
We'll try the other combinations now.
Can you write Python code for it?

testsvcmodel(inputC=1,inputgamma=50)

testsvcmodel(inputC=0.1,inputgamma=50)

testsvcmodel(inputC=100,inputgamma=0.1)

testsvcmodel(inputC=0.1,inputgamma=0.1)


I am working on the file "Class06/Class06.ipynb".
The first task is:
So we've reduced the number of support points and increased the performance of the model! The boundary is no longer a straight line, but it seems to do better than the straight line did. This is an example of reducing complexity and improving performance. That's the direction we want to go! Let's try the other hyperparameter: $\gamma$.
Can you write Python code for it?


I am working on the file "Class06/Class06.ipynb".
The first task is:
That performance was worse than the linear model (an RMS error of 0.0 means a perfect fit). Let's try adjusting the hyperparameters. We'll make a function to make it easy to repeat this.
Can you write Python code for it?

def svrtest(inputC, inputgamma):
    svrmodel = SVR(C=inputC,gamma=inputgamma)
    svrmodel.fit(features,labels)
    svrpredictions = svrmodel.predict(testinputs)

    # Prepare the model line
    Y_pred = svrmodel.predict(X_plot[:,None])
    trainfig, ax = plt.subplots()
    ax.set_title("RMS Error: {0:.3f}".format( np.sqrt(np.mean((svrpredictions - actuals) ** 2))))
    # First plot our points
    ax.scatter(x=testinputs, y=actuals)
    ax.plot(X_plot,Y_pred,c='r')
svrtest(100,1)

svrtest(0.1,1)

svrtest(1,100)


I am working on the file "Class06/Class06.ipynb".
The first task is:
This looks a lot like the linear boundary! We'll try changing the values to see what they do.
Can you write Python code for it?

svcmodel5 = SVC(kernel='rbf',C=0.1, gamma=1)
svcmodel5.fit(features_train,labels_train)
y_pred = svcmodel5.predict(features_test)
# Prepare the boundary
Z = pd.Series(svcmodel5.predict(np.c_[xx.ravel(), yy.ravel()]), dtype='category').cat.codes.values.reshape(xx.shape)
support_points = features_train[svcmodel5.support_]

fig5 = plt.figure()
ax1 = fig5.add_subplot(121)
cnf_matrix = confusion_matrix(labels_test, y_pred,labels=class_labels)
show_confusion_matrix(cnf_matrix,class_labels,ax=ax1)
matt_score = metrics.matthews_corrcoef(labels_test, y_pred)
print("Matthews Correlation Coefficient (MCC): {}".format(matt_score))

ax2 = fig5.add_subplot(122)
# First plot our points
ax2.pcolormesh(xx, yy, Z, cmap= plt.cm.cool, alpha=0.1)
ax2.set_title("N support points: {}".format(len(support_points)))
ax2.set_aspect(1)
ax2.set_xlabel('Grade')
ax2.set_ylabel('Bumpiness')
# Circle the support points

ax2.scatter(x=support_points[:,0],y=support_points[:,1],s=20, facecolors='none', edgecolors='r')


I am working on the file "Class06/Class06.ipynb".
The first task is:
We see that there are actually a large number of support points in this model! So, even though it ends up being a linear decision boundary, it has quite a high complexity. Does the improvement in performance justify this increase in complexity? Perhaps...
Before we go on, we need to note that the SVC model only works well for small-ish datasets (less than 100,000 points). Beyond that, it will slow down considerably. It may be worth your time to try using this model on larger datasets, but I wanted to warn you about that.
First Hyperparameter: C
Let's dive into the model hyperparameters! The only hyperparameter for the model at this point is the C parameter. The SVC model is always looking for a perfect boundary: it wants to perfectly classify every point in the training set. So, what do we do about points that just aren't going to be classified perfectly? We apply a "penalty" for those points and try and lower the penalty as much as we can to do as best as we can. The size of the penalty we apply is related to the C parameter. Let's try two extremes with our linear decision boundary to see what they do (it defaults to the value of C=1.0 if we don't tell the model to use a different value).
We're going to use a set of sub-plots so we can put both the confusion matrix and the support vector points on the same line.
Can you write Python code for it?

# Start with a value of C=0.05
svcmodel2 = SVC(kernel='linear',C=.05)
svcmodel2.fit(features_train,labels_train)
Z = pd.Series(svcmodel2.predict(np.c_[xx.ravel(), yy.ravel()]), dtype='category').cat.codes.values.reshape(xx.shape)
support_points = features_train[svcmodel2.support_]

fig = plt.figure()
ax1 = fig.add_subplot(121)

y_pred = svcmodel2.predict(features_test)
cnf_matrix = confusion_matrix(labels_test, y_pred,labels=class_labels)
show_confusion_matrix(cnf_matrix,class_labels,ax=ax1)
matt_score = metrics.matthews_corrcoef(labels_test, y_pred)
print("Matthews Correlation Coefficient (MCC): {}".format(matt_score))

ax2 = fig.add_subplot(122)
# First plot our points
ax2.pcolormesh(xx, yy, Z, cmap= plt.cm.cool, alpha=0.1)
ax2.set_title("N support points: {}".format(len(support_points)))
ax2.set_aspect(1)
ax2.set_xlabel('Grade')
ax2.set_ylabel('Bumpiness')
# Circle the support points

ax2.scatter(x=support_points[:,0],y=support_points[:,1],s=20, facecolors='none', edgecolors='r')


I am working on the file "Class07/Class07.ipynb".
The first task is:
So our decision boundary is cleaned up significantly and we got a bump in the test performance of the model. Let's check one more value to see if we can do any better.
Can you write Python code for it?

# Load the model and fit the data
dtmodel = DecisionTreeClassifier(min_samples_leaf=5,random_state=32)
dtmodel.fit(features_train,labels_train)

y_pred = dtmodel.predict(features_test)

# Predict the boundary
Z = pd.Series(dtmodel.predict(np.c_[xx.ravel(), yy.ravel()]), dtype='category').cat.codes.values.reshape(xx.shape)

# First plot our points
testfig1, ax = plt.subplots()

plt.pcolormesh(xx, yy, Z, cmap= plt.cm.cool, alpha=0.1,axes=ax)
ax.set_aspect(1)

# Plot test points
groups = test.groupby('Speed')
# The next step is to cycle through the groups (based on our categories) and plot each one on the same axis.
for name, group in groups:
    ax.plot(group['Grade'], group['Bumpiness'], marker='o', linestyle='', ms=8, label=name)
ax.legend(bbox_to_anchor=(1.2,0.5))
ax.set_xlabel('Grade')
ax.set_ylabel('Bumpiness')
matt_score = metrics.matthews_corrcoef(labels_test, y_pred)
print("Matthews Correlation Coefficient (MCC): {}".format(matt_score))


I am working on the file "Class07/Class07.ipynb".
The first task is:
Take a look at the decision boundary for this classifier: it is all over the place! The tree tries to account for every point and so it creates braches where there shouldn't be branches. We have a classic case of overfitting! And the model performance isn't great either with an MCC of 0.84. It is time to tune the hyperparameters to see if we can do better. Let's start by tuning the minimum number of samples in the leaf nodes of the tree.
Can you write Python code for it?

# Load the model and fit the data
dtmodel = DecisionTreeClassifier(min_samples_leaf=10,random_state=32)
dtmodel.fit(features_train,labels_train)

y_pred = dtmodel.predict(features_test)

# Predict the boundary
Z = pd.Series(dtmodel.predict(np.c_[xx.ravel(), yy.ravel()]), dtype='category').cat.codes.values.reshape(xx.shape)

# First plot our points
testfig1, ax = plt.subplots()

plt.pcolormesh(xx, yy, Z, cmap= plt.cm.cool, alpha=0.1,axes=ax)
ax.set_aspect(1)

# Plot test points
groups = test.groupby('Speed')
# The next step is to cycle through the groups (based on our categories) and plot each one on the same axis.
for name, group in groups:
    ax.plot(group['Grade'], group['Bumpiness'], marker='o', linestyle='', ms=8, label=name)
ax.legend(bbox_to_anchor=(1.2,0.5))
ax.set_xlabel('Grade')
ax.set_ylabel('Bumpiness')
matt_score = metrics.matthews_corrcoef(labels_test, y_pred)
print("Matthews Correlation Coefficient (MCC): {}".format(matt_score))


I am working on the file "Class07/Class07.ipynb".
The first task is:
We got an MCC of 0.894 with a fairly simple decision boundary. That's good! There are, perhaps, a few too many wiggles in the boundary, but overall it is looking pretty good. Note that all of the boundaries are straight lines- that is because the decision tree is choosing cutoff values of "Grade" and "Bumpiness" to split the dataset along those lines. Overall this isn't too bad.
Ensemble Methods
The decision tree did a reasonable job of modeling our data but we only used one tree and one set of random values. What if we could do this many times and average the results. There are tools to do that! One of the strategies that ensemble methods will use is to scramble which of the training features it uses for each trial run. Let's take a quick look at that method, called a "bootstrap" sample.
We will start with 100 data points in our training sample. The ensemble model will break this up into 10 chunks of 10 data points each (each chunk labeled A-J). For the first model it takes chunks A-I and trains on them, then validates that model with chunk J. The next model will take chunks A-H and J, leaving chunk I for validation. It repeats this as many times as it needs. Thus the ensemble is doing training and validation all on the same set of data!
Data Snooping Warning

Although the ensemble is doing its own validation, that doesn't mean you can train with all of your data. You still need to keep the test data locked away and not used for training. This means we can compare the ensemble model to the other models without cheating ourselves.

We'll try out the simplest version of this first, called the RandomForestClassifier.
Can you write Python code for it?

# Load the model and fit the data
from sklearn.ensemble import RandomForestClassifier

rfmodel = RandomForestClassifier(n_estimators=100,random_state=32)
rfmodel.fit(features_train,labels_train)

y_pred = rfmodel.predict(features_test)

# Predict the boundary
Z = pd.Series(rfmodel.predict(np.c_[xx.ravel(), yy.ravel()]), dtype='category').cat.codes.values.reshape(xx.shape)

# First plot our points
testfig1, ax = plt.subplots()

plt.pcolormesh(xx, yy, Z, cmap= plt.cm.cool, alpha=0.1,axes=ax)
ax.set_aspect(1)

# Plot test points
groups = test.groupby('Speed')
# The next step is to cycle through the groups (based on our categories) and plot each one on the same axis.
for name, group in groups:
    ax.plot(group['Grade'], group['Bumpiness'], marker='o', linestyle='', ms=8, label=name)
ax.legend(bbox_to_anchor=(1.2,0.5))
ax.set_xlabel('Grade')
ax.set_ylabel('Bumpiness')
matt_score = metrics.matthews_corrcoef(labels_test, y_pred)
print("Matthews Correlation Coefficient (MCC): {}".format(matt_score))


I am working on the file "Class07/Class07.ipynb".
The first task is:
We see that the ensemble does a reasonable job- perhaps not better, in this case, than the decision tree by itself. However, there is something else that we get out of using the ensemble: it will tell us the relative importance of the different features it used in making the decision boundary. The list of feature importances are given in terms of percentage importance of each feature. This can be helpful in deciding which features to use as inputs to the model. If the ensemble says that a feature is not very important, you may be able to drop it and simplify your model.
Let's look at our feature importances:
Can you write Python code for it?


I am working on the file "Class07/Class07.ipynb".
The first task is:
We'll import the DecisionTreeClassifier and use all of the default values except for the random_state. We'll provide that so that the output is consistent run-to-run. The decision tree classifier uses the random number generator to make decisions about branching, so if we don't set this, we'll get different results every time we run the algorithm.
Can you write Python code for it?

import numpy as np
from sklearn.cross_validation import train_test_split
from sklearn.tree import DecisionTreeClassifier

# Create our decision boundary mesh
# point in the mesh
x_min = 0.0; x_max = 1.0 # Mesh x size
y_min = 0.0; y_max = 1.0  # Mesh y size
h = .01  # step size in the mesh
xx, yy = np.meshgrid(np.arange(x_min, x_max+h, h), np.arange(y_min, y_max+h, h))

# Split the data into training and testing sets and prepare the features and labels
train, test = train_test_split(speeddf, test_size=0.2, random_state=23)
features_train = train[['Grade','Bumpiness']].values
labels_train = train['Speed'].values
features_test = test[['Grade','Bumpiness']].values
labels_test = test['Speed'].values
class_labels = ["slow", "fast"]

# Load the model and fit the data
dtmodel = DecisionTreeClassifier(random_state=32)
dtmodel.fit(features_train,labels_train)

y_pred = dtmodel.predict(features_test)

# Predict the boundary
Z = pd.Series(dtmodel.predict(np.c_[xx.ravel(), yy.ravel()]), dtype='category').cat.codes.values.reshape(xx.shape)

# First plot our points
testfig1, ax = plt.subplots()

plt.pcolormesh(xx, yy, Z, cmap= plt.cm.cool, alpha=0.1,axes=ax)
ax.set_aspect(1)

# Plot test points
groups = test.groupby('Speed')
# The next step is to cycle through the groups (based on our categories) and plot each one on the same axis.
for name, group in groups:
    ax.plot(group['Grade'], group['Bumpiness'], marker='o', linestyle='', ms=8, label=name)
ax.legend(bbox_to_anchor=(1.2,0.5))
ax.set_xlabel('Grade')
ax.set_ylabel('Bumpiness')

import sklearn.metrics as metrics

recall_score = metrics.recall_score(labels_test, y_pred,labels=class_labels,average=None)
prec_score = metrics.precision_score(labels_test, y_pred,labels=class_labels,average=None)
f1_score = metrics.f1_score(labels_test, y_pred,labels=class_labels,average=None)

acc_score = metrics.accuracy_score(labels_test, y_pred)
matt_score = metrics.matthews_corrcoef(labels_test, y_pred)

print("Class-dependent Metrics")
print("Sensitivity/Recall Score:...


I am working on the file "Class07/Class07.ipynb".
The first task is:
XGBoost
This last ensemble method is new enough to not be a part of the regular sklearn toolbox yet. However, it has made a fairly big splash in the machine learning community for its performance on real-world data.
Can you write Python code for it?

import xgboost
xgbmodel = xgboost.XGBClassifier(n_estimators=100, seed=32)
xgbmodel.fit(features_train,labels_train)

y_pred = xgbmodel.predict(features_test)

# Predict the boundary
Z = pd.Series(xgbmodel.predict(np.c_[xx.ravel(), yy.ravel()]), dtype='category').cat.codes.values.reshape(xx.shape)

# First plot our points
testfig1, ax = plt.subplots()

plt.pcolormesh(xx, yy, Z, cmap= plt.cm.cool, alpha=0.1,axes=ax)
ax.set_aspect(1)

# Plot test points
groups = test.groupby('Speed')
# The next step is to cycle through the groups (based on our categories) and plot each one on the same axis.
for name, group in groups:
    ax.plot(group['Grade'], group['Bumpiness'], marker='o', linestyle='', ms=8, label=name)
ax.legend(bbox_to_anchor=(1.2,0.5))
ax.set_xlabel('Grade')
ax.set_ylabel('Bumpiness')
matt_score = metrics.matthews_corrcoef(labels_test, y_pred)
print("Matthews Correlation Coefficient (MCC): {}".format(matt_score))


I am working on the file "Class08/Class08.ipynb".
The first task is:
Again, there is one point in the 'A' data that stands out. That is certainly an outlier. But we'd like to use a tool that does this scaling for us and will let us take the 'B' data into consideration, too. Fortunately there is a tool: the Mahalanobis distance explanation here. Let's implement this tool using just the 'A' column to compare it to our calculation.
Can you write Python code for it?

from sklearn.covariance import EmpiricalCovariance

# We initialize the model and fit in one step here.
emp_covA = EmpiricalCovariance().fit(X)

# The Mahalanobis function returns the distance squared, so we take the square root.
mahal_dist = np.sqrt(emp_covA.mahalanobis(X))

# And plot the data
fig, ax1 = plt.subplots()
ax1.scatter(xv,X,marker='^')
ax1.set_xlabel('Point Number')
ax1.set_ylabel('A Points')
ax2 = ax1.twinx()
ax2.set_ylabel('Mahalanobis Distance',color='m')
ax2.scatter(xv,mahal_dist,color='m',marker='v')


I am working on the file "Class08/Class08.ipynb".
The first task is:
Functional Outlier Detection
So that first set of data was fairly easy to work with: the distribution of data points looked Gaussian (it was) and so the outliers were easy to detect using methods that depend on normal distributions. What if our data aren't normally distributed? Let's look at another fake dataset that has a couple of outliers.
Can you write Python code for it?

dfb = pd.read_csv("Class08_fakedata2.csv")
dfb.plot.scatter(x='x',y='y')


I am working on the file "Class08/Class08.ipynb".
The first task is:
It looks like one of the points is around the 100 row point. But where is the other? Let's try plotting the 'B' column.
Can you write Python code for it?


I am working on the file "Class08/Class08.ipynb".
The first task is:
Now we can sort the data by the residual column and look for the largest values.
Can you write Python code for it?


I am working on the file "Class08/Class08.ipynb".
The first task is:
Now we'll create a new dataset where we select only the data where the residuals are less than a threshold value. We'll try fitting this data to see if our model has improved without the outliers.
Can you write Python code for it?

threshold = 0.3

dfbcl = dfb[dfb['residual'] <= threshold]

svrmodel2 = SVR(C=10,gamma=1)
svrmodel2.fit(dfbcl['x'].values.reshape(-1,1), dfbcl['y'].values)

Y_pred = svrmodel2.predict(X_plot[:,None])

fig, ax = plt.subplots()
# First plot our points
ax.scatter(dfbcl['x'].values,dfbcl['y'].values,color='black')
ax.plot(X_plot,Y_pred,c='r')
print("Score: {}".format(svrmodel2.score(dfb['x'].values.reshape(-1,1),dfb['y'].values)))


I am working on the file "Class08/Class08.ipynb".
The first task is:
Outlier Category
Instead of removing the points, we could label them or otherwise mark them. We'll do this by creating a True-False categorical column that marks the potential outliers as True.
Can you write Python code for it?

dfa['outlier'] = (dfa['dist'] > cutoff_distance).astype('category')
print(dfa.dtypes)
dfa.head()


I am working on the file "Class08/Class08.ipynb".
The first task is:
That doesn't get us anywhere. How about trying the Mahalanobis distance?
Can you write Python code for it?

xy=dfb[['x','y']].values
emp_covxy = EmpiricalCovariance().fit(xy)
mahal_distxy = np.sqrt(emp_covxy.mahalanobis(xy))

trainfig, ax = plt.subplots()
ax.scatter(dfb['x'].values,dfb['y'].values,color='black')
ax.set_title("Mahalanobis distances")

# Show contours of the distance functions
xx, yy = np.meshgrid(np.linspace(plt.xlim()[0], plt.xlim()[1], 100),
                     np.linspace(plt.ylim()[0], plt.ylim()[1], 100))
zz = np.c_[xx.ravel(), yy.ravel()]

mahal_emp_cov = emp_covxy.mahalanobis(zz)
mahal_emp_cov = mahal_emp_cov.reshape(xx.shape)
emp_cov_contour = ax.contour(xx, yy, np.sqrt(mahal_emp_cov),
                                  cmap=plt.cm.PuBu_r,
                                  linestyles='dashed')
ax.clabel(emp_cov_contour, inline=1, fontsize=10)
ax.set_xlabel('x')
ax.set_ylabel('y')


I am working on the file "Class08/Class08.ipynb".
The first task is:
This didn't really help: it maybe spotted an outlier in the 'B' data, but it has way too many in the 'A' data. Let's try going at this another way. We'll start with the 'A' data.
First we'll calculate the mean $\mu$ and standard deviation $\sigma$ for the data. Then we'll scale the data like this: we subtract the mean from each data point $x_i$ and divide by the standard deviation.
$\frac{x_i - \mu}{\sigma}$
This gives us a scaled "distance" between each data point and the mean where the scale is the standard deviation. Let's run this for the 'A' data.
Can you write Python code for it?

X=dfa['A'].values.reshape(-1,1)
xv= range(0,len(X))
mean = X.mean()
stdev = X.std()

# Now calculate the scaled distance (which is always a positive number, so we take the absolute value of the distance)
dist = np.abs((X - mean)/stdev)

#Plot the data
fig, ax1 = plt.subplots()
ax1.scatter(xv,X,marker='^')
ax1.set_xlabel('Point Number')
ax1.set_ylabel('A Points')
ax2 = ax1.twinx()
ax2.set_ylabel('Scaled Distance',color='m')
ax2.scatter(xv,dist,color='m',marker='v')


I am working on the file "Class08/Class08.ipynb".
The first task is:
This looks exactly like our manual distance calculation. That's good. Now let's extend this to use both columns as the inputs. We'll again calculate the distances and plot them for each point.
Can you write Python code for it?

XAB=dfa[['A','B']].values
emp_covAB = EmpiricalCovariance().fit(XAB)
mahal_dist2 = np.sqrt(emp_covAB.mahalanobis(XAB))
fig, ax1 = plt.subplots()
ax1.scatter(xv,X,marker='^')
ax1.set_xlabel('Point Number')
ax1.set_ylabel('A Points')
ax2 = ax1.twinx()
ax2.set_ylabel('Mahalanobis Distance',color='m')
ax2.scatter(xv,mahal_dist2,color='m',marker='v')


I am working on the file "Class08/Class08.ipynb".
The first task is:
Well, it isn't clear from this plot that there are any problems in the 'B' column. There is another way we can visualize the data: using a boxplot. This type of plot looks at the mean and standard deviation of the data. Then it provides a single plot that shows various limits based on these values. Here's an explanation of the ranges of data used by the boxplot (explanation from here). Any points outside of this range are plotted as separate data points and should be visible.
Can you write Python code for it?


I am working on the file "Class09/Class09.ipynb".
The first task is:
That's not really what I want. What I want is the frequency of earthquakes over time. How about plotting a histogram of the number of earthquakes of each magnitude over the time window. We'll need to create a feature where we round the magnitude first. Then we can use a seaborn function to plot the different magnitudes.
Can you write Python code for it?

import seaborn as sns
eqINdf['maggroup'] = eqINdf['mag'].apply(np.round)
g= sns.FacetGrid(eqINdf, col="maggroup",sharey=False)
g = g.map(plt.hist, "time",bins=100)
g.set_xticklabels(rotation=45)


I am working on the file "Class09/Class09.ipynb".
The first task is:
The algorithm does the best it can with what we gave it- it found three clusters of data. But the decision boundaries do not match the data very well. So let's try 4 instead.
Can you write Python code for it?

kmeans4 = KMeans(n_clusters=4).fit(df1)

Z = kmeans4.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
centroids = kmeans4.cluster_centers_

plt.imshow(Z, interpolation='nearest',
           extent=(xx.min(), xx.max(), yy.min(), yy.max()),
           cmap=plt.cm.Paired,
           aspect='auto', origin='lower')

plt.plot(df1['Distance_Feature'], df1['Speeding_Feature'], 'k.', markersize=2)
# Plot the centroids as a white X
plt.scatter(centroids[:, 0], centroids[:, 1],
            marker='x', s=169, linewidths=3,
            color='w', zorder=10)
plt.title('K-means with 4 Clusters')
plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)
plt.ylabel('Speeding_Feature')
plt.xlabel('Distance_Feature')


I am working on the file "Class09/Class09.ipynb".
The first task is:
We will make a visualization like we've done with the classification algorithms: we want to map out which regions will be predicted to be which class. That will take a bit of work.
Can you write Python code for it?

import numpy as np

# Step size of the mesh. Decrease to increase the quality of the plot.
h = 0.5     # point in the mesh [x_min, x_max]x[y_min, y_max].

# Create the mesh for plotting the decision boundary
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

# Obtain labels for each point in mesh. Use last trained model.
Z = kmeans2.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)

# Get the centroid (or center position) of each region so we can plot that, too.
centroids = kmeans2.cluster_centers_

# First plot the mesh that has the predictions for each point on the mesh.
plt.imshow(Z, interpolation='nearest',
           extent=(xx.min(), xx.max(), yy.min(), yy.max()),
           cmap=plt.cm.Paired,
           aspect='auto', origin='lower')

# Now put the points on top
plt.plot(df1['Distance_Feature'], df1['Speeding_Feature'], 'k.', markersize=2)

# Plot the centroids as a white X
plt.scatter(centroids[:, 0], centroids[:, 1],
            marker='x', s=169, linewidths=3,
            color='w', zorder=10)
plt.title('K-means with 2 Clusters')

# And fix the plot limits and labels.
plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)
plt.ylabel('Speeding_Feature')
plt.xlabel('Distance_Feature')


I am working on the file "Class10/Class10.ipynb".
The first task is:
Standarization scaling
As we noted above, the goal is to turn these features into unitless parameters. That means we have to divide the feature by a quantity with the same units. There are two typical ways of doing this: we'll start with standardization or Z-scale normalization. We need to calculate the standard deviation of the feature, then divide the entire column by the standard deviation. Because the standard deviation has the same units as the feature itself, this takes care of the normalization.
We'll also do one more thing: we'll subtract the mean value of the distribution from each data point first. That way we end up with a distribution that is centered at zero. Of course there is a library that will do this for us. We need to fit the library on the data we want to scale, then use the transform() function to transform the input data based on the scaler. We can do both features at the same time: the scaler keeps track of them individually.
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler

std_scaler = StandardScaler().fit(df[['Alcohol', 'Malic acid']])
df_std = std_scaler.transform(df[['Alcohol', 'Malic acid']])

print('Mean before standardization:\nAlcohol={:.2f}, Malic acid={:.2f}'
      .format(df['Alcohol'].mean(), df['Malic acid'].mean()))

print('\nMean after standardization:\nAlcohol={:.2f}, Malic acid={:.2f}'
      .format(df_std[:,0].mean(), df_std[:,1].mean()))

print('\nStandard deviation before standardization:\nAlcohol={:.2f}, Malic acid={:.2f}'
      .format(df['Alcohol'].std(), df['Malic acid'].std()))

print('\nStandard deviation after standardization:\nAlcohol={:.2f}, Malic acid={:.2f}'
      .format(df_std[:,0].std(), df_std[:,1].std()))


I am working on the file "Class10/Class10.ipynb".
The first task is:
There are a number of different features here. We'll focus on the first two: Alcohol which has units (percent/volumne) and Malic acid with units (g/l). Any machine learning algorithm that uses these features is going to treat them as if they were on the same scale. However, they aren't. So we need to re-scale them to unitless values in order to work with them.
Let's first look at the distribution of points in those columns.
Can you write Python code for it?

# Plot the first two feature columns
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('white')

#plt.figure(figsize=(8,6))

plt.scatter(df['Alcohol'], df['Malic acid'])
plt.xlabel('Alcohol (%/L)')
plt.ylabel('Malic Acid (g/L)')
plt.xlim(0,16)
plt.ylim(0,16)
plt.axes().set_aspect('equal')


I am working on the file "Class10/Class10.ipynb".
The first task is:
We've accomplished what we set out to do: the new means are close to zero and the standard deviations are 1. Let's see how it changed the shape of the distributions.
Can you write Python code for it?

fig, ax = plt.subplots(1,2)

for a,d,l in zip(range(len(ax)),
               (df[['Alcohol', 'Malic acid']].values, df_std),
               ('Input scale',
                'Standardized [$N  (\mu=0, \; \sigma=1)$]')
                ):
    for i,c in zip(range(1,4), ('red', 'blue', 'green')):
        ax[a].scatter(d[df['Wine Class'].values == i, 0],
                  d[df['Wine Class'].values == i, 1],
                  alpha=0.5,
                  color=c,
                  label='Class %s' %i
                  )
        ax[a].set_aspect('equal')
    ax[a].set_title(l)
    ax[a].set_xlabel('Alcohol')
    ax[a].set_ylabel('Malic Acid')
    ax[a].legend(loc='upper left')
    ax[a].grid()
plt.tight_layout()


I am working on the file "Class11/Class11.ipynb".
The first task is:
As we saw with the accuracy score, we're doing pretty good. There doesn't appear to be a lot of obvious mistakes here: It looks like Tony Blair was mistaken for some of the other people, but nothing stands out dramatically.
Computer Vision: Finding Faces in an Image
Although this is not a class on computer vision, there is a connection between the two. We may want to use images as our input data. If we can reduce the size of the features by, for example, identifying the face out of an image, we can reduce the work we have to do by focusing on the face. The way we identify faces is connected to the eigenface plot we made above. The basic trick is to search for features that are common to all human faces. Then, once we found some of the basic features, we iteratively look more closely at those areas of the image, honing in on the face.
There are some good resouces on how this works here, here, and here.
We can test out the image recognition using the OpenCV (cv2) package for python.
Haar Cascade
The tool we want to use is called the Haar Cascade classifier. We'll need to load it in, then use it to find faces in a picture.
Can you write Python code for it?

import cv2
import matplotlib.pyplot as plt
# Load the classifier
face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')

# read the image
img = cv2.imread('ABBA.png')

# make sure it is grayscale - the classifier does not work on color images
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

plt.imshow(gray,cmap=plt.cm.gray)
plt.axis('off')


I am working on the file "Class11/Class11.ipynb".
The first task is:
It looks like it is an array of numbers that appear to by our $(x,y)$ coordinates. Let's try plotting this to see what we've got.
Can you write Python code for it?

# We can turn the labels into a categorical column, then get the category codes to use them to plot the different labels.
ylabels = train_labels.astype('category').cat.codes

# Make the scatter plot and label the axis
plt.scatter(pca_train_features[:,0],pca_train_features[:,1],c=ylabels,cmap=plt.cm.spectral)
plt.xlabel('PCA_0')
plt.ylabel('PCA_1')


I am working on the file "Class11/Class11.ipynb".
The first task is:
Let's image this and see what the images look like. We will pick a random person, list their name, and show their picture. We can rerun the cell to get different random pictures.
Can you write Python code for it?

pickaperson = np.random.randint(0,n_images)
sns.set_style('white')
personname = target_names[imagelabels[pickaperson]]
print("Person {} :".format(pickaperson) + personname)
personimage = imagefeatures[pickaperson].reshape(h,w)
plt.axis('off')
plt.imshow(personimage,cmap=plt.cm.gray)


I am working on the file "Class11/Class11.ipynb".
The first task is:
Now that we've transformed the training features, let's take a look at what we got. Let's start by looking at the output from the transform() function.
Can you write Python code for it?


I am working on the file "Class11/Class11.ipynb".
The first task is:
Now we want to detect the faces in the image. We'll apply the facial detection algorithm.
Can you write Python code for it?

# Detect faces in the image
faces = face_cascade.detectMultiScale(
    gray,
    scaleFactor=1.1,
    minNeighbors=5,
    minSize=(30, 30))
print("Found {0} faces".format(len(faces)))


I am working on the file "Class11/Class11.ipynb".
The first task is:
Ok, back to business. Let's split off our training/testing dataset. We want to be a little careful about this: we don't want to accidentally get all images of one person in our training dataset. We can split it based on the label so that we have equal distributions of all the different classes. We use the stratify=imagelabels option that tells train_test_split to use equal numbers of the different classes in each split.
We will then find the principal components using the PCA analysis. In this case we'll use 150 components. So the net effect is that we are shrinking the number of features from 1850 to 150.
We're using two more options for the PCA: when working with images, I've found that the performance is better when we use the randomized SVD solver and set the whiten=True option  (see this page for more information).
Can you write Python code for it?

# split into a training and testing set
features_train, features_test, labels_train, labels_test = train_test_split(imagefeatures, imagelabels, stratify=imagelabels, test_size=0.20, random_state=23)

# Compute the PCA (eigenfaces) on the face dataset
n_components = 150

imgpca = PCA(n_components=n_components,svd_solver='randomized',
          whiten=True).fit(features_train)

pca_features_train = imgpca.transform(features_train)


I am working on the file "Class11/Class11.ipynb".
The first task is:
So our machine learning algorithm would have to train on 1850 different features! Let's take a look at a couple of the images. We'll first get out the images and their labels. How to do this comes from the documentation for this dataset.
Can you write Python code for it?

# First we get the list of features
imagefeatures = people.data
# And the list of labels - this is a set of numbers (0-4) corresponding to the 5 different people
imagelabels = people.target
# We'd like to get the people's names, too.
target_names = people.target_names
print(target_names)


I am working on the file "Class11/Class11.ipynb".
The first task is:
We can look at the way in which the PCA has broken up the images- in this case it is something like looking at which axis (or eigenvector if you've had linear algebra) it used. These components tell us something about the features that were most common to all of the images. That is why the PCA algorithm chose them.
Can you write Python code for it?

# First get the components and reshape them to the right size for images
eigenfaces = imgpca.components_.reshape((n_components, h, w))

# A helper function to make plots of the components

def plot_gallery(images, titles, h, w, n_row=3, n_col=5):
    plt.figure(figsize=(1.8 * n_col, 2.4 * n_row))
    plt.subplots_adjust(bottom=0, left=.01, right=.99,
                        top=.90, hspace=.35)
    for i in range(n_row * n_col):
        plt.subplot(n_row, n_col, i + 1)
        plt.imshow(images[i].reshape((h, w)), cmap=plt.cm.gray)
        plt.title(titles[i], size=12)
        plt.xticks(())
        plt.yticks(())

# Plot the gallery of the most significative eigenfaces
eigenface_titles = [
    "eigenface %d" % i for i in range(eigenfaces.shape[0])]

plot_gallery(eigenfaces, eigenface_titles, h, w)
plt.tight_layout()


I am working on the file "Class11/Class11.ipynb".
The first task is:
We now have just two input features to work with: PCA_0 and PCA_1. Any future machine learning will now be simplified especially since we see that the data are mostly separated in this two-dimensional space.
We'll run through a comparison to see how the PCA has changed the performance of a couple of different machine learning algorithms that we've seen before (Suppor Vector Classifiers and Decision Tree Classifiers). Remember that the PCA features are half the size of the original dataset (we've shrunk them down to 2 columns instead of 4). Before we fit the data, we need to transform the test features using the PCA tranformation we trained earlier. Remember that we didn't use the test data to train the PCA, so we haven't cheated on this one.
Can you write Python code for it?

pca_test_features = sklearn_pca.transform(test_features)

# Import both the SVC and DTC classifiers
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier

# Train a basic SVC (default values) on the entire dataset and score it
svcmodel = SVC()
svcmodel.fit(train_features,train_labels)
print("SVC on complete dataset score: {}".format(svcmodel.score(test_features, test_labels)))

# Train a similar SVC model using the PCA reduced features and score it
svcmodelPCA = SVC()
svcmodelPCA.fit(pca_train_features,train_labels)
print("SVC on PCA dataset score     : {}".format(svcmodelPCA.score(pca_test_features, test_labels)))

# Try the decision tree version on the entire model
dtmodel = DecisionTreeClassifier(random_state=32)
dtmodel.fit(train_features,train_labels)
print("DTC on complete dataset score: {}".format(dtmodel.score(test_features,test_labels)))

dtmodelPCA = DecisionTreeClassifier(random_state=32)
dtmodelPCA.fit(pca_train_features,train_labels)
print("DTC on PCA dataset score     : {}".format(dtmodelPCA.score(pca_test_features,test_labels)))


I am working on the file "Class11/Class11.ipynb".
The first task is:
We'll come back to these images in a little bit- they give us the start of an algorithm to detect faces out of other images.
Now that we've got the PCA features, let's try an run a machine learning algorithm to see if we can detect the faces. We'll try using both the complete dataset as well as the PCA reduced data set. This time let's compare how long it takes to train the model.
Can you write Python code for it?

from time import time
t0 = time()
svcmodel = SVC().fit(features_train, labels_train)
accuracy_score = svcmodel.score(features_test,labels_test)
print("Done in {0:.3f}s with accuracy {1:.3f}".format(time() - t0,accuracy_score))

pca_features_test = imgpca.transform(features_test)

t0 = time()
svcmodel_pca = SVC().fit(pca_features_train, labels_train)
accuracy_score_pca = svcmodel_pca.score(pca_features_test,labels_test)
print("done in {0:.3f}s with accuracy {1:.3f}".format(time() - t0,accuracy_score_pca))


I am working on the file "Class11/Class11.ipynb".
The first task is:
Wow! The PCA dataset scores the same as the full datasets using both machine learning algorithms. Now we haven't optimized anything, but this is a good indication that the PCA is doing its job: shrinking the number of input features while keeping the performance of the machine learning algorithm.
Working with Facial Images
Now that we've got a basic idea on how the PCA works and when to use it, let's try using it on image data. We are interested in training a machine learning algorithm to recognize the faces of specific people: we provide a picture of a person and the machine tells us who it is. We're going to follow this tutorial. I've already downloaded the data and stored it in a binary storage container (called a pickle in python). We'll load the data and take a look at what we've got.
Can you write Python code for it?

import pickle
people = pickle.load(open('people.pkl', 'rb')) # This is how we load pickled data

#Find out how many faces we have, and
#the size of each picture from.
n_images, h, w = people.images.shape
print("{0} images that are {1} pixels by {2} pixels in size with a total of {3} features per image".format(n_images,h,w,h*w))


I am working on the file "Class12/Class12.ipynb".
The first task is:
We see that there are large regions where the probabilities are pretty flat, but right near the decision boundary there is a smooth transition up from $0$ to $1$. This is just what we want.
Image Class Prediction
Even though our model isn't very sophisticated, we can still use it to train a model to predict the class for a set of images. We'll use the MNIST data that we first saw way back in Class01. Here's a quick review of what the data look like.
Can you write Python code for it?

digitDF = pd.read_csv('../Class01/Class01_digits_data.csv')
testnum = 61

testimage = digitDF.loc[testnum][0:64].values.reshape((8,8))
print('Expected Digit: {0:d}'.format(digitDF['target'][testnum]))
plt.gray()
plt.matshow(testimage)


I am working on the file "Class12/Class12.ipynb".
The first task is:
We'll now split the data in to train/test sections and train a logistic regression.
Can you write Python code for it?

digittrain, digittest = train_test_split(digitDF, test_size=0.2, random_state=23)
features = digittrain.ix[:, digittrain.columns != 'target'].values
labels = digittrain['target'].values
class_names = range(10) # class labels are just 0-9

logreg = LogisticRegression(C=1e5)
logreg.fit(features,labels)

testinputs = digittest.ix[:, digittest.columns != 'target'].values
predictions = logreg.predict(testinputs)
actuals = digittest['target'].values

cnf_matrix = metrics.confusion_matrix(actuals, predictions)

def show_confusion_matrix(cnf_matrix, class_labels):
    plt.matshow(cnf_matrix,cmap=plt.cm.YlGn,alpha=0.7)
    ax = plt.gca()
    ax.set_xlabel('Predicted Label', fontsize=16)
    ax.set_xticks(range(0,len(class_labels)))
    ax.set_xticklabels(class_labels)
    ax.set_ylabel('Actual Label', fontsize=16, rotation=90)
    ax.set_yticks(range(0,len(class_labels)))
    ax.set_yticklabels(class_labels)
    ax.xaxis.set_label_position('top')
    ax.xaxis.tick_top()

    for row in range(len(cnf_matrix)):
        for col in range(len(cnf_matrix[row])):
            ax.text(col, row, cnf_matrix[row][col], va='center', ha='center', fontsize=16)

show_confusion_matrix(cnf_matrix,class_labels=class_names)
score = metrics.accuracy_score(actuals, predictions)
print("Accuracy Score: {}".format(score))


I am working on the file "Class13/Class13.ipynb".
The first task is:
Convolution Neural Networks
One of the things about image recognition is that there is often a lot of information about the image hiding in its edges, shapes, and patterns. If we only look at the image pixels, we miss using these other types of information. A simple way to train a model using more information is to use a trick called convolution. This essentially gives the model additional features to look at. We build a new neural network using both fully connected layers as well as convolution layers.
A lot of work has been done by researchers to find sets of neural network layers that work well for different types of images. The following set works fairly well for the handwriting images or other simple images with patterns.
Can you write Python code for it?

# Simplified LeNet model
#-------------------------------------------------------------------------------
def leNet_model(pixdim1, pixdim2):

    from keras.layers.core import Activation, Dense
    from keras.models import Sequential

    from keras.layers.convolutional import Convolution2D, MaxPooling2D
    from keras.layers import Dropout, Flatten

    # create model
    model = Sequential()
    model.add(Convolution2D(20, 5, 5, border_mode='valid', input_shape=(1, pixdim1, pixdim2)))
    model.add(Activation('tanh'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Convolution2D(50, 3, 3))
    model.add(Activation('tanh'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Flatten())

    model.add(Dense(500))
    model.add(Activation('tanh'))
    model.add(Dense(40))

    model.add(Dense(num_classes, activation='softmax'))
    # Compile model
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model

# build the model
model = leNet_model(pixdim1,pixdim2)
# Fit the model
model.fit(features_train_set, labels_train_cat, validation_data=(features_test_set, labels_test_cat), nb_epoch=2, batch_size=600, verbose=1)
# Final evaluation of the model
scores = model.evaluate(features_test_set, labels_test_cat, verbose=0)
print("Baseline Accuracy {:.2f}%".format(scores[1]*100))


I am working on the file "Class13/Class13.ipynb".
The first task is:
Finally, we need to shift our output predictions to a dummy variable since we are looking for a categorical output. Whether this step is necessary is dependent on the specific neural network tool you are using: some don't require this, others do. Because the Keras tool we will be using requires that classification models be set up this way, we'll do it. There is a Keras tool to make this easy. We'll also get the number of classes in the output test dataset.
Can you write Python code for it?

from keras import backend as K
K.set_image_dim_ordering('th')
from keras.utils import np_utils

# one hot encode outputs
labels_train_cat = np_utils.to_categorical(labels_train)
labels_test_cat = np_utils.to_categorical(labels_test)
num_classes = labels_test_cat.shape[1]


I am working on the file "Class13/Class13.ipynb".
The first task is:
Simple One-Layer Network
We're now ready to build our first neural network. This typically requires a couple of steps:

Design the neural network's layers
Train the network

The specific code you use to design the layers depends on the neural network tool you are using. The basic types of layers are the same for all of the tools, though. In this case we will do the following.
Can you write Python code for it?

# One Dense Layer model
#-------------------------------------------------------------------------------
def simpleModel():

    from keras.layers.core import Activation, Dense
    from keras.models import Sequential

    # create model and tell the tool that each layer we define will be added in sequential order
    model = Sequential()

    # our first layer is called a "Dense" layer - it is a fully connected set of nodes (every input connects to every node)
    # We tell it what shape to expect the input: in this case it is the total number of input pixels.
    # The output dimension is the number of nodes to create for this layer. We want one node for each pixel, so we use the number of pixels as our output dimension
    model.add(Dense(input_dim=num_pixels, output_dim=num_pixels, init='normal'))

    # The next thing we need to do is tell the network how to adjust the probabilities that the nodes compute. We'll use the hyperbolic tangent:
    # It is like the logistic regression, but on the scale of -1 to 1 instead of 0 to 1.
    model.add(Activation('tanh'))
    # Finally we need to create out output layer. In this tool we need to specify an output dimension the same as the number of classes we are trying to predict
    model.add(Dense(output_dim=num_classes, activation='softmax'))
    # This time, the softmax activation is the same as the logistic regression, but generalized for more than 2 output choices (we have 10 classes here)
    #model.add(Activation('softmax'))
    # Compile model
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model


I am working on the file "Class13/Class13.ipynb".
The first task is:
So we now have 60,000 training images and 10,000 test images. Each image is $28 \times 28$ pixels and has an output label indicating what number is supposed to have been drawn.
We'll take a quick look at the data to make sure it makes sense. Visualizing is usually our best option.
Can you write Python code for it?

import matplotlib.pyplot as plt
testnum = 45076
testimage = features_train[testnum]
print('Expected Digit: {0:d}'.format(labels_train[testnum]))
print('Max value in image: {}'.format(testimage.max()))
plt.gray()
plt.matshow(testimage)


I am working on the file "Class13/Class13.ipynb".
The first task is:
So we're above 90% accuracy after only 2 epochs. Try increasing the number of epochs to see if there is a point where this levels out.
Let's look at the confusion matrix to see if there are any particularly problematic numbers.
Can you write Python code for it?

actuals = labels_test
predictions = model.predict_classes(X_test)

import sklearn.metrics as metrics

cnf_matrix = metrics.confusion_matrix(actuals, predictions)

def show_confusion_matrix(cnf_matrix, class_labels):
    fig, ax = plt.subplots(figsize=(20, 10))
    ax.matshow(cnf_matrix,cmap=plt.cm.YlGn,alpha=0.7)
    ax.set_xlabel('Predicted Label', fontsize=16)
    ax.set_xticks(range(0,len(class_labels)))
    ax.set_xticklabels(class_labels)
    ax.set_ylabel('Actual Label', fontsize=16, rotation=90)
    ax.set_yticks(range(0,len(class_labels)))
    ax.set_yticklabels(class_labels)
    ax.xaxis.set_label_position('top')
    ax.xaxis.tick_top()

    for row in range(len(cnf_matrix)):
        for col in range(len(cnf_matrix[row])):
            ax.text(col, row, cnf_matrix[row][col], va='center', ha='center', fontsize=16)

show_confusion_matrix(cnf_matrix,class_labels=range(10))
score = metrics.accuracy_score(actuals, predictions)
print("Accuracy Score: {}".format(score))


I am working on the file "Class13/Class13.ipynb".
The first task is:
The last thing we need to do is to flatten out our input arrays. This simple neural network works better if we are only working with a single dimension of data.
Then we are ready now to train our model. We'll do a couple of things all at the same time: first we'll give the fit() function the test data for validation. It will use that data at every step to figure out how it is doing in terms of it's fit.
We'll also set up a couple of other parameters:
* Number of Epochs: how many times should we repeat the network training, learning from each previous iteration
* How large should we make our batches: how many images do we pile on at one time while training - typically about 1/100 of the total samples works well (as a general rule of thumb).
Can you write Python code for it?

# flatten 28*28 images to a 784 vector for each image
X_train = features_train.reshape(features_train_set.shape[0], num_pixels)
X_test = features_test.reshape(features_test_set.shape[0], num_pixels)

# build the model
model = simpleModel()
# Fit the model
model.fit(X_train, labels_train_cat, validation_data=(X_test, labels_test_cat), nb_epoch=2, batch_size=600, verbose=1)
# Final evaluation of the model
scores = model.evaluate(X_test, labels_test_cat, verbose=0)
print("Baseline Accuracy {:.2f}%".format(scores[1]*100))


I am working on the file "Classes+and+Objects.ipynb".
The first task is:
Abstract Classes and Interfaces
Abstract classes are not intended to be instantiated because all the method definitions are empty – all the insides of the methods must be implemented in a subclass.
They serves as a template for suitable objects by defining a list of methods that these objects must implement.
Can you write Python code for it?

# Abstract Classes

class shape2D:
    def area(self):
        raise NotImplementedError()

class shape3D:
    def volume(self):
        raise NotImplementedError()

sh1 = shape2D()
sh1.area()

class shape2D:
    def area(self):
        raise NotImplementedError()

class shape3D:
    def volume(self):
        raise NotImplementedError()

class Square(shape2D):
    def __init__(self,width):
        self.width = width

    def area(self):
        return self.width ** 2

s1 = Square(2)
s1.area()